Elasticsearch synonym analyzer not working

Question

EDIT: To add on to this, the synonyms seem to be working with basic querystring queries.

"query_string" : {
    "default_field" : "location.region.name.raw",
    "query" : "nh"
}

This returns all of the results for New Hampshire, but a "match" query for "nh" returns no results.

---

I'm trying to add synonyms to my location fields in my Elastic index, so that if I do a location search for "Mass," "Ma," or "Massachusetts" I'll get the same results each time. I added the synonyms filter to my settings and changed the mapping for locations. Here are my settings:

analysis":{
    "analyzer":{
        "synonyms":{
            "filter":[
                "lowercase",
                "synonym_filter"
            ],
        "tokenizer": "standard"
    }
},
"filter":{
    "synonym_filter":{
        "type": "synonym",
        "synonyms":[
            "United States,US,USA,USA=>usa",
            "Alabama,Al,Ala,Ala",
            "Alaska,Ak,Alas,Alas",
            "Arizona,Az,Ariz",
            "Arkansas,Ar,Ark",
            "California,Ca,Calif,Cal",
            "Colorado,Co,Colo,Col",
            "Connecticut,Ct,Conn",
            "Deleware,De,Del",
            "District of Columbia,Dc,Wash Dc,Washington Dc=>Dc",
            "Florida,Fl,Fla,Flor",
            "Georgia,Ga",
            "Hawaii,Hi",
            "Idaho,Id,Ida",
            "Illinois,Il,Ill,Ills",
            "Indiana,In,Ind",
            "Iowa,Ia,Ioa",
            "Kansas,Kans,Kan,Ks",
            "Kentucky,Ky,Ken,Kent",
            "Louisiana,La",
            "Maine,Me",
            "Maryland,Md",
            "Massachusetts,Ma,Mass",
            "Michigan,Mi,Mich",
            "Minnesota,Mn,Minn",
            "Mississippi,Ms,Miss",
            "Missouri,Mo",
            "Montana,Mt,Mont",
            "Nebraska,Ne,Neb,Nebr",
            "Nevada,Nv,Nev",
            "New Hampshire,Nh=>Nh",
            "New Jersey,Nj=>Nj",
            "New Mexico,Nm,N Mex,New M=>Nm",
            "New York,Ny=>Ny",
            "North Carolina,Nc,N Car=>Nc",
            "North Dakota,Nd,N Dak, NoDak=>Nd",
            "Ohio,Oh,O",
            "Oklahoma,Ok,Okla",
            "Oregon,Or,Oreg,Ore",
            "Pennsylvania,Pa,Penn,Penna",
            "Rhode Island,Ri,Ri & PP,R Isl=>Ri",
            "South Carolina,Sc,S Car=>Sc",
            "South Dakota,Sd,S Dak,SoDak=>Sd",
            "Tennessee,Te,Tenn",
            "Texas,Tx,Tex",
            "Utah,Ut",
            "Vermont,Vt",
            "Virginia,Va,Virg",
            "Washington,Wa,Wash,Wn",
            "West Virginia,Wv,W Va, W Virg=>Wv",
            "Wisconsin,Wi,Wis,Wisc",
            "Wyomin,Wi,Wyo"
        ]
    }
}

And the mapping for the location.region field:

"region":{
    "properties":{
        "id":{"type": "long"},
        "name":{
            "type": "string",
            "analyzer": "synonyms",
            "fields":{"raw":{"type": "string", "index": "not_analyzed" }}
        }
    }
}

But the synonyms analyzer doesn't seem to be doing anything. This query for example:

"match" : {
    "location.region.name" : {
        "query" : "Massachusetts",
        "type" : "phrase",
        "analyzer" : "synonyms"
    }
}

This returns hundreds of results, but if I replace "Massachusetts" with "Ma" or "Mass" I get 0 results. Why isn't it working?

Otti

Answer 1

The order of the filters is

filter":[
    "lowercase",
    "synonym_filter"
]

So, if elasticsearch is "lowercasing" first the tokens, when it executes the second step, synonym_filter, it won't match any of the entries you have defined.

To solve the problem, I would define the synonyms in lower case

Beckford

I want to ask a question about work mechanism of filters. How filters work in analyzer? In this example, lowercase filter work and return tokens and tokens are taken by synonym_filter, and synonym_filter work and return new filtered tokens. Is this scenario right or how? – Christie

Yes, the scenario you are describing is right :) In general, tokenizer (in this case standard tokenizer) is executed and then token filters in the defined order (in this case first lowercase and then synonym_filter). The documentation explains it pretty good elastic.co/guide/en/elasticsearch/reference/1.5/… – Beckford

Answer 2

You can also define your synonyms filter as case insensitive:

    "filter":{
        "synonym_filter":{
            "type": "synonym",
            "ignore_case" : "true",
            "synonyms":[
                ...
            ]
        }
    }

Exultant

ignore_case is deprecated now – Claudication