Text fields are not optimised for operations that require per-document field data

Question

Having switched from Elasticsearch to Opensearch, my application now fails to run a simple query with:

"Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [status] in order to load field data by uninverting the inverted index. Note that this can use significant memory."

There's a question concerning the same error at Searchkick / Elasticsearch Error: Please use a keyword field instead. Alternatively, set fielddata=true on [name], but there the problem was only affecting tests and I only get the problem (so far) in development mode.

Here's the query being run:

::Record.search(q ? q : "*",
                where: where_clause,
                fields: fields,
                match: :word_middle,
                per_page: max_per_page(per_page) || 30,
                page: page || 1,
                order: sort_clause,
                aggs: aggs,
                misspellings: {below: 5}

If I take out aggs then the search is fine, but they're essential for the application. Removing :status from the list of aggregation fields causes the error to name the next field in the array as the problem. So, I presumably need to specify the correct type for each field used in aggregations. But how?

The Searchkick docs suggest this example under "Advanced Mapping" (https://github.com/ankane/searchkick):

class Product < ApplicationRecord
  searchkick mappings: {
    properties: {
      name: {type: "keyword"}
    }
  }
end

So, I tried this:

# in models/Record.rb
mapping_properties = {}
aggregation_fields.each do |af|
  mapping_properties[af] = { type: 'keyword' }
end
searchkick mappings: {
             properties: mapping_properties
          }

But, the same problem persists. I also tried something similar to that shown in the linked post, e.g.

mappings: {
      properties: {
        name: {
          type: "text",
          fielddata: true,
          fields: {
            keyword: {
              type: "keyword"
            }
          }
        }
      }
    }

...but similarly without luck.

Can anyone suggest how this might be fixed?

Israelitish

Its just related to make your field keyword on which you are doing aggs, can you do this from fresh and it should work, might be requirted change isn't applied to your mapping, you can check that using _mapping Api of Elasticsearch, OS should have similar api to check mapping of index – Perot

Searchkick is creating the indexes, therefore I need to know how to instruct it to do so in a manner to which Opensearch will not object. How to do that remains rather unclear. – Israelitish

sorry, I can help with the APis not with code, and ultimately its creating index in Elasticsearch/os which you can easily check with their REST api – Perot

Indeed, the fields are created as type "text" rather than "keyword", as the error message indicates, despite the attempts described above to create them as Opensearch requires. – Israelitish

Answer 1

This was not related to using Opensearch. Instead, the method of indexing used caused the indexes to be generated incorrectly.

See: Searchkick memory leak

Israelitish

Answer 2

The immediate issue was dealt with by changing all the fields used for aggregations, so rather than:

aggs = %w(field1 field2 field3 ...)

...in the above search query. I used:

aggs = %w(field1.keyword field2.keyword field3.keyword ...)

Israelitish

Yes, as explained in the first comment of mine, you needed the .keyword, good that you were using the default mapping, so these are created automatically for you but its not good always :) – Perot

I could not infer that from your comment at all. – Israelitish

ok, still glad you were able to find the solution 😊 – Perot

Answer 3

If you have a Tomcat running with Elastic Search while you delete your index it will trigger this error.

Stop your tomcat, again delete your index and you're good to go.

Hope this helps someone

Eph