What does "Limit of total fields [1000] in index [] has been exceeded" means in Elasticsearch

Question

I have ElasticSearch index created which has approx 350 fields (including nested fields), I have defined mapping only for few of them. While calling the _update API I am getting below exception with 400 Bad request.

Limit of total fields [1000] in index [test_index] has been exceeded

I want to understand the exact reason for this exception since my number of fields and fields for which mapping is defined both are less than 1000.

Note: I have nested fields and dynamic mapping enabled.

Cynicism

Answer 1

You can fix the issue by increasing the value of index.mapping.total_fields.limit (default 1000)

index.mapping.total_fields.limit

The maximum number of fields in an index. Field and object mappings, as well as field aliases count towards this limit. The default value is 1000.

To increase total fields limit to 2000, try this

PUT test_index/_settings
{
  "index.mapping.total_fields.limit": 2000
}

The reason to limit the number of fields is :

Defining too many fields in an index is a condition that can lead to a mapping explosion, which can cause out of memory errors and difficult situations to recover from. This is quite common with dynamic mappings. Every time a document contains new fields, those will end up in the index’s mappings

Related Resources :
1. Get the number of fields on an index
2. Settings to prevent mappings explosion

Tapis

Thanks for the quick reply ! Actually I checked the number of fields in my index using the link you shared , its 928 , so I am not able to figure out the exact cause – Cynicism

@Cynicism Maybe you are trying to index new document that contains more than 72 new fields – Tapis

Answer 2

I landed here via google. This worked for me (from the shell):

curl -s -XPUT http://localhost:9200/test-index/_settings  -H 'Content-Type: application/json' -d '{"index.mapping.total_fields.limit": 2000}'

Curcuma

If somebody decides just to copy/paste this.. There should be empty space between -d option and header content. – Georgia

I downvoted because this doesn’t answer the question, in addition to duplicate a previous answer. – Forkey

Please do not just copy this "fix". While it does remove the error, it is more a quickfix that will introduce performance issues in the long run. Try and reduce your index to only the fields you really need during search (and not complete objects) – Prisoner

Answer 3

Another approach to handle this is to carefully design the mappings as the OP seems to be doing and to turn off dynamic mapping by setting dynamic = false (or even dynamic = strict)

Note that this approach can be applied to the entire mapping or to other properties/nested objects within, which enables quite a bit good degree of flexibility.

Reference to Dynamic mapping in the docs

Randolph

This would mean that you will not index the new dynamic field that being added. This is not a good approach when it comes to query. – Taper

@JojoJosephKunathucherry that depends on the use case. If you know what fields you need to search on, then there is no need to index any random field that may show up in your logs/documents. By doing so you can save significantly indexing time and disk space – Randolph

Answer 4

From (Kibana) . You can also convert it to curl and hit it.

PUT project-documents/_settings
{
  "index.mapping.total_fields.limit": 1100
}

Use this with caution. Do not increase the size too much as it will degrade the performance.

Taper

Answer 5

Another possible solution, esp. when working in Groovy context: check that your data is a (well-formatted) String instance. In my case, I had been transforming my mapped data to Json using myMappedData as JSON in Groovy. The solution was to use (myMappedData as JSON).toString(). See https://mcmap.net/q/243044/-elasticsearch-indexes-json-with-escaped-quotation-marks-quot-limit-of-total-fields-1000-has-been-exceeded-quot.

Wilkerson

Answer 6

One entirely tangential cause might be that you are feeding Elasticsearch with recursive data. In my case, I was using nxlog to unwrap a json formatted nginx log entry and feed it into graylog using GELF.

I left the original Message field, in addition to the unpacked fields, so that the GELF message looked like:

{"timestamp": 123456789, ...,
 "message": "timestamp": 123456789, ...
}

Only after instructing nxlog to overwrite the original message with a flat string, my messages started to appear again in Elasticsearch.

Heaven know which part of my pipeline caused Elasticsearch to get its panties in a bunch, but making sure the "message" field doesn't contain nested json seems to have resolved my failure mode...

Overblouse

Answer 7

Maybe as a side note and question. How do you know you are coming close to 1000 fields? Currently, we are typically bouncing against it and then need to fix it. But could you create an alert? What I have not found is a GET method that returns the number of fields in use on an index. (also not in _stats). So we currently read the _mappings and I wrote a piece of code that tries counting the fields. I am not sure it's correct but each time we bounce against the problem, the counter returns something very close to 1000, typically meaning that the next document tried adding a few fields, that didn't work, and so the document did not get indexed and you are stuck on 997 or so. The gist of it is to run over the fields in the mapping, and count the field itself, and one more if there is a .sort, one more for .keyword and one more for .norm. Feels like a big workaround...

Deficiency

if you have a workaround to solve the problem, then just show your workaround and please don't ask questions in your answer post. if you have a question, put it in a question post. – Deify