Can't close ElasticSearch index on AWS?

Question

I've created a new AWS ElasticSearch domain, for testing. I use ES on a different host right now, and I'm looking to move to AWS.

One thing I need to do is set the mapping (analyzers) on my instance. In order to do this, I need to "close" the index, or else ES will just raise an exception.

Whenever I try to close the index, though, I get an exception from AWS:

Your request: '/_all/_close' is not allowed by CloudSearch.

The AWS ES documentation specifically says to do this in some cases:

 curl -XPOST 'http://search-weblogs-abcdefghijklmnojiu.us-east-1.a9.com/_all/_close'

I haven't found any documentation that says why I wouldn't be able to close my indices on AWS ES, nor have I found anyone else who has this problem.

It's also a bit strange that I've got an ElasticSearch domain, but it's giving me a CloudSearch error message, since I thought those were different services, though I suppose one is implemented in terms of the other.

thanks!

Hybridism

nice finding, did you contact AWS directly for this issue? – Crosshead

BMW: I've been getting an (even more generic) error when trying to post to their forums, as well. :P – Hybridism

Answer 1

AWS Elasticsearch does not support the "close" operation on indexes.

http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-managedomains.html

"Currently, Amazon ES does not support the Elasticsearch _close API"

Mohican

Yes, they changed the documentation. I guess their documentation and implementation teams don't actually talk to each other. Even stranger, there's no record of this change in the Document History. – Hybridism

If you want to add mappings, you'll have to reindex. There are some good articles about using aliases to do this. Basic steps - create a read alias and a write alias. Point read alias to current index. Create new index with new mappings. Point write alias to new index. Then copy old index to new index (read from read alias, write to write alias). This involves modifying existing code to use the aliases instead. elastic.co/guide/en/elasticsearch/guide/current/reindex.html elastic.co/guide/en/elasticsearch/reference/current/… – Mohican

Can confirm that 4 years on from this answer, _close is still not supported on AWS Elasticsearch (at least not on a 6.0 instance I created yesterday), but it is no longer mentioned in the above linked document. We successfully use reindex to update mappings. – Jobbery

Answer 2

According to the AWS document I found recently, you have to first upgrade your elastic search domain to version 7.4 or greater.

https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-handling-errors.html#aes-troubleshooting-close-api

Coulombe

Answer 3

If you don't care about reindexing, you can just create a new index with the settings changes and then reindex:

PUT my-new-index
{
  "index": {
    "refresh_interval": "5s",
    "blocks": {
      "write": "false"
    },
    "max_result_window": "500000",
    "number_of_replicas": "0",
    // your settings changes
  }
}

then

POST /_reindex
{
  "source": {
    "index": "my-index"
  },
  "dest": {
    "index": "my-new-index"
  }
}

Additive

Answer 4

Since closing all indices at once is a dangerous action, it is maybe disabled by default on your cluster. You need to make sure that your elasticsearch.yml configuration file doesn't contain this:

action.destructive_requires_name: true

You can either set this in your configuration file and restart your cluster, but I strongly advise against that since this opens the door to all kinds of other destructive actions, like deleting all your indices at once.

action.destructive_requires_name: false

What you should do instead is to temporarily update the cluster settings using

curl -XPUT localhost:9200/_cluster/settings -d '{
    "persistent" : {
        "action.destructive_requires_name" : false
    }
}'

Then close all your indices

curl -XPOST localhost:9200/_all/_close

And then reset the settings to a safer value:

curl -XPUT localhost:9200/_cluster/settings -d '{
    "persistent" : {
        "action.destructive_requires_name" : true
    }
}'

Deaver

AWS ES is a higher-level service, and doesn't provide direct access to elasticsearch.yml. I get a "payload is not allowed" error when I try to PUT to /_cluster/settings like that. – Hybridism

Are you at least able to close a single index using /your_index/_close? – Deaver

Your answer gave me an idea. I'm doing .close(index=INDEX_NAME) from my client library, but it seems like it's hitting the wire as /_all/_close for some reason. So I tried curl -XPOST host/my_index/_close, and AWS reports that this "is not allowed by CloudSearch", either. Darn. – Hybridism

Can you run this: curl -XGET host/_cluster/settings ? Since that seems to be allowed. – Deaver

AWS Cloudsearch is a different beast than AWS Elasticsearch, though. Are you sure you're hitting the correct host? That's the only explanation I see. – Deaver

"Your request: '/_cluster/settings' is not allowed for verb: GET". – Hybridism

See my last comment. You're definitely hitting an AWS CloudSearch host not an AWS Elasticsearch one. – Deaver

You're right that CloudSearch is a different beast. I've never done anything with CloudSearch, which is why it's strange I'm getting CloudSearch error messages. The "CloudSearch Management Console" tells me: "You have no search domains". The "Amazon Elasticsearch Service dashboard" lists the one ES domain that I'm using here. – Hybridism