ElasticSearch entered "read only" mode, node cannot be altered

Question

Something happened during the night to my ES cluster (composed of 5 data nodes, 3 master nodes).

I have no idea what happened but all the indices and data were deleted and the cluster entered a "read only" mode, possibly hacked?

When trying to get Kibana running, I get the following:

Tried restarting Kibana - it restarted, nothing changed. Tried restarting Elastic - it restarted (all nodes), nothing changed.

I then had a look at the cluster settings and this is what I got:

{
  "persistent": {
    "cluster": {
      "routing": {
        "allocation": {
          "enable": "all"
        }
      },
      "blocks": {
        "read_only": "true"
      }
    }
  },
  "transient": {
    "cluster": {
      "routing": {
        "allocation": {
          "enable": "all"
        }
      }
    }
  }
}

I tried undoing the read only as follows:

PUT _cluster/settings
{
  "persistent": {
    "blocks.read_only": false
  }
}

No luck as you can see:

{
  "error": {
    "root_cause": [
      {
        "type": "cluster_block_exception",
        "reason": "blocked by: [FORBIDDEN/6/cluster read-only (api)];"
      }
    ],
    "type": "cluster_block_exception",
    "reason": "blocked by: [FORBIDDEN/6/cluster read-only (api)];"
  },
  "status": 403
}

Any ideas?

UPDATE: Problem solved by Andrei Stefan, now for the more important part - why? What happened and why? I've lost all data and my cluster entered a read-only mode.

Arriola

Can you check if you have a blocks.read_only: true setting in your elasticsearch.yml file? – Oxus

@Oxus Checked, it does not exist in any of the nodes.. – Arriola

Answer 1

The correct command is:

PUT /_cluster/settings
{
  "persistent" : {
    "cluster.blocks.read_only" : false
  }
}

Chauvin

curl -u elastic:changeme -XPUT 'localhost:9200/_cluster/settings' -H 'Content-Type: application/json' -d '{"persistent":{"cluster.blocks.read_only":false}}' unfortunately didn't work for me :( although it does change the read only property: {"acknowledged":true,"persistent":{"cluster":{"blocks":{"read_only":"false"}}},"transient":{}} – Tolmach

Answer 2

It turns out ES has some thresholds for available disk space, and when the "flood" one is hit, it puts the indeces into read only mode.

In order to set it back (tested with ES6), you will need to do the following:

PUT /[index_name]/_settings
{
  "index.blocks.read_only_allow_delete": null
}

More information can be found on the following page of the documentation: https://www.elastic.co/guide/en/elasticsearch/reference/current/disk-allocator.html

Warranty

this is exactly what happened in our case! thank you for sharing – Mustache

Appreciated, for resetting all indices at once could be issued something like: curl -X PUT "localhost:9200/*/_settings" -H 'Content-Type: application/json' -d'{"index.blocks.read_only_allow_delete": null}' – Sawmill