All shards failed

Question

I was working on elastic search and it was working perfectly. Today I just restarted my remote server (Ubuntu). Now I am searching in my indexes, it is giving me this error.

{"error":"SearchPhaseExecutionException[Failed to execute phase [query_fetch], all shards failed]","status":503}

I also checked the health. The status is red. Can anyone tell me what's the issue.

Procambium

Answer 1

It is possible on your restart some shards were not recovered, causing the cluster to stay red.
If you hit:
http://<yourhost>:9200/_cluster/health/?level=shards you can look for red shards.

I have had issues on restart where shards end up in a non recoverable state. My solution was to simply delete that index completely. That is not an ideal solution for everyone.

It is also nice to visualize issues like this with a plugin like:
Elasticsearch Head

Nubia

Hi @mconlin, how do you figure which index to be deleted in this case? – Flapjack

using head you will see greyed out unrecovered shards on the last row. – Nubia

If you are on docker, try to force recreate elasticsearch and kibana – Goalie

Note that a large number of shards will take a long time to initialize; try reducing the number, see: elastic.co/blog/… – Comestible

Answer 2

If you're running a single node cluster for some reason, you might simply need to do avoid replicas, like this:

curl -XPUT -H 'Content-Type: application/json' 'localhost:9200/_settings' -d '
{
    "index" : {
        "number_of_replicas" : 0
    }
}'

Doing this you'll force to use es without replicas

Cushiony

what will this do?? @paulo , pls explain – Bunn

Tells ES that all your indices are on a single machine: no replicas. – Eichmann

I had one machine and one elasticsearch node installed on it without replicas and this command worked for me. – Etymon

Answer 3

first thing first, all shards failed exception is not as dramatic as it sounds, it means shards were failed while serving a request(query or index), and there could be multiple reasons for it like

1. Shards are actually in non-recoverable state, if your cluster and index state are in Yellow and RED, then it is one of the reasons.
2. Due to some shard recovery happening in background, shards didn't respond.
3. Due to bad syntax of your query, ES responds in all shards failed.

In order to fix the issue, you need to filter it in one of the above category and based on that appropriate fix is required.

The one mentioned in the question, is clearly in the first bucket as cluster health is RED, means one or more primary shards are missing, and my this SO answer will help you fix RED cluster issue, which will fix the all shards exception in this case.

Pressmark

Answer 4

For Elasticsearch > 5.0 it's possible to get some more information from this endpoint:

http://localhost:9200/_cluster/allocation/explain?pretty

I just ran into a case where I hit the virtual disk limit configured in Docker Desktop and adding an additional, unrelated container caused ES to fail.

Kharkov

Thank You for posting this comment. It allowed me to debug my issue – Crusade

Answer 5

If you encounter this apparent index corruption in a running system, you can work around it by deleting all files called segments.gen. It is advisory only, and Lucene can recover correctly without it.

From ElasticSearch Blog

Knesset

The current link is redirecting to the main elastic.co page. It no longer shows the blog entry. Edit submitted. – Leyte

Answer 6

If you are upgrading the Elasticsearch and have multiple versions you can face this issue. Continue to upgrade ALL nodes. And run the daemon reload.

sudo systemctl daemon-reload

Intention

Answer 7

Depending on how you serve the request body, you may have parameters that are not required, or filled in by default causing this.

Using the Swagger UI to run & test my endpoints caused this problem. I had yellow shards but this wasn't the fault. It was the pre-populated body.

This caused the problem:

{
  "foo": "string",
  "bar": "string",
  "page": 0,
  "pageSize": 0
}

This solved the problem:

{
  "foo": "",
  "bar": "",
  "page": 1,
  "pageSize": 10
}

The pagination numbers are arbitrary

Mezzorilievo