Elasticsearch 503 error when checking server status

Question

I've been using elasticsearch as a search engine for my Rails application, however it stopped working properly due to a reason i can't understand. When making a curl request to elasticsearch server i get a 503 error.

curl -XGET http://localhost:9200
{
  "ok" : true,
  "status" : 503,
  "name" : "Killpower",
  "version" : {
    "number" : "0.90.3",
    "build_hash" : "5c38d6076448b899d758f29443329571e2522410",
    "build_timestamp" : "2013-08-06T13:18:31Z",
    "build_snapshot" : false,
    "lucene_version" : "4.4"
  },
  "tagline" : "You Know, for Search"
}

I've tried to solve the problem with reloading elasticsearch service and installing a new version without any success.

Here is what I found in logs

[2013-09-03 12:31:45,320][INFO ][node                     ] [Killpower] version[0.90.3], 
pid[4222], build[5c38d60/2013-08-06T13:18:31Z]
[2013-09-03 12:31:45,321][INFO ][node                     ] [Killpower] initializing ...
[2013-09-03 12:31:45,327][INFO ][plugins                  ] [Killpower] loaded [], sites []
[2013-09-03 12:31:47,248][INFO ][node                     ] [Killpower] initialized
[2013-09-03 12:31:47,248][INFO ][node                     ] [Killpower] starting ...
[2013-09-03 12:31:47,313][INFO ][transport                ] [Killpower] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/172.20.46.32:9300]}
[2013-09-03 12:31:51,443][INFO ][discovery.zen            ] [Killpower] master_left [[Amphibion][IMB4uACSTyOx3MO2u-FsWg][inet[/fe80:0:0:0:52e5:49ff:fec2:9718%2:9300]]], reason [do not exists on master, act as master failure]
[2013-09-03 12:31:51,446][INFO ][discovery                ] [Killpower] elasticsearch/iavpGrMtRHmWLHMTNpscGQ
[2013-09-03 12:31:51,483][INFO ][http                     ] [Killpower] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/172.20.46.32:9200]}
[2013-09-03 12:31:51,484][INFO ][node                     ] [Killpower] started
[2013-09-03 12:31:54,712][WARN ][transport.netty          ] [Killpower] exception caught on transport layer [[id: 0xa929f24d, /127.0.0.1:50456 => /127.0.0.1:9300]], closing connection

After that also follows a long java stacktrace. How can i fix the problem?

Jerejereld

Hi, You might want to look at #19794016 – Reata

Answer 1

There could be another instance in your local network with elasticsearch server up and running. Since shards in elasticsearch are working from the box and enabled by default there could be a conflict of master node. Please review your elasticsearch.log file. If there is something like

{"error":"MasterNotDiscoveredException[waited for [30s]]","status":503} 

You should go to your config file /etc/elasticsearch/elasticsearch.yml and add this line:

discovery.zen.ping.multicast.enabled: false

Gelatinate

Answer 2

in addition check value of

discovery.zen.ping.unicast.hosts

because it is necessary to discover elastic nodes

Tenney

what is the expected value? – Slipover

Your hosts list, e.g. for single node use 127.0.0.1:10101 – Tenney

Answer 3

In my case, I'm using a proxy on my terminal which runs the rails server. After disabling the proxy and restart rails server, the problem solved.

Resnatron

Answer 4

Install head so that you can see the status in more detail. I think 503 can appear in status when the cluster does not recover properly eg lost/non-recoverable shard

In the elasticsearch directory:

bin/plugin -install mobz/elasticsearch-head

Then in your browser:

http://127.0.0.1:9200/_plugin/head/

Largely

If the status is 503, head plugin cannot get information from the server. You can only read the log files. – Monograph