ElasticSearch Healthcheck on docker-compose failing

Question

Elasticsearch healthcheck on docker-compose stops any dependent services because the container is always unhealthy. I see this when I run docker ps -a --format "table {{.Names}}\t{{.Image}}\t{{.Status}}"

NAMES           IMAGE                  STATUS
elasticsearch   elasticsearch:7.12.1   Up 26 seconds (unhealthy)

I am attempting to start metricbeat such that elasticsearch, kibana and logstash are started:

metricbeat:
  image: elastic/metricbeat:7.12.1
  user: root
  depends_on:
    elasticsearch:
      condition: service_healthy
    kibana:
      condition: service_healthy
    logstash:
      condition: service_healthy
    redis:
      condition: service_healthy

How can I ensure that the elasticsearch (and other containers are healthy) and allow for metricbeat to start with all the resources available?

I would avoid creating a Docker image for any of these unless absolutely required.

My docker-compose configuration looks like this:

version: '3.7'
services:
  elasticsearch:
    # specifying discovery.type='single-node' bypasses bootstrapping checks.
    image: elasticsearch:7.12.1
    container_name: elasticsearch
    healthcheck:
      test: [ "CMD", "curl",  "--fail" , "http://elasticsearch:9200/_cluster/health?wait_for_status=green&timeout=1s", "||", "exit", "1" ]
      interval: 5s
      timeout: 3s
          
    volumes:
      - elasticsearch-data:/usr/share/elasticsearch/data
    networks:
      - elastic
    ports:
      - 9200:9200
      - 9300:9300
    labels:
      co.elastic.metrics/module: "elasticsearch"
      co.elastic.metrics/hosts: "http://elasticsearch:9200"
      co.elastic.metrics/metricsets: "node_stats,node"
      co.elastic.metrics/xpack.enabled: "true"
    environment:
      - node.name=elasticsearch
      - cluster.name=cluster-7
      - discovery.type=single-node
      - 'ES_JAVA_OPTS=-Xms512m -Xmx512m'
      - xpack.monitoring.enabled=true
      - xpack.monitoring.elasticsearch.collection.enabled=true
      
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    cap_add:
      - IPC_LOCK

Psychro

Answer 1

The approach can be different depending on elastic version. Also you need to use something else if you use OpenSearch they have bit different output for health (but just for info)

I`m adding what I use in my docker-compose

healthcheck:
  interval: 10s
  retries: 80
  test: curl --write-out 'HTTP %{http_code}' --fail --silent --output /dev/null http://localhost:8200/

or

healthcheck:
   test: curl -s http://elasticsearch01:9200 >/dev/null || exit 1
   interval: 30s
   timeout: 10s
   retries: 50

Antiworld

Answer 2

For d-c file ver. 3.7 i use this construction:

healthcheck:
  test: curl -u elastic:elastic -s -f elasticsearch:9200/_cat/health >/dev/null || exit 1
  interval: 30s
  timeout: 10s
  retries: 5

Nashoma

Answer 3

for healthchecks we tried out several:

With ssl:

    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --user elastic:${ELASTIC_PASSWORD} --cacert /usr/share/elasticsearch/config/certs/elasticsearch/elasticsearch.crt -X GET http://localhost:9200/_cluster/health?pretty | grep status | grep -q '\\(green\\|yellow\\)'"
        ]
      interval: 10s
      timeout: 10s
      retries: 24

Without ssl:

    healthcheck:
      test:     
        [
          "CMD-SHELL",
          "curl -s --user elastic:${ELASTIC_PASSWORD} -X GET http://localhost:9200/_cluster/health?pretty | grep status | grep -q '\\(green\\|yellow\\)'"
        ]
      interval: 10s
      timeout: 10s
      retries: 24

The cacert location may differ per docker build. If you use deviantony/docker-elk for example, the base command is docker compose exec -it elasticsearch /bin/bash -c 'curl -u elastic:I4Umonitor --cacert /usr/share/elasticsearch/config/ca.crt -XGET https://localhost:9200/_cluster/health?pretty | grep status | grep "\(green\|yellow\)"'

so the healthcheck looks like this:

    healthcheck:
      test:     
        [
          "CMD-SHELL",
          "curl -s --user elastic:${ELASTIC_PASSWORD} --cacert /usr/share/elasticsearch/config/ca.crt -XGET https://localhost:9200/_cluster/health?pretty | grep status | grep -q '\\(green\\|yellow\\)'"
        ]
      interval: 10s
      timeout: 10s
      retries: 24

Mont

Answer 4

Troubleshot quick tips:

1 - Quick check service health with docker inspect:

sudo docker inspect --format "{{json .State.Health }}" elasticsearch | jq

2 - If not "healthy", go inside docker and test:

sudo docker exec -it <container_id> bash

root@container_id:/# curl -s http://elasticsearch:9200

It should return something like:

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    66  100    66    0     0   6186      0 --:--:-- --:--:-- --:--:--  6600

If not, then your domain name or auth is probably wrong.

Else try it again in docker healthcheck as suggested:

test: curl -s http://elasticsearch:9200 >/dev/null || exit 1

Note: If you're using environment variables, they might not being recognized with a single dollar

$VARIALBE

Then try with double dollars

$$VARIALBE

Hope it helps...

Concrescence