Docker container unhealthy but no error in the logs

Question

I am using the official docker-compose file of airflow to spin it up.

Some of my containers seem unhealthy:

34d8698d67e7   apache/airflow:2.0.2   "/usr/bin/dumb-init …"   31 minutes ago   Up 28 minutes (unhealthy)   0.0.0.0:5555->5555/tcp, :::5555->5555/tcp, 8080/tcp   airflow_flower_1
a291cf238b9f   apache/airflow:2.0.2   "/usr/bin/dumb-init …"   31 minutes ago   Up 29 minutes               8080/tcp                                              airflow_airflow-init_1
fdb20e9152f3   apache/airflow:2.0.2   "/usr/bin/dumb-init …"   31 minutes ago   Up 29 minutes (unhealthy)   0.0.0.0:8080->8080/tcp, :::8080->8080/tcp             airflow_airflow-webserver_1
abf5a16aa846   apache/airflow:2.0.2   "/usr/bin/dumb-init …"   31 minutes ago   Up 29 minutes               8080/tcp                                              airflow_airflow-worker_1
f6dc352f407b   apache/airflow:2.0.2   "/usr/bin/dumb-init …"   31 minutes ago   Up 28 minutes               8080/tcp                                              airflow_airflow-scheduler_1
12dfc71e518f   redis:latest           "docker-entrypoint.s…"   31 minutes ago   Up 29 minutes (healthy)     0.0.0.0:6379->6379/tcp, :::6379->6379/tcp             airflow_redis_1

However the logs of one of them for example do not seem very informative.

# docker logs -f fdb20e9152f3
WARNING! You should run the image with GID (Group ID) set to 0
         even if you use 'airflow' user (UID=50000)
 You started the image with UID=50000 and GID=50000
 This is to make sure you can run the image with an arbitrary UID in the future.
 See more about it in the Airflow's docker image documentation
     http://airflow.apache.org/docs/docker-stack/entrypoint
BACKEND=postgresql+psycopg2
DB_HOST=my-db-endpoint
DB_PORT=5432
WARNING! You should run the image with GID (Group ID) set to 0
         even if you use 'airflow' user (UID=50000)
 You started the image with UID=50000 and GID=50000
 This is to make sure you can run the image with an arbitrary UID in the future.
 See more about it in the Airflow's docker image documentation
     http://airflow.apache.org/docs/docker-stack/entrypoint
BACKEND=postgresql+psycopg2
DB_HOST=my-db-endpoint
DB_PORT=5432

Regardless of any airflow - specific issues, how can I check docker - wise what's going on?

Docker seems to be aware of a couple of containers not being healty.

edit: both failing containers have the healtcheck condition

healthcheck:
      test: ["CMD", "curl", "--fail", "http://localhost:5555/"]

and

healthcheck:
      test: ["CMD", "curl", "--fail", "http://localhost:8080/"]

that seems to be failing by looking into their inspect output

 Failed to connect to localhost port 8080: Connection refused

but I cannot pinpoint what is causing the failure.

edit: I have tried following the instructions to start the init service first as well

# docker-compose up airflow-init
Starting airflow_redis_1 ... done
Starting airflow_airflow-init_1 ... done
Attaching to airflow_airflow-init_1
airflow-init_1       | BACKEND=postgresql+psycopg2
airflow-init_1       | DB_HOST=my-db-endpoint
airflow-init_1       | DB_PORT=5432

but it never exits, it prints the above message and that's it...

Brazell

Did you execute docker-compose up airflow-init ? – Terefah

It is one of the services, it runs with just docker-compose up, no? – Brazell

The most weird part (to start with) is the fact that the only thing I see in my logs is the message I am printing in my question – Brazell

Did you follow the instructions in the docs? airflow.apache.org/docs/apache-airflow/stable/start/… – Terefah

yes, check my new edit on the question – Brazell

If the curl request to http://localhost:8080 is failing then there may be a firewall or other networking issue preventing access to the port. Are you able to view the airflow webserver GUI in a browser? – Slyke

Answer 1

I ran into similar issue and it was docker volume causing the issue. As I was running lots of containers on my mac, there wasn;t enough disk space. I managed to fixed this issue my pruning the docker volume.

docker volume prune

This will remove any unused volume on your mac book. Before running this command please check if you got any useful data.

Iamb

Answer 2

For docker-compose, from the entrypoint, the default value of group id is 0.

"${AIRFLOW_UID:-50000}:${AIRFLOW_GID:-0}"

Edit your docker-compose.yaml file or ad ass env.sh file in your Airflow project repository.

Robillard

Answer 3

It does seems to be an error due to less memory allocated to docker for running this image, try to increase the resources available to docker and see the magic

Ebeneser

Answer 4

I ran into similar issue and the healthchecks were "causing" this. I was running them using default container's user.

Just to give a try, I changed the healthckeck command to start using airflow user instead, as follow:

$ runuser -u airflow -- <healthckeck command>

And it solved. I'm gonna change the user whom runs docker compose up to airflow from now on.

Pasho