Containers not restarted after update with restart always in docker-compose.yml

Question

I have some containers which all of them have the always restart value in the docker-compose file like this:

version: "3.7"
services:

  container:
    image: ghost:latest
    container_name: some_container
    restart: always
    depends_on:
       - ...
    ports:
       - ...
...

As soon as the OS (Flatcar Linux / CoreOS) has updated itself none of the containers restart. But if I just do $ sudo docker ps all of the containers starts at once. Whats up with that and how do I fix it so my containers automatically restarts after an update?

EDIT:

Not sure what is unclear about my question, restart: always is turned on. Unless I'm missing some vital thing in the documentation, this command should restart the container even if the docker daemon is restarted (after an os reboot).

Copy of one my comments from below:

Ok, so help me out here. As you can see in my question, I have restart: always turned on. All these containers are started successfully and are running well. Then the OS updates itself automatically and restarts itself. After this restart the docker daemon is restarted. But for some reasons the containers I had running WITH RESTART: ALWAYS turned on DOES NOT START. If I enter my server at this moment, type sudo docker ps to list my running containers, suddenly all containers are booted up and I see the list. So why wasn't the containers started, even though the daemon is running?

Runner

so you want to start docker containers automatically after a host reboot? – Nesmith

I would like the containers which has restart: always in the config to start on reboot, yes. – Runner

How is docker installed? And can you verify the engine is running (dockerd process) before running docker ps? – Armet

Flatcar-linux is a fork of CoreOS where docker comes pre-installed. Why would a docker command work if the service isn't running? How do I check that? – Runner

Is the dockerd process running before you run a docker command? I'm wondering if the docker.sock listener (often done in systemd) waits until there's a connection to launch the daemon. – Armet

You can also check systemctl status docker to see if systemd started it. – Armet

Thanks. Should have checked that! It was inactive after restart, I assumed it was active as docker ps worked. Did systemctl enable docker and now its working as expect. Write an answer! – Runner

Answer 1

From the comments it appears the docker service was not configured to automatically start on boot. Docker is a client server app, and the server runs from systemd with a separate service for the docker socket used by the client to talk to the server. Therefore it's possible for any call with the docker command to cause the server to get launched by hitting the docker socket.

The service state in systemd can be checked with:

systemctl status docker

or you may want to check:

systemctl is-enabled docker

It can be manually started with:

systemctl start docker

And it can be enabled to start with:

systemctl enable docker

All of the above commands need to be run as root.

Armet

Answer 2

If the docker container has been created before, it's [restart policy][1] may not be updated automatically by changing it in the docker compose YAML file. If you change Restart Policy in the YAML file:

# cat docker-compose.yml 
version: "3"
services:
  <your-service>:
    restart: always

You can see the container details in which RestartPolicy has old value yet:

# docker inspect <your-container> | fgrep -i restart -A 5
        
    "RestartCount": 0,
--
            "RestartPolicy": {
                "Name": "",

Name is the Restart Policy name! and has no value that means no restart policy is set and the default value [no][1] is used.
So you may not only need to update the Restart Policy in the file, but also update pre created container manually:

# docker update <your-container> --restart always

So new value is changed:

# docker inspect <your-container> | fgrep -i restart -A 5
        "RestartCount": 0,
--
            "RestartPolicy": {
                "Name": "always",
                "MaximumRetryCount": 0
            },

Just this :(
[1]: https://docs.docker.com/config/containers/start-containers-automatically/

Siclari

exactly my case. Embarrassing is the RestartPolicy is not updated even if I run docker-compose restart. I expected Docker-compose to re-read the configuration file and apply the new config but this does not happen! Weird. – Kilowatt

Answer 3

always Always restart the container if it stops. If it is manually stopped, it is restarted only when the Docker daemon restarts, or the container itself is manually restarted.

unless-stopped Similar to always, except that when the container is stopped (manually or otherwise), it is not restarted even after the Docker daemon restarts.

If you had an already running container that you wanted to change the restart policy for, you could use the docker update command to change that, and the below command will ensure all currently running containers will be restarted unless stopped

$ docker update --restart unless-stopped $(docker ps -q)

NOTE: Keep the following in mind when using restart policies

1. A restart policy only takes effect after a container starts successfully. In this case, starting successfully means that the container is up for at least 10 seconds and Docker has started monitoring it. This prevents a container that does not start at all from going into a restart loop.

2. If you manually stop a container, its restart policy is ignored until the Docker daemon restarts, or the container is manually restarted. This is another attempt to prevent a restart loop.

3. Restart policies only apply to containers. Restart policies for swarm services are configured differently

Documentation

Nesmith

Ok, so help me out here. As you can see in my question, I have restart: always turned on. All these containers are started successfully and are running well. Then the OS updates itself automatically and restarts itself. After this restart the docker daemon is restarted. But for some reasons the containers I had running WITH RESTART: ALWAYS turned on DOES NOT START. If I enter my server at this moment, type sudo docker ps to list my running containers, suddenly all containers are booted up and I see the list. So why wasn't the containers started, even though the daemon is running? – Runner

I changed the policy to restart always but I think docker-compose stop/start do not reload those settings. This command did the trick for me – Reside

Answer 4

This requires the Docker service to get started on boot instead of using the default socket activation that starts on-demand like you decribed with execution of "docker ps"

Here is the required Container Linux Config to enable the Docker service while disabling socket activation:

systemd:
  units:
    # Ensure docker starts automatically instead of being socket-activated
    - name: docker.socket
      enabled: false
    - name: docker.service
      enabled: true

Dimorphism

What is this? Where does it go? What file needs to be updated? Something on the system or something in a compose file? – Idempotent

On Flatcar Container Linux, unit files are located at /etc/systemd/system. You have to create a simple unit file (e.g. docker.service) including the lines above. – Dimorphism

Answer 5

It's container restart policy. restart: always restart the container if it stops. If it is manually stopped, it is restarted only when Docker daemon restarts or the container itself is manually restarted.Please check this link restart_policy.

Baerman

I don't see how this is helpful. As coreos updates itself automatically I don't manually turn off the containers. Once the system has rebooted the containers don't start again for some reason, even though the Docker daemon is started. – Runner