Prevent Docker-Engine restart during apt-get upgrade

Question

I've got an issue here with Ubuntu 14.04

Whenever I do an apt-get upgrade, if docker-engine is eligible for upgrade, it restarts the service after upgrading. This causes the containers to get restarted.

One of the containers is one that requires manual intervention during startup, others are gathering time-critical data, so that a restart of the containers is only desired at very specific times.

How can I exclude the docker-engine service from these automatic restarts?

AFAIK, mongodb, for instance, doesn't restart mongod after an apt-get upgrade, why can't docker behave the same way?

Horatius

Answer 1

I am not entirely sure what mongodb does but I am pretty sure it does restart itself on update.

The problem with docker-engine is not that it doesn't restart itself, it actually does, which you can verify by running the following command after update:

sudo service docker status

It's just that it doesn't restart the running containers. I don't exactly know why it has to be like this because it could in theory be able to bring back the running containers if the service is restarted but for whatever reason, it doesn't do it.

There are two ways to workaround this issue.

The first and easy way it to exclude the update of docker-engine and do the update manually if you see it is available.

sudo apt-mark hold docker-engine
sudo apt-get upgrade

You should see a notice if there was an update and if the update was kept back

...
The following packages were kept back:
docker-engine
...

If you have containers that you cannot afford to restart at upgrade, you should go this route.

If however, you can afford to have them restarted, you can use the second approach. You can set restart policies when you start the containers, like so:

sudo docker run --restart=always -d image-name

This way, when docker-engine is restarted, the container is started with it. This means that on update, the container started this way gets started automatically.

Marienthal

Ah, ok, I forgot about the not restarting issue because I was then shutting down everything manually before an upgrade, and then doing a reboot which executes a script which starts all containers. I think it's the sudo apt-mark docker-machine hold which I want. Is it docker-machine or docker-engine? – Horatius

Answer 2

The right way to do this appears to be this live-restore option for the docker daemon:

https://docs.docker.com/config/containers/live-restore/

For Linux docker servers (as you seem to be running here), you can enable live-restore for the daemon with the following in /etc/docker/daemon.json:

{
  "live-restore": true
}

Then run this to reload the configuration without restarting any containers:

sudo service docker reload

Once done, your apt upgrade should not take down any containers as it reloads the daemon for an upgrade, and should not cause any issues, as long as the upgrade is not a major release.

That last caveat is important, as upgrading the docker daemon over a major release boundary while containers are running could make it unable to reattach to the containers, meaning you would need to restart them manually. I would plan to watch for the new version and for any major release, plan for some downtime and stop the containers before upgrading.

In general, I feel you should be more aware and vigilant about your versioning and configuration if you have this version on, as the synchronization of versions and config will not be handled for you with upgrades.

But for standard upgrades to docker-ce and associated packages, which appear to happen very frequently, this has been working well for me so far.

Byelection

Answer 3

That's a design of the pre/post install scripts that come in the deb package and a decision of whoever builds the package (though pulling apart the latest jessie build on apt.dockerproject.org, I'm not seeing the stop inside of their control files, only the register and start).

Best suggestion I can make is to not run the system upgrade when you can't tolerate any downtime. Minimize the time for the upgrade by doing a download only (apt-get upgrade -d) before, and then kick off the upgrade during your outage window. You can also place the docker-engine package on hold so that it isn't automatically upgraded using dselect and pressing = on the package name.

Amaliaamalie