How to setup docker to use cache from registry on every build step

Question

I have two servers with docker and one server with my private registry.

I built Dockerfile on the first machine; then I pushed the image to the registry.

Is it possible to build Dockerfile on the second machine immediately using cache from my registry? If no, is there any way to speed up building "almost" same Dockerfiles without writing my own cache?

It tried to setup --registry-mirror but it didn't help.

Serranid

Did you tried this? blog.docker.com/2015/10/registry-proxy-cache-docker-open-source – Kier

@JosipIvic Yes but builds ignore layers pulled from the private registry anyway. – Third

Answer 1

for docker > 1.10, I found something on this issue: https://github.com/docker/docker/issues/20316#issuecomment-221289631

Given this Dockerfile

FROM busybox
RUN mkdir this-is-a-test
RUN echo "hello world"

run docker build -t caching-test .

Then we can see the layers comprising the image with docker history caching-test

3e4a484f0e67        About an hour ago   /bin/sh -c echo "Hello world!"                  0 B                 
6258cdec0c4b        About an hour ago   /bin/sh -c mkdir this-is-a-test                 0 B                 
47bcc53f74dc        9 weeks ago         /bin/sh -c #(nop) CMD ["sh"]                    0 B                 
<missing>           9 weeks ago         /bin/sh -c #(nop) ADD file:47ca6e777c36a4cfff   1.113 MB  

The change to save/load in 1.11 preserves the relationship between parent and child layers, but only when they are saved via docker save together. We can see the parent of the final test image by running docker inspect test | grep Parent.

$ docker inspect caching-test | grep Parent
"Parent": "sha256:6258cdec0c4bef5e5627f301b541555883e6c4b385d0798a7763cb191168ce09", 

This is the second-to-top layer from our Docker history output.

In order to recreate the cache using save and load, you need to save out all of the images and layers that are referenced as parents. In practice this typically means that you need to save each layer, as well as the FROM image, in the same command.

docker save caching-test 6258cdec0c4b busybox > caching-test.tar -- note that we can also give the layer names instead of IDs to the save command.

Let's purge everything and then reload the image from the tar file. docker rmi $(docker images -q). Confirm that no images exist.

Then run docker load -i caching-test.tar. If you look at the images, you'll see busybox, and then caching-test. Running docker history caching-test will show you the exact same output as when the image was initially built. This is because the parent/child relationships were preserved via save and load. You can even run docker inspect caching-test | grep Parent and see the exact same ID given as the parent layer.

And running a rebuild of the same Dockerfile will show you that the cache is being used.

Sending build context to Docker daemon 5.391 MB
Step 1 : FROM busybox
 ---> 47bcc53f74dc
Step 2 : RUN mkdir this-is-a-test
 ---> Using cache
 ---> 6258cdec0c4b
Step 3 : RUN echo "hello world"
 ---> Using cache
 ---> 3e4a484f0e67
Successfully built 3e4a484f0e67

EDIT: Below this works only Before docker 1.10

On the second machine you can docker pull theimagefromthefirstdockerfileontheregistry before building the new one.

That way you are sure that every layer is present on the second machine.

The docker-engine doesnt query the registry each time a layer is built (it doesn't even knows it), it would be too slow/heavy so I dont think there is another way.

Arana

Sorry but this is not what I observe on Docker 1.11. Pulling an image does not pull every layers of it. docker history shows every layers except the last one as <missing> and docker build rebuilds from scratch. – Third

@Third Wow, you are right, I dont know when this behavior appeared, but I'm pretty sure that what I said was right not so long ago. I'll investigate – Arana

Indeed I found this post too. I need docker > 1.9.1 so I can't rely on the old behavior. Can you update your answer to precise this only works prior to Docker 1.10 ? – Third

I added a workaround found on another issue, I tested and it works, even if it's more cumbersome – Arana

Thanks for your answer. I ended up doing this in my CI environment. In addition to your answer : It is possible to save all available layers for an image with this command docker save IMAGE_NAME $(docker history -q IMAGE_NAME | tail -n +2 | grep -v \<missing\> | tr '\n' ' ') > TAR_FILE_NAME.tar. To conclude, this workaround works even though I have to store really big artifacts (all layers must be saved and this method does not allow to reuse pushed layers like a registry would). I hope that in the future Docker will address this issue. – Third

Answer 2

Note: issue 20316 ("Pulling build cache") has been closed because PR 26839 ("Implement build cache based on history array") has been merged.

It allows for instance to specify in --cache-from the image from a previous CI build.

Adds capability to specify images used as a cache source on build. These images do not need to have local parent chain and can be pulled from other registries. User needs to make sure to only use trusted images as sources.

Usage:

docker pull myimage:v1.0
docker build --cache-from myimage:v1.0 -t myimage:v1.1 .

See merge commit 7944480, for docker 1.13 (January 2017).

As commented by javipolo:

In case someone is going nuts with reusing layers as I did, the "trick" is to pass to --cache-from the image you are rebuilding (and have it pulled already) and ALSO the image that it uses as base in the FROM.

Example:
Dockerfile for image custom-gource:0.1

FROM base_image:2.2.1
RUN apt-get update && apt-get install gource
COPY myscript.sh /myscript.sh

In order to rebuild in other host without doing the apt-get again, you'll need to:

docker pull custom-gource:0.1
docker build --cache-from=base_image:2.2.1,custom-gource:0.1 . -t custom-gource:0.2

It might seem too obvious but I've been struggling long time with this until I got that you need to include the base image too

Crabbing

Any idea when this is landing in docker compose as well ? – Minaret

@Minaret No, I do not. – Crabbing

When I enter docker --version I get 17 but when i enter docker-compose --version I get 1.14. can you please clarify, – Garik

@Garik the docker daemon and the docker client are not the same (a docker info should display that) – Crabbing

I really don't know the diefference right now but when i entered docker info they only mentioned that the server version is 17 – Garik

@Garik It depends on your exact setup (OS for the host, OS where docker is running). See for instance github.com/docker/machine/issues/2147#issuecomment-222871384. But more importantly: ask a new question. – Crabbing

ok. i will erase these comments. can you please help with this: #45579153 – Garik

Answer 3

For Docker versions >= 19.03, you can use the new BuildKit features to avoid having to pull the remote image before building. Activate BuildKit by setting the DOCKER_BUILDKIT enviroment variable, then turn on inline caching to store the build cache along with your image in your registry using the BUILDKIT_INLINE_CACHE build argument, as follows:

export DOCKER_BUILDKIT=1
docker build -t registry/imagename:tag --cache-from registry/imagename:tag --build-arg BUILDKIT_INLINE_CACHE=1 .
docker push registry/imagename:tag

Thanks to Nicholas Dille for this solution.

Penelopa

Answer 4

Also you can switch to google solution kaniko

https://cloud.google.com/build/docs/kaniko-cache#kaniko-build

Aylmar