How to cache package manager downloads for docker builds?

Question

If I run composer install from my host, I hit my local composer cache:

  - Installing deft/iso3166-utility (1.0.0)
    Loading from cache

Yet when building a container having in its Dockerfile:

RUN composer install -n -o --no-dev

I download all the things, e.g.:

  - Installing deft/iso3166-utility (1.0.0)
    Downloading: 100%         

It's expected, yet I like to avoid it. As even on a rebuilt, it would also download everything again.

I would like to have a universal cache for composer that I could also reshare for other docker projects.

I looked into this and found the approach to define a volume in the Dockerfile:

ENV COMPOSER_HOME=/var/composer
VOLUME /var/composer

I added that to my Dockerfile, and expected to only download the files once, and hit the cache afterwards.

Yet when I modify my composer, e.g. remove the -o flag, and rerun docker build ., I expected to hit the cache on build, yet I still download the vendors again.

How are volumes supposed to work to have a data cache inside a docker container?

Auston

The volume will persist for a container instance, but not on build. You could create a named volume that will persist in a static location but then you are coupling a solution to an environment. You need to think about what you want to do in the build vs after. You could set up some kind of local repository to act as a cache/proxy. – Formularize

I'm struggling with this as well. Did you figure out a way out of this? I thought maybe using an HTTP proxy and directing Docker to use that could work. Potentially excluding hub.docker.com from the proxy to not keep two copies of downloaded images. – Zakaria

@gooli Posted an answer on how I deal with the problem. Thx for the reminder. – Auston

Answer 1

Use the experimental feature : Docker buildkit (Supported Since docker 18.09, docker-compose 1.25.4)

In your dockerfile

# syntax=docker/dockerfile:experimental
FROM ....
# ......  
RUN --mount=type=cache,target=/var/composer composer install -n -o --no-dev

Now before building, make sure the env var is exported:

export DOCKER_BUILDKIT=1
docker build ....

If you are using docker-compose, make sure to export also COMPOSE_DOCKER_CLI_BUILD :

export COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1
docker-compose build ...

If it does not work with docker-compose, make sure your docker-compose version is above 1.25.4

docker-compose version

Uproar

What's the min version of the docker engine to use this experimental feature? – Auston

18.09 .. i will add it to the answer – Uproar

And instead of exporting the var, one can also include it in the command itself: DOCKER_BUILDKIT=1 docker build . – Auston

@Auston 😁 I am a Linux superman .. Or you can make alias . .or .. and or .. but the idea is that we are answering questions for the world wide. Considering backgrounds of everyone .. that's why. – Uproar

Composer cache is located in /tmp/cache as of composer 1.10 – Plauen

Answer 2

I found two ways of dealing with this problem, yet none deal with composer volumes anymore.

1. Fasten composer download process: Use hirak/prestissimo

   composer global require "hirak/prestissimo:^0.3"
   

💡 With Composer 2.0, the above step is no longer required for faster downloads. In fact, it won't install on Composer 2.0 environments.

2. Force docker to use a cached composer install.
   Docker uses a cache on a RUN if the added files didn't change. If you only do COPY . /your-php-app, docker build will refresh all the cashes and re-run composer install even if only one unrelated file in the source tree changed.
   In order to make docker build to run composer install only install on package changes, one has to add composer.json and composer.lock file before adding the source files. Since one also needs the source files anyway, one has to use different folders for composer install and rsync the content back to the then added folder; furthermore one then has to run the post-install scripts manually.
   It should look something like this (untested):

   WORKDIR /tmp/
   COPY composer.json composer.lock ./
   RUN composer install -n -o --no-dev --no-scripts
   
   WORKDIR /your-php-app/ 
   COPY . /your-php-app/
   RUN rsync -ah /tmp/* /your/php-app/
   RUN composer run-script post-install-cmd
   

or combine the two =)

Auston

This answer is quite strange... it seems to bypass the point of having a bind mount from your local composer cache to your image composer cache. What is happening now (from what I can guess) is that your vendor/ dir is completely copied and therefore composer doesn't need to re-download (or install) anything. This might have cross-platform consequences though. – Khartoum

@Khartoum What kind of cross-platform issues are you thinking about? I use this Dockerfile both for development and production. – Auston

If you use some composer package that uses a platform-specific executable, from experience, composer is not intelligent enough to realize that the package should be reinstalled. – Khartoum

Anyway, to answer your (implied) question above, it seems that docker compose (and maybe docker?) somehow does not mount volume binds during build time. An option that I know that works is to use something like GitLab CI instead of building with a plain docker-compose. – Khartoum

Instead of using temp dir and using rsync to align the image, I would rather use a multistaged build when able to use docker 17.05 or higher. – Auston

Although this answer doesn't solve the question posted exactly, I have to say prestissimo is awesome! – Jordanson

Answer 3

Anyone looking for this from now should be able to use

RUN --mount=type=cache,target=/root/.composer/ composer install

This general principle seems to work for most package installers, you just need to find the right cache dir (npm uses $HOME/.npm, etc.), and it will obviously change if you are not building as root.

Bolme

Answer 4

I would like to have a universal cache for composer that I could also reshare for other docker projects.

Using a shared volume for the Composer cache works great when working with containers. If you want to go broader than just containers, and use a shared cache for e.g. local development as well, I've developed a solution for that called Velocita - how it works.

Basically, you use one global Composer plugin for local projects and inside and build containers. This not only speeds up downloads tremendously, it also helps with 3rd party outage for example.

Circumcise

Answer 5

I would consider utilizing the $HOME/.composer/cache/files directory. This is where composer reads/write to when using composer install.

If you are able to mount it from your host to your container that would work. Also you could just tar it up after each time your run composer install and then drop that in before you run composer install the next time.

This is loosely how Travis CI recommends doing this.

Also, consider using the --prefer-dist flag with your composer install command.

Info on that can be found here: https://getcomposer.org/doc/03-cli.md#install

--prefer-dist: Reverse of --prefer-source, composer will install from dist if possible. This can speed up installs substantially on build servers and other use cases where you typically do not run updates of the vendors. It is also a way to circumvent problems with git if you do not have a proper setup.

Some references on utilizing the composer cache for you:

• https://blog.wyrihaximus.net/2015/07/composer-cache-on-travis/
• https://github.com/travis-ci/travis-ci/issues/4579

Weighted