Docker and symlinks

Question

I've got a repo set up like this:

/config
   config.json
/worker-a
   Dockerfile
   <symlink to config.json>
   /code
/worker-b
   Dockerfile
   <symlink to config.json>
   /code

However, building the images fails, because Docker can't handle the symlinks. I should mention my project is far more complicated than this, so restructuring directories isn't a great option. How do I deal with this situation?

Pythagorean

Answer 1

Docker doesn't support symlinking files outside the build context.

Here are some different methods for using a shared file in a container:

---

Build Time

Copy from a config image (Docker buildkit)

Recent versions of Docker allow RUN steps to bind mount from a named image or previous build stage with the --mount=type=bind,target=/dir,source=/dir,from=image-or-stage-name

Create a Dockerfile for the base me/worker-config image that includes the shared config/files.

FROM scratch
COPY config.json /config.json

Build and tag the config image me/worker-config

docker build -t me/worker-config:latest .

Mount the me/worker-config image during the real build

RUN --mount=type=bind,target=/worker-config,source=/,from=me/worker-config:latest \
    cp /worker-config/config.json /app/config.json;

Share a base image

Create a Dockerfile for the base me/worker-config image that includes the shared config/files.

COPY config.json /config.json

Build and tag the image me/worker-config

docker build -t me/worker-config:latest .

Source the base me/worker-config image for all your worker Dockerfiles

FROM me/worker-config:latest

Build script

Use a script to push the common config to each of your worker containers.

./build worker-n

#!/bin/sh
set -uex 
rundir=$(readlink -f "${0%/*}")
container=$(shift)
cd "$rundir/$container"
cp ../config/config.json ./config-docker.json
docker build "$@" .

Build from URL

Pull the config from a common URL for all worker-n builds.

ADD http://somehost/config.json /

Increase the scope of the image build context

Include the symlink target files in the build context by building from a parent directory that includes both the shared files and specific container files.

cd ..
docker build -f worker-a/Dockerfile .

All the source paths you reference in a Dockerfile must also change to match the new build context:

COPY workerathing /app

becomes

COPY worker-a/workerathing /app

Using this method can make all build contexts large if you have one large build context, as they all become shared. It can slow down builds, especially to remote Docker build servers. Note that only the .dockerignore file from the base of the build context is referenced.

Alternate build that can mount volumes

Other projects that strive for Dockerfile compatibility may support volumes at build time. For example a podman build / buildah support a --volume option to bind mount files from the host into a build container.

podman build --volume /project/config:/worker-config:ro,Z -t me/worker-a .

Then the build can reference the mounted volume

COPY /worker-config/config.json /app

---

Run time

Mount a config directory from a named volume

Volumes like this only work as directories, so you can't specify a file like you could when mounting a file from the host to container.

docker volume create --name=worker-cfg-vol
docker run -v worker-cfg-vol:/config worker-config cp config.json /config

docker run -v worker-cfg-vol:/config:/config worker-a

Mount config directory from data container

Again, directories only as it's basically the same as above. This will automatically copy files from the destination directory into the newly created shared volume though.

docker create --name wcc -v /config worker-config /bin/true
docker run --volumes-from wcc worker-a

Mount config file from host at runtime

docker run -v /app/config/config.json:/config.json worker-a

Sakmar

Although it is a very comprehensive answer, still sad to see it since docker was supposed to simplify these things for developers so they could do their real job... – Gelya

For my use case, "Increase the scope of the image build context" is the best solution. Unfortunately, this breaks .dockerignore which can result in huge container images that contain extraneous code from other directories in the increased scope (e.g. parent directory). I've been searching for a few hours, but there doesn't seem to be a good solution for this. 😕 – Dissoluble

Related: github.com/docker/compose/issues/5523 – Dissoluble

@Dissoluble interesting, I had assumed it was like .gitignore in sub dirs. There's also this workaround and this proposal – Sakmar

@Dissoluble Now you can specify location of dockerignore. Check details here (accepted answer). Some say they had problems setting it up but I can confirm it works for me. – Elbe

Answer 2

Node.js-specific solution

I also ran into this problem, and would like to share another method that hasn't been mentioned above. Instead of using npm link in my Dockerfile, I used yalc.

1. Install yalc in your container, e.g. RUN npm i -g yalc.
2. Build your library in Docker, and run yalc publish (add the --private flag if your shared lib is private). This will 'publish' your library locally.
3. Run yalc add my-lib in each repo that would normally use npm link before running npm install. It will create a local .yalc folder in your Docker container, create a symlink in node_modules that works inside Docker to this folder, and rewrite your package.json to refer to this folder too, so you can safely run install.
4. Optionally, if you do a two stage build, make sure that you also copy the .yalc folder to your final image.

Below an example Dockerfile, assuming you have a mono repository with three packages: models, gui and server, and the models repository must be shared and named my-models.

# You can access the container using:
#   docker run -it my-name sh
# To start it stand-alone:
#   docker run -it -p 8888:3000 my-name

FROM node:alpine AS builder
# Install yalc globally (the apk add... line is only needed if your installation requires it)
RUN apk add --no-cache --virtual .gyp python make g++ && \
  npm i -g yalc
RUN mkdir /packages && \
  mkdir /packages/models && \
  mkdir /packages/gui && \
  mkdir /packages/server
COPY ./packages/models /packages/models
WORKDIR /packages/models
RUN npm install && \
  npm run build && \
  yalc publish --private
COPY ./packages/gui /packages/gui
WORKDIR /packages/gui
RUN yalc add my-models && \
  npm install && \
  npm run build
COPY ./packages/server /packages/server
WORKDIR /packages/server
RUN yalc add my-models && \
  npm install && \
  npm run build

FROM node:alpine
RUN mkdir -p /app
COPY --from=builder /packages/server/package.json /app/package.json
COPY --from=builder /packages/server/dist /app/dist
# Make sure you copy the yalc registry too.
COPY --from=builder /packages/server/.yalc /app/.yalc
COPY --from=builder /packages/server/node_modules /app/node_modules
COPY --from=builder /packages/gui/dist /app/dist/public
WORKDIR /app
EXPOSE 3000
CMD ["node", "./dist/index.js"]

Hope that helps...

Odisodium

yalc seems interesting, but is it really any better than just running npm pack? Seems like the same number of steps. I'm not sure what yalc's lockfile does. – Serotonin

Answer 3

The docker build CLI command sends the specified directory (typically .) as the "build context" to the Docker Engine (daemon). Instead of specifying the build context as /worker-a, specify the build context as the root directory, and use the -f argument to specify the path to the Dockerfile in one of the child directories.

docker build -f worker-a/Dockerfile .
docker build -f worker-b/Dockerfile .

You'll have to rework your Dockerfiles slightly, to point them to ../config/config.json, but that is pretty trivial to fix.

Also check out this question/answer, which I think addresses the exact same problem that you're experiencing.

How to include files outside of Docker's build context?

Hope this helps! Cheers

Infinitesimal

Unfortunately, running docker from the root of the repo means my builds become significantly more cumbersome, as they have to copy over 250 meg of files. I can't .dockerignore them, because some of them are needed for other docker builds. – Pythagorean

Hmmm I understand that concern. Still might be worth a shot though, depending on where your Docker Engines are (eg. local, local data center, or cloud). The other best option I could think of would be to copy the config.json into each project folder. – Infinitesimal

Keeping 7 different config files in sync also seems pretty dumb. As does writing build scripts which copy the config files into the directory before the build happens. This is a pretty stupid restriction on the part of the Docker developers. – Pythagorean

You might get more traction if you post in the existing GitHub threads about symlinks. – Infinitesimal

Also, this solution doesn't turn out to work at all, without copying the full directory structure into the repo. You get symlinks which point to files which don't exist. – Pythagorean

I'm running into this issue right now. For various reasons, a base image won't work for me (because I really have several base images to combine). I think what I may end up doing is have a Makefile orchestrate copying the "shared" files into the build directory. – Catnap

Answer 4

An alternative solution is to upgrade all your soft links into hard links.

Loesceke

For many of us, the point of symlinks is that they can point to separate physical devices such as NFS. So this suggestion is not feasible for the use case where a symlink would be most critical (ie, where your data is huge and you therefore don't want to move it). – Labored

@Labored Absolutely true. However, in my situation I just needed a link to a script that was supposed to be baked into two containers. Hence +1 for your comment but also +1 for Benjamin Pastel :-) – Roley

@LaryxDecidua nice! That is good news :-) – Labored

This does appear to be the best solution we have right now. To help ease the setup process for those hard-links, I created a small file-syncer utility: stackoverflow.com/a/68765508 – Milburn

Bind mounts are another alternative to hard links, but annoyingly they would need to be in fstab or run once per boot. Would probably work ok in a build script. mount --bind /sourcefolder /dest/folder – Serotonin

@Serotonin You had me excited, but mount --bind isn't available on macOS without doing a bunch of stuff: setup, running services, or installing things. Sadly. – Manon

@Manon that is unfortunate. What about a loopback mount? – Serotonin

@Serotonin The issue isn't really compatibility or feasibility but complexity. Having extra commands and setup needed on local developer workstations, especially if there would be cleanup to remove resources, are problematic for the use case. I found something that works using BuildKit (already in Docker Desktop) and the RUN --mount syntax for secrets and for caches. It's not ideal that I can't pre-load the npm package cache with what's on the host, but that's more of an optimization, and the penalty is low, there's a second cache inside Docker that will keep things fast for subsequent runs. – Manon