Solving a Docker compose scenario involving dependant resources shared by containers?

Question

I'm learning Docker and I'm facing the following issue, mainly related to sharing files between services. I've prepared an illustration:

I have 3 stages in my Dockerfile: builder, php and caddy. My docker-compose.yml defines 2 services: php and caddy (the target is the correspoding stage in the Dockerfile). Requirements:

1. builder stage produces some files needed by php service at runtime (i.e. public/build/metatada.json)
2. builder stage produces some files needed by caddy service at runtime (i.e. public/js/foo.js)
3. php stage produces some files that should be available to caddy service at runtime (i.e. public/bundles/foo.js)
4. caddy service must share public/ folder with php service, because new files can be created (i.e. user public/uploads/newfile.pdf)
5. New static files in source code must be taken into account (i.e. public/img/newimage.png)

Solution 1: my first naive solution won't work because point 4: caddy service can't access resources created by the php services at runtime (i.e. public/uploads/new.png):

FROM node:current-alpine as builder
WORKDIR /app
COPY . .
RUN npm run build

FROM php:fpm-latest as php
COPY . .
COPY --from=builder /app/public/ public/

FROM caddy:latest as caddy
WORKDIR /srv
COPY --from=builder /app/public/ public/

Solution 2: bind mount source code public folder to caddy /srv/public and php /var/www/html/public folder. This solves 4 and 5, but completely wipes the builder output so points 1, 2, 3 are not respected.

Which other option I have? I already tried to share a volume (see my other SO question) but it won't work either.

Nickeliferous

What about adding a dependency from php and caddy to builder? This is supposed to wait for the container to be up before starting the others .. Plus it is weird that bind mount wipes files. Can you post the command/volume definition for mounting? – Legislate

If you can access Internet, try cloud object storage service like amazon s3 – Reeducate

Why use bind mount instead of mount? Seems you can share one folder(public/) to all the containers and containers can access all files in that folder. – Reeducate

@Hicomputer @Legislate I cannot use a bind mount the public folder because during the build my images will populate sub-folders of public folder. On startup, the local content will replace the one populated by imaes. That's how bind-mount works. – Nickeliferous

You'll need to find a directory structure that allows you to bind mount the necessary empty document (uploads) folders, while the static assets remain baked in your image. And regarding the file access problem, there may be a mismatch of UIDs used by the various involved processes, and it's possible to fix that by overriding configuration files or even just adding user/group to the compose file. – Astolat

Disk dependency makes your app not portable in modern architectures. Even WordPress and Drupal use plugins to manage the files outside with some service like aws s3, azure storage, etc – Wilser

as @Wilser said, although I dunno about this plugins otherwise, you can still do bind mount but right after compose build and before compose up , have a script (I use Makefile) to delete and recreate the public folder for the permissions issue, I already have some resuable commands to create www-data or similar (dpeend on distro) and assign them to same GID/UID ensuring they all can read/write/access the needed resources – Hellion

Answer 1

Forgive me if I'm also being naive with this approach, I have read your other question as well. Just a suggestion to try.

Could you not keep both public folder's contents elsewhere in the container, use solution 2 and load the container's public folder contents after boot using the entrypoint script in each container, if a certain file can't be found? That way nothing is obscured by the bind and you still have all the files.

Something like:

entrypoint.sh

...
while [ ! -e "/app/public/bundles/foo.js" ];do
    cp -r /apptmp/public/* /app/public/
    break;
done
...

Where /apptmp/public contains the files generated during the build process and /app/public is the mounted directory.

(I used while but if is fine too).

EDIT: If the files are needed at runtime you can script a wait into the entrypoints as well.

Autoeroticism