Docker: Exporting Image for Multiple Architectures

Question

I have a scenario where I need to bundle a multi-container system in an exportable, transportable format. Specifically, I need to get an image from Docker Hub and store it on portable media (e.g. USB drive) that I can then load into an air-gapped machine with no internet access.

I can normally do this with:

# On the development server
docker pull nginx:1.23
docker save -o nginx.tar nginx:1.23

... store the .tar on portable media...

# On the air-gapped machine
docker load < myimage.tar

However, although the image is built for and supports multiple architectures, this appears to only store the image for a single architecture (the architecture of the machine that ran the docker pull). I need it to export images for all architectures to .tar files, so they can be loaded on machines of any architecture.

I've tried:

docker pull --platform linux/amd64 nginx:1.23
docker pull --platform linux/arm64 nginx:1.23

To manually specify that both the amd64 and arm64 images should be pulled, but the docker save still exports only a single image (which we can tell from the fact that the manifest.json file only has a single entry).

I've also tried:

docker pull --platform linux/amd64 --platform linux/arm64 nginx:1.23

But it has the same result.

How can I export/save an image with support for multiple different architectures when I docker load it? Ideally, I'd be able to docker load a single .tar file on a machine of any supported architecture, have it import the image for that architecture, then run the image.

EDIT:

After some further testing, I've found that if I do the following:

docker pull --platform linux/amd64 nginx:1.23
docker save -o nginx.tar

Then the .tar file has a manifest.json with a single image (for amd64). However, if I do this:

docker pull --platform linux/arm64 nginx:1.23
docker save -o nginx.tar

Then the .tar file has a manifest.json with two images (for amd64 and arm64).

This isn't an issue with "you have to pull both" though; if I pull the amd64 version again, and docker save again, we're back to a single image in the manifest.json. The order of pulls does appear to matter.

Disputant

Could you push the images you want to transfer to a local registry on your host machine, then backup the registry files to your USB and restore the registry on the other machine? See weblog.wemanity.com/en/… where he creates a local registry and can backup and restore it. – Anastigmat

I already have the multi-arch image in a local registry. This is an interesting idea, but I need to be able to docker load the image into Docker engine on the target machine. Specifically, I'd like to have a single .tar file with the multi-arch image, then do docker load on the target machine, and have it load the image for the target machine's architecture (from the .tar file) into the target machine's Docker engine to then be run. – Disputant

I presume/expect the registry will have the multi-arch images in it, so if you restore the registry on the target machine and then export the image you want you will get the arch-specific one you need and you can then docker load that? – Anastigmat

Yeah, theoretically I wouldn't even need to docker load, I could just do a docker pull from the local registry and it would pull the image for the correct arch. But, this solution isn't great for this use case, because it requires starting a local registry, which is a major pain on an air-gapped machine. Normally you'd do it with a registry Docker container, but then we're back at the situation of "how do you get the registry container for the right arch?". – Disputant

Answer 1

From my post on the vastly superior DevOps SE:

This is answered in the Docker blog post on multi-platform docker builds. For your specific situation, you would perform a build and push for each platform you'd like, then create a manifest file and push that. Then you can perform the save. It would look something like this:

$ docker buildx build --platform linux/amd64 --push -t localhost:5000/myimage:amd64 .
$ docker buildx build --platform linux/arm64 --push -t localhost:5000/myimage:arm64 .
$ docker manifest create myimage:latest myimage:amd64 myimage:arm64
$ docker manifest push localhost:5000/myimage:latest
$ docker image save -o myimage.tar localhost:5000/myimage:latest

Callista

I did actually try this out. The reason I didn't do it is because the docker manifest command is still listed as being experimental (even though it's been around for 4+ years now?), and I'm not super comfortable using experimental tools in this production build system. Is this a valid concern, or is it really not actually experimental? – Disputant

"Experimental" or "unstable" features that have been in the pipeline for years on end are safe in my experience. If you test it and everything works reliably then nothing should break until something else changes. – Callista

Alright, I'll give it a shot and hope they don't change the specs when it's finalized. Fingers crossed... :) – Disputant

Is the saved image multiarch or just the arch for the current platform? – Rodrigues

This fails for me by the time I get to the docker image save command with the error Error response from daemon: reference does not exist. As far as I can tell the image save command can only save from an installed image, not from a registry (localhost:5000/myimage:latest). How could it have worked for you? Am I missing something? – Tamathatamaulipas

Why "vastly superior"? – Kiss

Answer 2

How can I export/save an image with support for multiple different architectures when I docker load it? Ideally, I'd be able to docker load a single .tar file on a machine of any supported architecture, have it import the image for that architecture, then run the image.

At present, you can't. The docker engine currently only supports images with a single platform, so when you pull an image, it gets dereferenced to your selected platform. This is changing soon with how docker uses the containerd backend, since containerd supports multi-platform images. Until then, the two places you can store a multi-platform image are using a registry and using an OCI Layout syntax for storing the image on disk (that format is a little different than the format docker save and load uses).

For copying between registries and/or the OCI layout, you don't need the docker engine. Various tools support this, including Google's crane, RedHat's skopeo, and my own regclient. E.g. for an isolated environment where you import tar files:

regctl image export nginx:1.23 nginx.tar
# copy tar image to host in isolated environment
regctl image import internal-registry.example.org/library/nginx nginx.tar
docker pull internal-registry.example.org/library/nginx

Carbonaceous

Upvoted. regctl (that I mentioned before here) is really awesome. Thank you for that tool. – Byproduct