Why are Docker multi-architecture needed (instead of the Docker Engine abstracting the differences)

Question

Short version

I would like to know the technical reasons why do Docker images need to be created for multiple architectures. Also, it is not clear whether the point here is creating an image for each CPU architecture or for an OS. Shouldn't the OS abstract the architecture?

Long version

I can understand why the Docker Engine must be ported to multiple architectures. It is a piece of software that will interact with the OS, make system calls, and ultimately it is just code that is represented as a sequence of instructions within a particular instruction set, for a particular architecture. So the Docker Engine must be ported to multiple OS/architectures much like, let's say, Microsoft Word would have to be ported.

The same thing would occur to - let's say - the JVM, or to VirtualBox.

But, different than with Docker, software written for the JVM on Windows would run on Linux. The JVM would abstract the differences of the underlying OS/architectures, and run the same code on both platforms.

Why isn't that the case with Docker images? Why can't the Docker Engine just abstract the differences, and provide a common interface, so the image itself wouldn't need to be compatible with a specific OS/architecture?

Is this a decision (like "let's make different images per architecture because it is better for reason X"), or a consequence of how Docker works (like "we need to do it this way because Docker requires Y")?

Note

• I'm not crying "omg, why??". This is not a rant or criticism, I'm just looking for a technical explanation for the need of different images for different architectures.
• I'm not asking how to create a multi-architecture image.
• I'm not looking for an answer like "multi-architecture images are needed so you can run your images on various platforms", which answers "what for?", but not "why is that needed?" (which is my question).

Besides that, when you see an image, it usually has an os/arch in the digest, like this:

What exactly the image is targeting? The OS, the architecture, or both? Shouldn't the OS abstract the underlying architecture?

---

edit: I'm starting to assume that the need for different images per architecture is on the lines of: the image will contain applications inside it. Let's say, it will contain the Go compiler. The Go compiler itself is a binary that must have been complied to different architectures. The image for x86-64 will contain the Go compiler compiled to x86-64, and so on. Is this correct? If this is correct, is this the only reason?

Mariellamarielle

there can be a big difference using the same base image for different architectures in the name of packages you install. example centos 7: the is a package called llvm-toolset-7 on intel and it is called llvm-toolset-7.0 on ppc64le. and this happens often. some repos do not even have packages for a specific architecture. it's an impossible task to abstract that. – Supply

Thanks, but I still don't get the actual difference. You provided an answer talking about the names of packages for different architectures, but I don't know how does this relates to the operating system (does the operating system downloads different packages depending on the underlying architecture?), and what are actually the difference between the packages for different architectures (besides the name) – Mariellamarielle

Docker is not a VM, it just configures the OS so that a program is: a) constrained (cgroups); b) isolated (namespaces + chroot + layered fs). A container run inside the host OS directly. Different hosts have different CPUs. Docker doesn't sit between the CPU (or any other hardware) and the container. – Celestyna

Thanks @MargaretBloom, that starts to answer my question. "Docker is not a VM" -> that is Docker 101, I think the level of my question should imply that I already know that. "Docker doesn't sit between the CPU" -> that is better information. Still though: what there is in the image itself that it needs to be ported to different OS/architecture? Are we talking about the compiled programs that the image might contain, or something else? And which one is the compatibility target here: the OS or the CPU? Shouldn't the OS abstract the CPU (honest question, don't know how it should be)? – Mariellamarielle

@RafaelEyng The compiled programs, of course. You compile a program for a specific CPU. That's why Java and CIL (.NET) bytecode is... bytecode and not machine code. Docker only supports Linux (it runs a Linux VM on Win and Mac) and while some features are provided by the docker images, the kernel seen by a container is the host kernel. So that's another source of possible incompatibility. The OS doesn't abstract the CPU as it is what run the os in the first place. Not even VMs totally abstract the CPU, emulators do. – Celestyna

@MargaretBloom nice answer, I think I'm getting there! So, if the OS does not abstract the CPU, why exactly do we need different versions for different operating systems running on the same CPU? Is it because of the way syscalls of different OSes translate to CPU instructions? But still, if the underlying CPU is the same, it should work, given the instruction set is unchanged. So if the OS does not abstract the CPU, I don't understand why different OSes need different binaries. Could you elaborate on that? And also, why on Docker multi arch the specify both the arch and the OS (why both)? – Mariellamarielle

Hi, I totally agree with @MargaretBloom. I think you should make your own answer instead of just comments. Docker focuses on one specific task and if you need to homogenize your machines you require another solution: virtualization. Maybe both. Notice that they don't recommend doing this in Docker docs. I imagine it's a matter of performance. – Loess

Answer 1

All tough the promise of Docker is the elimination of differences when moving software between machines, you'll still face the problem that Docker runs with the host machine's CPU architecture, which can't be crossed in Docker.

Neither Docker, nor a virtual machine, abstract a CPU to enable full cross compatibility.

Emulators do. However, if both Docker and VM's would run on Emulators, they would be less performant as they are today.

The docker buildx command and --build-arg ARCH flag leverages the advantage of the qemu emulator, emulating the full system with an architecture during a build. The downside of emulation is that it runs much slower, since it has to translate commands from one CPU architecture to another.

Bushwhacker

Answer 2

Why can't the Docker Engine just abstract the differences, and provide a common interface

Performance would be a major factor. Consider how slow Cygwin is for some things when providing a POSIX API on top of Windows by emulating some POSIX things that don't map directly to the Windows API. (e.g. fork() / exec separately, instead of CreateProcess).

And that's just source compatibility; the resulting binaries are specific to Cygwin on Windows. It's even worse if you want to do that at runtime (binary compat instead of source compat).

---

There's also the amount of complexity Docker would need to provide an efficient portable JIT-compiling VM on top of various OSes, especially across various CPU ISAs like x86-64 vs. AArch64 that don't even share common machine code.

If Docker had gone this route, it would really just be re-inventing a JVM or .NET CLR bytecode-based VM.

Or more likely, instead of reinventing that wheel, it would just use an existing VM and add image management on top of that. But then it couldn't work with native programs written in C, unless it transpiled them to Java or CLR bytecode.

Yoshida

After reading your answer, I still don't get it. "Consider how slow Cygwin" -> never used that in my life, and exactly a good detailing of the reason for that being slow is the kind of answer I need to understand this. "It's even worse if you want to do that at runtime" why? Is the reason behind it that I need to understand. I've never programmed OSes, so there is a ton going on that I actually don't understand. Just saying that something doesn't work or is "even worse" won't help me understand that. What I mean is: you are assuming too much previous knowledge in your answer. – Mariellamarielle

Also, your answer does not touch Docker images. I don't exactly understand how a Docker image is implemented, and why does it have to contain to be compatible with a specific architecture. – Mariellamarielle

@RafaelEyng: That's fair, my answer does require significant background knowledge to understand. I can't think of a simple way to explain it, though. Emulating a portable API on top of whatever each OS provides would add overhead for any case where there's a mismatch, and/or stop an application from taking advantage of the power of the native API in any case where the hypothetical portable API is more limited / simpler. If you don't want to take my word for that, you're going to need to understand how things like JVMs work on an asm level, and that's a much longer answer than I want to write – Yoshida

@RafaelEyng: I don't use Docker myself, but my understanding is that it's something like a filesystem image containing normal programs. It runs in their own namespace for PIDs and sockets, etc, but the fact is that there are native x86-64 Linux binaries (for example) running under the host kernel. This is very light weight, but of course you need a separate build for every target platform. For the same reason you need different installers for Windows vs. Linux vs. Mac: even having a common package format (docker image) doesn't solve the problem of needing different contents in the package. – Yoshida

"you're going to need to understand how things like JVMs work on an asm level" -> that is exactly what I'm trying to do, and "and that's a much longer answer than I want to write" -> that is exactly why I still can't find an answer to that. The lower I try to go on the stack, the less there are good, reachable material, and people willing to explain that to others. Must be a hell of a secret society. – Mariellamarielle

@RafaelEyng: There's plenty of info about how asm in general works, e.g. links in stackoverflow.com/tags/x86/info. You don't really need to know the details, just the fact that CPUs work by fetching bytes of machine code and decoding them as instructions (realworldtech.com/sandy-bridge)). Some programs (like a JVM) can generate machine code in RAM and then call it (JIT compiling), or for ahead-of-time compiled languages like C++, that machine code can just be in files on disk. – Yoshida

@RafaelEyng: It's not secret, it's just a lot of separate pieces of knowledge, like the fact that different OSes have different system call APIs. e.g. POSIX fork() on cygwin: cygwin.com/faq.html#faq.api.fork – Yoshida

"You don't really need to know the details" -> how can I, let's say, try to work for a company that implements container runtimes if I can't have access to this kind of knowledge? "just the fact that CPUs work by fetching bytes of machine code and decoding them as instructions" -> this is the kind of explanation that I actually need, an how does this relate to Docker images that makes it necessary to have a different image for each architecture – Mariellamarielle

I've edited my question with my current understanding of it. – Mariellamarielle

"like the fact that different OSes have different system call APIs" -> how does that information relate to a Docker image? The Docker image itself requires these system call APIs to be present, or is because some software that may be packaged inside the images need them? – Mariellamarielle

@RafaelEyng: I was just talking about the software package inside the image. The parts of the docker image other than the programs (and their data) is just AFAIK metadata to be read by the docker engine, which already has to be ported to each platform. – Yoshida

@RafaelEyng: BTW, I know this answer isn't complete or details, but I hoped it would be better than nothing, e.g. perhaps as pointers for further avenues of research. I was going to just comment but it was almost an answer so I posted it as such. If you're lucky, someone else who has the time and interest will go into more depth. – Yoshida