SIGSEGV error from sbt compile in Docker shell with OpenJDK 8 and Apple M1 chip

Question

Just got a new Mac with the Apple M1 chip and am trying to set up my dev environment for a project I'm working on. I'm using Docker Desktop and added the platform flag to both the docker-compose.yml (platform: linux/x86_64) and Dockerfile (FROM --platform=linux/amd64 openjdk:8-jdk-stretch). I’m using OpenJDK 8 and sbt 0.13.15

The containers create fine and I can run sbt -Dsbt.ivy.home='.ivy2' -Dsbt.global.base='.sbt' -Dsbt.repository.config='.sbt/repositories' from inside the docker shell and it will create a sbt shell, but if I run compile in this shell I get this error:

[info] Compiling 153 Scala sources and 2 Java sources to /opt/target/scala-2.10/classes...
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x000000400d9d7447, pid=9, tid=0x00000040b87ab700
#
# JRE version: OpenJDK Runtime Environment (8.0_242-b08) (build 1.8.0_242-b08)
# Java VM: OpenJDK 64-Bit Server VM (25.242-b08 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# J 12257 C2 scala.reflect.internal.Types$$anonfun$57.apply(Ljava/lang/Object;)Ljava/lang/Object; (12 bytes) @ 0x000000400d9d7447 [0x000000400d9d7080+0x3c7]
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /opt/hs_err_pid9.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#
qemu: uncaught target signal 6 (Aborted) - core dumped
Aborted

Looking through the hs_err_pid9.log that it created, the only out of the ordinary snippet is:

Event: 66.528 Thread 0x00000040a8020800 Exception <a 'java/io/FileNotFoundException'> (0x00000000f7eaf680) thrown at [/home/openjdk/jdk8u/hotspot/src/share/vm/prims/jni.cpp, line 710]
Event: 66.545 Thread 0x00000040a8020800 Exception <a 'java/io/FileNotFoundException'> (0x00000000f7eb0b38) thrown at [/home/openjdk/jdk8u/hotspot/src/share/vm/prims/jni.cpp, line 710]
Event: 67.214 Thread 0x00000040a801f800 Exception <a 'java/io/FileNotFoundException'> (0x00000000f7d343b8) thrown at [/home/openjdk/jdk8u/hotspot/src/share/vm/prims/jni.cpp, line 710]
Event: 67.219 Thread 0x00000040a801f800 Exception <a 'java/io/FileNotFoundException'> (0x00000000f7d35048) thrown at [/home/openjdk/jdk8u/hotspot/src/share/vm/prims/jni.cpp, line 710]
Event: 68.016 Thread 0x00000040a801f800 Implicit null exception at 0x000000400bd16a9b to 0x000000400bd16c51
Event: 69.405 Thread 0x00000040a801f800 Implicit null exception at 0x000000400c118986 to 0x000000400c118ea5
Event: 69.407 Thread 0x00000040a801f800 Implicit null exception at 0x000000400bdba136 to 0x000000400bdba4ed
Event: 69.556 Thread 0x00000040a801f800 Implicit null exception at 0x000000400bd29275 to 0x000000400bd2945d
Event: 69.567 Thread 0x00000040a801f800 Implicit null exception at 0x000000400be3adaf to 0x000000400be3ae19
Event: 69.835 Thread 0x00000040a801f800 Implicit null exception at 0x000000400be93a35 to 0x000000400be93de1

Here is the codebase for anybody that wants to take a look at the steps to set up the environment as well as the docker-compose.yml and Dockerfile which are both in the root directory.

Hanleigh

Can you reproduce this with some of the variables taken out? Particularly, with a native arm64 JVM running directly on the host? If what you have is at its core a problem with Rosetta's x86 emulation, that's not something we're likely to be able to give you a fix for (beyond "don't do that"). – Exceptional

(I do understand the need for getting a consistent runtime environment if you're pulling in native dependencies -- personally, the tool I use for doing that across MacOS and Linux is Nix, which also does a lot more work to model software build processes as pure functions, and uses a proper functional programming language to do so). – Exceptional

I'm not sure what you mean, what is Nix supposed to be a replacement for? – Hanleigh

Docker, as a mechanism to provide a consistent dependency set (which Nix does more effectively to boot -- since Docker RUN steps aren't sandboxed from network access, there's no guarantee of consistent behavior across invocations; a RUN apt-get update is even expected to have different results each time). While Nix can be used to build Docker images directly (see dockerTools in nixpkgs), one can also write code that assembles native dependencies for whichever OS one prefers. – Exceptional

...that said, if you don't have any native dependencies and this is all native JVM code, there's arguably not a lot of cause to use Docker or Nix; Maven and siblings/competitors/&c have been around for a while -- so I'd expect running sbt on an amd64-native JVM to suffice. – Exceptional

I’m working on a relatively large project with multiple people, so moving away from Docker would be a pretty heavy lift and isn’t going to happen right now. In an attempt to debug further, are you suggesting that I try to install OpenJDK 8 and sbt directly on my computer (not inside a Docker container), and seeing what happens if I try to compile there? – Hanleigh

Also posted on forums.docker.com/t/… – Pandit

Yes, I do recommend installing an amd64-native JVM and sbt directly on your computer. That way you rule out any problems caused by Rosetta not perfectly emulating a real x86_64 CPU. (Those aren't really Docker's problems but Apple's, if they exist, insofar as Rosetta is an Apple-provided component). – Exceptional

Why do you set two different platform names instead of using linux/amd64 for both? – Humblebee

Answer 1

It's likely that you're hitting known bugs in qemu. There are issues when emulating amd64 on an ARM chip. In that case the solution is to move the workload to an amd64 system, or else rebuild the image(s) for ARM.

See the Known Issues section here:

https://docs.docker.com/desktop/mac/apple-silicon/

The relevant text as of Feb. 2022:

However, attempts to run Intel-based containers on Apple silicon machines under emulation can crash as qemu sometimes fails to run the container. In addition, filesystem change notification APIs (inotify) do not work under qemu emulation. Even when the containers do run correctly under emulation, they will be slower and use more memory than the native equivalent.

In summary, running Intel-based containers on Arm-based machines should be regarded as “best effort” only. We recommend running arm64 containers on Apple silicon machines whenever possible, and encouraging container authors to produce arm64, or multi-arch, versions of their containers. We expect this issue to become less common over time, as more and more images are rebuilt supporting multiple architectures.

Rebuild the images as ARM

Since you are building your own image, you should be able to rebuild them for the ARM platform.

You'd want to start from an ARM base image. The one you use currently (openjdk:8-jdk-stretch) is not distributed for ARM, but other versions of the same repository are (e.g. openjdk:19-jdk-buster).

(It's also possible you could rebuild the current base image as ARM yourself, but that would require you to find its source material - sometimes they can be found on GitHub, GitLab, etc.)

Change your Dockerfile to use a base image that's available for ARM, and then rebuild your own image as ARM. Doing that is simple - don't specify platform information, and Docker will use your platform by default.

Multi-platform builds

Why are you using amd64 images? I considered two possible reasons.

1. Because your base image is only available as amd64
2. Because your production environment is amd64, therefore you want the resulting image to be amd64.

If it's only the first reason, then that is covered above. If you rebuild for ARM, you're done.

For the second case, you can use ARM on your development environment, but build amd64 versions for production when you are finished with your work.

To create a multi-platform image and push it to your registry, you can use the buildx command, as below:

docker buildx build --platform linux/amd64,linux/arm64 \
    -t <your image tag> --push .

Multi-platform images are nice, because they will allow you to use ARM images, which are faster and more stable. It also allows any of your teammates with Apple Silicon devices to do the same. But your production platform (usually amd64) and other developers on an amd64 platform can work from the same image (but built for their native platform) also.

Humblebee

Answer 2

Are you using some IDE to do **run compile in the shell**?.

If so, you can try opening the terminal/shell and run the command mentioned in the error

ulimit -c unlimited

After that, do run compile from the same terminal/shell via command. To run it from your IDE, you need to find in IDE settings how to set this param

Frazzled