How to prevent Java from exceeding the container memory limits?

Question

I'm running a Java program inside a Docker container that has a hard memory limit of 4GB. I've set the max heap to 3GB but still the Java program exceeds the limit and gets killed (OOMKilled).

My question is: How can I configure Java to respect the set container limit and throw an OutOfMemoryException instead of trying to allocate beyond the limit and get its ass kicked by the host kernel?

Update: I'm an experienced Java developer and have a fair understanding of the JVM. I know how to set the max heap, but I wonder if anyone knows of a way to set a limit to the total memory that the JVM process claims from the OS.

Wivinah

Can you share the full command you use to run your JVM? – Benner

I only set the max heap to 3GB. – Wivinah

@GeertSchuring We need to see how you do that. Please, share the full command – Benner

Well this trigger an Error and it is unlikely that you are able to get back on your feet after a OOME. To prevent the JVM to use too much memory, you need to reduce the consumption of memory in your application or you need to upgrade your container with more memory allocated. – Bradberry

Dont explain what you think you did. Put down the exact command line you are using (and probably add the exact JRE version you are using, too ) – Birkner

This is the command that is used by Docker to start the application: java -Xmx3g -jar application.jar – Wivinah

I'll clarify the questions since I notice that its not clear. – Wivinah

Please edit your question to include the java console command. Updates in the comments are not as noticeable for future readers. Thanks! – Cowskin

Answer 1

When a Java application is executed inside a container, the JVM ergonomics (which is responsible for dynamically assign resources based on the host's capabilities) does not know it is running inside a container and it calculates the number of resources to be used by the Java app based on the host that is executing your container. Given that, it does not matter if you set limits to your container, the JVM will take your host's resources as the base for doing that calculation.

From JDK 8u131+ and JDK 9, there’s an experimental VM option that allows the JVM ergonomics to read the memory values from CGgroups. To enable it you must pass the following flags to the JVM:

-XX:+UnlockExperimentalVMOptions and -XX:+UseCGroupMemoryLimitForHeap

If you enable these flags, the JVM will be aware that is running inside a container and will make the JVM ergonomics to calculate the app's resources based on the container limits and not the host's capabilities.

Enabling the flags:

$ java -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -jar app.jar

You can dynamically pass the JVM options to your container with ENV variables.

Example:

The command to run your app would like something like:

 $ java ${JAVA_OPTIONS} -jar app.jar

And the docker run command needs to pass the ENV variable like this:

$ docker run -e JAVA_OPTIONS="-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap" myJavaImage

Hope this helps!

Boise

well crafted and explained. it does not matter if you set limits to your container, the JVM will take your host's resources as the base for doing that calculation. got it. (1)But does not this beat the whole point of namespace and cgroups. (2) from wiki - cgroup namespace type has existed since March 2016 in Linux 4.6. So what is the point of this new cgroup namespace. Why should applications i.e JVM care about adding container support - docker engine i.e containerd should do it by default. Bit difficult to wrap the head around this ???? – Nievelt

Answer 2

In addition to Fabian Rivera's answer I've found that Java 10 has good support for running in containers without any custom startup parameters. By default it uses 25% of the containers memory as heap, which might be a bit low for some users. You can change this with the following parameter:

-XX:MaxRAMPercentage=50

To play around with Java 10 run the following docker command:

docker run -it --rm -m1g --entrypoint bash openjdk:10-jdk

It will give you a bash environment where you can run executables from the JDK. For instance, to run a small piece of script you can use jrunscript like this:

jrunscript -e "print(Packages.java.lang.Runtime.getRuntime().maxMemory()/(1<<20) + 'M')"

This will show you the size of the heap in MB. To change the percentage of total container memory that is used for the heap add the MaxRAMPercentage parameter like this:

jrunscript -J-XX:MaxRAMPercentage=50 -e "print(Packages.java.lang.Runtime.getRuntime().maxMemory()/(1<<20) + 'M')"

Now you can play around with the sizing of the container and the max percentage of heap.

Wivinah

-XX:+PrintFlagsFinal does not seem to work though. it still seems to show host config. Why is that? – Enidenigma