Why are my Gradle builds dying with exit-code 137?

Question

I've been trying to compile and test a large project to use Gradle. The test run fine until they die unexpectedly. I dug around and resources said that this is due to a memory issue. If I reduce the number of tests in the suite, it runs fine.

I increased the memory by 4x increased the debugging level but I still don't follow what causes this. Here's the horribly cryptic stacktrace. The last line (scroll right) shows the memory settings that I've defined.

...
...
...

1125 tests completed, 30 failed, 9 skipped
:test FAILED
:test (Thread[Daemon worker,5,main]) completed. Took 8 mins 39.684 secs.

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':test'.
> Process 'Gradle Test Executor 1' finished with non-zero exit value 137

* Try:
Run with --debug option to get more log output.

* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':test'.
    at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:69)
    at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:46)
    at org.gradle.api.internal.tasks.execution.PostExecutionAnalysisTaskExecuter.execute(PostExecutionAnalysisTaskExecuter.java:35)
    at org.gradle.api.internal.tasks.execution.SkipUpToDateTaskExecuter.execute(SkipUpToDateTaskExecuter.java:64)
    at org.gradle.api.internal.tasks.execution.ValidatingTaskExecuter.execute(ValidatingTaskExecuter.java:58)
    at org.gradle.api.internal.tasks.execution.SkipEmptySourceFilesTaskExecuter.execute(SkipEmptySourceFilesTaskExecuter.java:52)
    at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:52)
    at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:53)
    at org.gradle.api.internal.tasks.execution.ExecuteAtMostOnceTaskExecuter.execute(ExecuteAtMostOnceTaskExecuter.java:43)
    at org.gradle.execution.taskgraph.DefaultTaskGraphExecuter$EventFiringTaskWorker.execute(DefaultTaskGraphExecuter.java:203)
    at org.gradle.execution.taskgraph.DefaultTaskGraphExecuter$EventFiringTaskWorker.execute(DefaultTaskGraphExecuter.java:185)
    at org.gradle.execution.taskgraph.AbstractTaskPlanExecutor$TaskExecutorWorker.processTask(AbstractTaskPlanExecutor.java:66)
    at org.gradle.execution.taskgraph.AbstractTaskPlanExecutor$TaskExecutorWorker.run(AbstractTaskPlanExecutor.java:50)
    at org.gradle.execution.taskgraph.DefaultTaskPlanExecutor.process(DefaultTaskPlanExecutor.java:25)
    at org.gradle.execution.taskgraph.DefaultTaskGraphExecuter.execute(DefaultTaskGraphExecuter.java:110)
    at org.gradle.execution.SelectedTaskExecutionAction.execute(SelectedTaskExecutionAction.java:37)
    at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:37)
    at org.gradle.execution.DefaultBuildExecuter.access$000(DefaultBuildExecuter.java:23)
    at org.gradle.execution.DefaultBuildExecuter$1.proceed(DefaultBuildExecuter.java:43)
    at org.gradle.execution.DryRunBuildExecutionAction.execute(DryRunBuildExecutionAction.java:32)
    at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:37)
    at org.gradle.execution.DefaultBuildExecuter.execute(DefaultBuildExecuter.java:30)
    at org.gradle.initialization.DefaultGradleLauncher$4.run(DefaultGradleLauncher.java:154)
    at org.gradle.internal.Factories$1.create(Factories.java:22)
    at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:90)
    at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:52)
    at org.gradle.initialization.DefaultGradleLauncher.doBuildStages(DefaultGradleLauncher.java:151)
    at org.gradle.initialization.DefaultGradleLauncher.access$200(DefaultGradleLauncher.java:32)
    at org.gradle.initialization.DefaultGradleLauncher$1.create(DefaultGradleLauncher.java:99)
    at org.gradle.initialization.DefaultGradleLauncher$1.create(DefaultGradleLauncher.java:93)
    at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:90)
    at org.gradle.internal.progress.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:62)
    at org.gradle.initialization.DefaultGradleLauncher.doBuild(DefaultGradleLauncher.java:93)
    at org.gradle.initialization.DefaultGradleLauncher.run(DefaultGradleLauncher.java:82)
    at org.gradle.launcher.exec.InProcessBuildActionExecuter$DefaultBuildController.run(InProcessBuildActionExecuter.java:94)
    at org.gradle.tooling.internal.provider.ExecuteBuildActionRunner.run(ExecuteBuildActionRunner.java:28)
    at org.gradle.launcher.exec.ChainingBuildActionRunner.run(ChainingBuildActionRunner.java:35)
    at org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:43)
    at org.gradle.launcher.exec.InProcessBuildActionExecuter.execute(InProcessBuildActionExecuter.java:28)
    at org.gradle.launcher.exec.ContinuousBuildActionExecuter.execute(ContinuousBuildActionExecuter.java:78)
    at org.gradle.launcher.exec.ContinuousBuildActionExecuter.execute(ContinuousBuildActionExecuter.java:48)
    at org.gradle.launcher.daemon.server.exec.ExecuteBuild.doBuild(ExecuteBuild.java:52)
    at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:36)
    at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
    at org.gradle.launcher.daemon.server.exec.WatchForDisconnection.execute(WatchForDisconnection.java:37)
    at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
    at org.gradle.launcher.daemon.server.exec.ResetDeprecationLogger.execute(ResetDeprecationLogger.java:26)
    at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
    at org.gradle.launcher.daemon.server.exec.RequestStopIfSingleUsedDaemon.execute(RequestStopIfSingleUsedDaemon.java:34)
    at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
    at org.gradle.launcher.daemon.server.exec.ForwardClientInput$2.call(ForwardClientInput.java:74)
    at org.gradle.launcher.daemon.server.exec.ForwardClientInput$2.call(ForwardClientInput.java:72)
    at org.gradle.util.Swapper.swap(Swapper.java:38)
    at org.gradle.launcher.daemon.server.exec.ForwardClientInput.execute(ForwardClientInput.java:72)
    at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
    at org.gradle.launcher.daemon.server.health.DaemonHealthTracker.execute(DaemonHealthTracker.java:40)
    at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
    at org.gradle.launcher.daemon.server.exec.LogToClient.doBuild(LogToClient.java:66)
    at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:36)
    at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
    at org.gradle.launcher.daemon.server.exec.EstablishBuildEnvironment.doBuild(EstablishBuildEnvironment.java:72)
    at org.gradle.launcher.daemon.server.exec.BuildCommandOnly.execute(BuildCommandOnly.java:36)
    at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
    at org.gradle.launcher.daemon.server.health.HintGCAfterBuild.execute(HintGCAfterBuild.java:41)
    at org.gradle.launcher.daemon.server.api.DaemonCommandExecution.proceed(DaemonCommandExecution.java:120)
    at org.gradle.launcher.daemon.server.exec.StartBuildOrRespondWithBusy$1.run(StartBuildOrRespondWithBusy.java:50)

BUILD FAILED    at org.gradle.launcher.daemon.server.DaemonStateCoordinator$1.run(DaemonStateCoordinator.java:246)

    at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:54)
    at org.gradle.internal.concurrent.StoppableExecutorImpl$1.run(StoppableExecutorImpl.java:40)

Caused by: org.gradle.process.internal.ExecException: Process 'Gradle Test Executor 1' finished with non-zero exit value 137
Total time: 9 mins 38.624 secs
    at org.gradle.process.internal.DefaultExecHandle$ExecResultImpl.assertNormalExitValue(DefaultExecHandle.java:367)
    at org.gradle.process.internal.DefaultWorkerProcess.waitForStop(DefaultWorkerProcess.java:161)
    at org.gradle.api.internal.tasks.testing.worker.ForkingTestClassProcessor.stop(ForkingTestClassProcessor.java:86)
    at org.gradle.api.internal.tasks.testing.processors.RestartEveryNTestClassProcessor.endBatch(RestartEveryNTestClassProcessor.java:60)
    at org.gradle.api.internal.tasks.testing.processors.RestartEveryNTestClassProcessor.stop(RestartEveryNTestClassProcessor.java:54)
    at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
    at org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
    at org.gradle.messaging.dispatch.FailureHandlingDispatch.dispatch(FailureHandlingDispatch.java:29)
    at org.gradle.messaging.dispatch.AsyncDispatch.dispatchMessages(AsyncDispatch.java:132)
    at org.gradle.messaging.dispatch.AsyncDispatch.access$000(AsyncDispatch.java:33)
    at org.gradle.messaging.dispatch.AsyncDispatch$1.run(AsyncDispatch.java:72)
    ... 2 more

Stopped 0 compiler daemon(s).
Received result Failure[value=org.gradle.initialization.ReportedException: org.gradle.internal.exceptions.LocationAwareException: Execution failed for task ':test'.] from daemon DaemonInfo{pid=48, address=[65adcc0f-8881-4511-b7a0-1b88c7016510 port:34390, addresses:[/0:0:0:0:0:0:0:1%lo, /127.0.0.1]], idle=false, context=DefaultDaemonContext[uid=14546872-57d6-4da6-8358-dc41cb7bb01d,javaHome=/usr/lib/jvm/java-8-oracle,daemonRegistryDir=/srv/myapplication/.gradle/daemon,pid=48,idleTimeout=120000,daemonOpts=-XX:MaxMetaspaceSize=320m,-XX:+DisableExplicitGC,-XX:+UseConcMarkSweepGC,-XX:NewSize=124m,-XX:SurvivorRatio=16,-Xms257m,-Xmx2049m,-Dfile.encoding=UTF-8,-Duser.country=US,-Duser.language=en,-Duser.variant]} (build should be done).

Bilocular

Answer 1

This issue seems to be related to Linux rather than Gradle as stated in the Jenkins docs:

In cases where virtual memory is running short the kernel OOM (Out of Memory) killer may forcibly kill Jenkins or individual builds. If this occurs on Linux you may see builds terminate with exit code 137 (128 + signal number for SIGKILL). The dmesg command output will show log messages that will confirm the action that the kernel took.

https://wiki.jenkins-ci.org/display/JENKINS/I'm+getting+OutOfMemoryError

Bilocular

I think this is not as useful it would look like there are another response below that is more useful. This is not a problem that apply only to Jenkins. It applies to any CI too. – Lumisterol

OOM-Kill is disabled by default on some Linux distributions, Ubuntu being one. – Paste

Answer 2

I've had similar issue on DigitalOcean's server, my gradle build failed completely on test stage with very similar stacktrace and without a single test being executed.

It is stated in Gradle docs that gradle daemon should not be run in CI environments. So I just added --no-daemon to my build command and everything worked well and good. Also stopping daemon with ./gradlew --stop has been useful, but it worked only for a single build - the next one also failed.

My build command:

./gradlew build --no-daemon

Firedamp

As indicated by the accepted answer in this thread #44786875 you might also need one more param; --no-daemon --max-workers 2 – Fortunate

As of 6.6.1, use of daemon is recommended in CI docs.gradle.org/6.6.1/userguide/… – Bovill

@Bovill Thanks for the update! I guess that my comment is still relevant, since the original documentation says: "However, if you suspect that Daemon makes your CI builds unstable, you can disable it to use a fresh runtime for each build since the runtime is completely isolated from any previous builds." – Firedamp

Answer 3

I had the same issue but in a CI environment where build are launched in docker container. In that specific case the JVM is not aware of how much memory it can use and you can experience this type of issues.

In order to let the JVM know how much memory is available, you can use

gradle build -Dorg.gradle.jvmargs=-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap

and also set it on your test tasks:

test {
  jvmArgs "-XX:+UnlockExperimentalVMOptions", "-XX:+UseCGroupMemoryLimitForHeap"
}

This is a new JVM feature introduced in 8u131+

see: http://royvanrijn.com/blog/2018/05/java-and-docker-memory-limits/

Bertiebertila

Simply adding the test jvm args to my gradle worked for me in CircleCi, didn't have to change the command. – Plump

Is not relevant to Java10 and newer – Binary

Answer 4

I've also been having the same problem on CircleCI, but I didn't have any luck with any of the above. This is what I found:

• Adding -Dorg.gradle.daemon=false to my CircleCI config.yml stopped the daemon from being used, but didn't fix the problem.
• Adding -Dorg.gradle.workers.max=2 to GRADLE_OPTS, or --max-workers 2 to the gradle command didn't seem to have much/any effect from what I could see. I tried --max-workers=2 just in case, because both formats seem to be floating around on Google. I connected to my CircleCI container, and in top I could still see 3-4 Java processes forking off, so not sure this is doing anything?
• I also tried max workers = 1 in the combinations above.
• Tried -XX:+UnlockExperimentalVMOptions and -XX:+UseCGroupMemoryLimitForHeap in both JVM args, and in the test {} configuration inside my build as suggested by Baptiste Mesta. I don't see how this could work; I would have thought the multiple forked processes don't know what proportion of the container memory the other processes are using up? Unless I'm not understanding it correctly.

In the end, I fixed it just by being nice and explicit with the memory settings, rather than using magic:

• Circle CI config: GRADLE_OPTS: -Xmx2048m -Dorg.gradle.daemon=false
• Gradle build: test { maxHeapSize = "512m" }

Edit: You may need to go lower, depending on whether you have other processes running.

Laugh

This worked for my Android project and circle ci, added environment: GRADLE_OPTS: -Xmx2048m -Dorg.gradle.daemon=false to circle ci's config.yml file. - Thanks – Torto

using an upgraded machine (resource_class) available in the paid accounts usually solves this issue – Ovaritis

I wonder why this solution works. The Gradle docs (docs.gradle.org/current/userguide/…) state that the actual build is run by the Gradle daemon, which is not affected by GRADLE_OPTS. However, under 'Configuring JVM memory' (docs.gradle.org/current/userguide/…) there is a note that says that if the daemon is disabled (which you do) and the client VM has the same settings (how would I know that?), then the build will not run in a separate VM fork, but in the client VM itself. – Disannul

Answer 5

Similar to Baptiste Mesta's answer but for JDK 11 I used the UseContainerSupport JVM option to solve my Jenkins build.

./gradlew test --no-daemon -Dorg.gradle.jvmargs=-XX:+UseContainerSupport

Nealson

Isn't UseContainerSupport enabled by deafult? – Binary

FWIW UseContainerSupport was required for this to work for me. – Chemar

Answer 6

I had a similar issue with Bamboo agents (running on Docker) running Gradle 4.6. The test task just abruptly terminated with

    build   27-Dec-2018 22:00:20    22:00:20.018 [DEBUG] [org.gradle.process.internal.DefaultExecHandle] Process 'Gradle Test Executor 1' finished with exit value 137 (state: FAILED)

We have over 3000 unit tests. In our case, the issue was solved using forkEvery to limit the number of test classes run per test executor process, and also limiting memory usage:

    test {
        maxHeapSize '512m'
        forkEvery 100
        jvmArgs '-Xmx512m', '-Xms512m'
    }

Roundworm

where would i add the test{} options? I tried in my build.gradle file and it failed to run – Unamuno

Answer 7

This is actually a memory issue, Generally, Docker container has a memory limit of 4G so you need to take care that your java heap doesn't cross that limit, there are few solutions to it I am taking reference of android

1. Add this to gradle.properties (change size according to your need)

   org.gradle.jvmargs=-Xmx10248m -XX:MaxPermSize=256m

2. Add this to your build.gradle

   android.testOptions.unitTests.all { maxHeapSize = "1024m" }

or

android {
    testOptions {
        unitTests {
            // Any other configurations
            all {
                maxHeapSize = "1024m"
            }
        }
    }

If you are still running into OOM issues you can also limit the max workers for gradle: ./gradlew test --max-workers 4

Hope this helps.

Ma

10248m 👈 10G? – Niue

Answer 8

If you are having trouble with this on CircleCi, where you are trying to run Robolectric Test

I found the solution to my problems on their documentation page for android

https://circleci.com/docs/2.0/language-android/

which was simply to add

android {
    testOptions {
        unitTests {
            returnDefaultValues = true
            includeAndroidResources = true

            all {
                maxHeapSize = "1024m"
            }
        }
    }

If you are still running into OOM issues you can also limit the max workers for gradle: ./gradlew test --max-workers 4

Daughterly

Answer 9

I'd similar issue on intellij. I upgraded the memory as stated in above answers and disabled daemon in gradle.properties file

org.gradle.daemon=false

Gradle Daemon is a long lived build process. It caches the information about files, tasks,project structure, etc from the previous build. Hence, making your subsequent builds efficient. Daemon process is enabled by default. But sometimes it gets corrupted, so you may try disabling daemon process.

Ponton

Answer 10

For my project, We are using Java 11 and Gradle 7.3.3

I fixed this issue by increasing the heap size and added jvm args in the gradle test task.

test {
   useJUnitPlatform()
   maxHeapSize("2048m")
   jvmArgs "-XX:+UnlockExperimentalVMOptions"
}

Fabliau