GitLab CI is failing due to maven-surefire-plugin with VM crash

Question

We have around of 10 different applications that are Spring Boot projects with Groovy.

All of our projects build correctly in all developer work stations and they were running correctly until yesterday, however suddenly all of them stopped working today only in our GitLab CI pipelines with below error:

[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:37 min
[INFO] Finished at: 2018-10-31T17:49:11Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.21.0:test (default-test) on project ctg-oms-component: There are test failures.
[ERROR] 
[ERROR] Please refer to /builds/ctg-integrations/ctg-oms-component/target/surefire-reports for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date]-jvmRun[N].dump, [date].dumpstream and [date]-jvmRun[N].dumpstream.
[ERROR] ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called?

I've pulled the same docker image that is using our GitLab CI pipeline, tested building the project and everything works correctly. However, the error occurs only in GitLab CI.

After an investigation looks like surefire is creating a fork that makes GitLab CI docker crash. In order to fix this, I've added below explicit configuration to avoid forked VM and this got rid of above error.

<!-- Needed only for GitLab CI -->
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <configuration>
        <forkCount>0</forkCount>
    </configuration>
</plugin>

Do you know why this is occurring? Is there another way to fix GitLab CI to avoid this issue? I don't really like this workaround, since it is just a way to avoid GitLab CI to explode but don't know how Docker is handled behind the scenes in Gitlab.

Adorl

Answer 1

The issues are related to the latest maven docker images.

There is an open github issue where people reported the same problem: https://github.com/carlossg/docker-maven/issues/90

After an investigation I could sort out the problem using alpine version that saved me of adding the maven-surefire-plugin workaround. It's important to mention that using the surefire workaround brings another problem such as plugins like jacoco don't run since they need the VM fork.

So, these images work seamlessly (no surefire workaround was needed):

• maven:3.3.9-jdk-8
• maven:3.5.3-jdk-8
• maven:3.5.4-jdk-8-alpine
• maven:3.6.0-jdk-8-alpine

However if we use the non alpine version the issue persists.

Adorl

Actually the maintainers of the maven docker image regularly overwrite the old image tags with newer ones which break the "fixed" builds again and again. The issue comes from a bug in the JDK. The only definite solution is to use a fixed image. I'm using maven-3.6.1 and jdk 8 u212 and it works fine as long as the image still exist on dockerhub, the sha256 image id is maven@sha256:97c2ba42d7a8119de291aa944b5549fa032134cb0f403874b082100d68042307 and can be found here github.com/docker-library/repo-info/blob/… – Acidforming

Answer 2

I had the same issue

You can find here the issue I created on gitlab for this, with detailed explanations : https://gitlab.com/gitlab-org/gitlab-ce/issues/53734

Also, I found a "workaround". maven:3.5.3-jdk-8 as the docker image on which the build is executing. Or, the forkCount=0 property on maven-surefire-plugin.

But this is disturbing. How, out of a sudden, builds start to fail? Don't know, and I don't think I have the experience necessary to solve this.

Till then, maybe this helps you

Cladding

Your answer solved the issue I had while setting up a SonarQube analysis, thank you! Be wary when setting the forkCount to 0, as it might result in a 0% JaCoCo code coverage: When using the maven-surefire-plugin or maven-failsafe-plugin you must not use a forkCount of 0 or set the forkMode to never as this would prevent the execution of the tests with the javaagent set and no coverage would be recorded. source – Chimney

Answer 3

Try adding <useSystemClassLoader>false</useSystemClassLoader> to your maven-surefire-plugin configuration.

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <configuration>
        <useSystemClassLoader>false</useSystemClassLoader>
    </configuration>
</plugin>

Dacha

Answer 4

We found the root case and fixed it in the versions 3.0.0-M4 and 3.0.0-SNAPSHOT:

https://issues.apache.org/jira/browse/SUREFIRE-1702

https://issues.apache.org/jira/browse/SUREFIRE-1703

https://issues.apache.org/jira/browse/SUREFIRE-1704

Brambly

New version 3.0.0-M6 was deployed to Maven Central. Pls let us know your feedback. Enjoy! – Brambly

Answer 5

I had the same issue when I used the maven:3-jdk-8 in my gitlab-ci.yml for the maven build. I changed it to maven:3-jdk-9 and the issue was gone.

Mountebank