Maven surefire plugin crashing jvm on java 11 (Corrupted STDOUT by directly writing to native stream in forked JVM 1)

Question

Running a maven build using java 11, the build issues the following warning while running tests:

[WARNING] Corrupted STDOUT by directly writing to native stream in forked JVM 1. See FAQ web page and the dump file /home/thomas/code/irdeto-control/fps-license-service/fps/target/surefire-reports/2019-04-11T14-05-32_318-jvmRun1.dumpstream

...followed by the following stderr output after the build fails:

$ cat error.message 
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.22.1:test (default-test) on project fps: There are test failures.
[ERROR] 
[ERROR] Please refer to /home/user/code/employer-control/fps-license-service/fps/target/surefire-reports for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd /home/user/code/employer-control/fps-license-service/fps && /usr/lib/jvm/jdk-11.0.2/bin/java '-javaagent:/home/user/.m2/repository/org/jacoco/org.jacoco.agent/0.8.0/org.jacoco.agent-0.8.0-runtime.jar=destfile=/home/user/code/employer-control/fps-license-service/fps/target/jacoco.exec,excludes=com.employer.rights.fairplay.*' -jar /home/user/code/employer-control/fps-license-service/fps/target/surefire/surefirebooter7853689441541829546.jar /home/user/code/employer-control/fps-license-service/fps/target/surefire 2019-04-11T14-05-32_318-jvmRun1 surefire11275213325677189658tmp surefire_01035200394006888746tmp
[ERROR] Error occurred in starting fork, check output in log
[ERROR] Process Exit Code: 134
[ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd /home/user/code/employer-control/fps-license-service/fps && /usr/lib/jvm/jdk-11.0.2/bin/java '-javaagent:/home/user/.m2/repository/org/jacoco/org.jacoco.agent/0.8.0/org.jacoco.agent-0.8.0-runtime.jar=destfile=/home/user/code/employer-control/fps-license-service/fps/target/jacoco.exec,excludes=com.employer.rights.fairplay.*' -jar /home/user/code/employer-control/fps-license-service/fps/target/surefire/surefirebooter7853689441541829546.jar /home/user/code/employer-control/fps-license-service/fps/target/surefire 2019-04-11T14-05-32_318-jvmRun1 surefire11275213325677189658tmp surefire_01035200394006888746tmp
[ERROR] Error occurred in starting fork, check output in log
[ERROR] Process Exit Code: 134
[ERROR]     at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:669)
[ERROR]     at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:282)
[ERROR]     at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:245)
[ERROR]     at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1183)
[ERROR]     at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1011)
[ERROR]     at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:857)
[ERROR]     at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
[ERROR]     at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)

//stack frames ommited

Here is an extract from the dump stream referring to the error above:

# Created at 2019-04-11T14:05:32.824
Corrupted STDOUT by directly writing to native stream in forked JVM 1. Stream 'FATAL ERROR in native method: processing of -javaagent failed'.
java.lang.IllegalArgumentException: Stream stdin corrupted. Expected comma after third character in command 'FATAL ERROR in native method: processing of -javaagent failed'.
        at org.apache.maven.plugin.surefire.booterclient.output.ForkClient$OperationalData.<init>(ForkClient.java:507)
        at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.processLine(ForkClient.java:210)
        at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:177)
        at org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:88)
        at java.base/java.lang.Thread.run(Thread.java:834)

Dropping the surefire version to 2.18 stops the jvm from crashing but does not execute the tests. Any version of surefire above 2.18 including the latest crashes the build with the errors above.

I am clearly not the only one experiencing the issue as it is also shown here

Furthermore the issue is also known by the surefire development community as it is discussed here

That said, unlike most issues that tend to be resolved in more recent version of plugins, this issue appears to raise its ugly head again with the more recent versions of surefire and the jvm.

Any solutions OR work-arounds are welcome. I am using the following versions:

Apache Maven 3.5.3 (3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T20:49:05+01:00)

Java version: 11.0.2, vendor: Oracle Corporation

OS name: "linux", version: "4.15.0-47-generic", arch: "amd64", family: "unix"

maven.compiler.source: 1.8

maven.compiler.target: 1.8

maven-surefire-plugin.version: 2.18 < VERSION_USED <= 3.0.0-M3

Oratorian

Are you using JaCoCo Plugin in your build? – Connivent

Having had the same problem we changed JaCoCo from 0.7.7 to 0.8.3 and the problem disappeared. – Alnico

I also was having the same problem using maven-surefire-plugin.version:3.0.0-M3 and org.jacoco:jacoco-maven-plugin:0.8.1 and upgrading to org.jacoco:jacoco-maven-plugin:0.8.5 fixed the issue. – Starshaped

please see the end of the discussion in github.com/GateNLP/gateplugin-Format_Bdoc/issues/8 – Femur

Use the latest version of JaCoCo (currently JaCoCo 0.8.5) along with Java 9 - Java 14. – Femur

Answer 1

Setting forkCount=0 is inherently bad idea. We decided to report these issues in the dump files in order to force you to search the libraries which break the standard output. Here the Maven users mentioned one framework, and we found the same problems in log4j and Arquillian which are already fixed for very long time. Some OSGi frameworks use the standard input and output in the forked JVM. Therefore the Surefire has decided to use another channel and it is a new TCP/IP. This channel won't be enabled by default in the version 3.0.0-M5. You have to enable this preview feature by yourself in future version 3.0.0-M5 as follows:

<forkNode implementation="org.apache.maven.plugin.surefire.extensions.SurefireForkNodeFactory"/>

It will be quite a big change which will be finally enabled in the version 3.0.0.

Femur

Had the same problem and when I try this solution, the build process just hangs forever in the test phase until I kill it. There was absolutely no problem with this under Java 8, why was this even done? The software tested here is one which NEEDS to read/write from stdin/stdout as part of its main functionality – Habitual

Setting forkCount to 0 DOES work, your suggested solution does not work and hangs. – Habitual

@johann-petrak forkCount=0 is bad idea. If something does not work, please talk with us here. The Apache team would help you to fix any problem. – Femur

as I said, the suggested "forkNode" solution made my tests hang, while forkCount=0 made it work, so there was not much choice for me, TBH. Let me know how I can provide more information here or on some issue tracker to figure this out better. – Habitual

We need to have your reproducible project. Ideally you can post a link with it at the GitHub or Gist or you can report an issue at the Apache JIRA and attach the zip of the project. – Femur

See github.com/GateNLP/gateplugin-Format_Bdoc/issues/8 – Habitual

The result of this discussion (see the GH link above): It was a problem related to the combination of 1. JaCoCo version 0.8.1, and 2. Java 11+ – Femur

This solution works. However, it has taken me quite some time to find it. It should be referenced in the FAQ, where currently, there is no solution whatsoever. – Thiol

@Thiol Would you pls participate in the open source in github.com/apache/maven-surefire and provide a pull-request with some hints in the FAQ document? Thx – Femur

This also happens when a module is not found. In my case I have seen: > Corrupted STDOUT by directly writing to native stream in forked JVM 1. Stream 'java.lang.module.FindException: Module jcardsim not found'. This seems to be completely unrelated to the problem description, but a not found module can obviously cause this with maven-surefire-plugin:3.0.0-M5. – Romain

New version 3.0.0-M6 was deployed to Maven Central. Pls let us know your feedback. Enjoy! – Femur

So still not understanding what the fix is. I am using jacoco 0.8.8 and surefire 3.0.0-M7. My teamcity build agent was recently upgraded from java 8 to 11. – Logging

I am using surefire version 3.1.0 but the default ForkNodeFactory still seems to be Process Pipe, not TCP/IP. Per @tibor17, the default should have been TCP/IP since 3.0.0? Another question is why directly writing to new FileOutputStream(FileDescriptor.out) is considered a bad idea? Isn't it the same as System.out is writing to? Any pointer is appreciated. Thanks. – Clarenceclarenceux

Answer 2

Not a fix, but a workaround, do not run tests concurrently, asynchronously:

    <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-surefire-plugin</artifactId>
      <version>${maven-surefire-plugin.version}</version>
      <configuration>
        <forkCount>0</forkCount>
      </configuration>
    </plugin>

Oratorian

You copy/pasted the same thing twice – Permanency

Dont use <forkCount>0</forkCount> and use the solution with forkNode. – Femur

Answer 3

This also happens when a module is not found.

In my case I have seen:

Corrupted STDOUT by directly writing to native stream in forked JVM 1. Stream 'java.lang.module.FindException: Module jcardsim not found'.

This seems to be completely unrelated to the problem description, but a not found module can obviously cause this with maven-surefire-plugin:3.0.0-M5.

Romain

Answer 4

Add these plugins in pom.xml

<plugin>
    <artifactId>maven-compiler-plugin</artifactId>
    <version>3.8.1</version>
</plugin>
<plugin>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>2.22.2</version>
</plugin>

Boutis

Answer 5

I had the same issue with java version "1.8.0_211" the process start a new java.exe console and fail with the message

"[ERROR] The forked VM terminated without properly saying goodbye. VM crash or System.exit called? [ERROR] Command was cmd.exe /X /C ""C:\Program Files\Java\jdk1.8.0_211\jre\bin\java"

I have solved the problem compiling with java version "1.8.0_202", but I have not understood what is the difference between this two java versions.

Grose