Strategy for debugging surefire "The forked VM terminated without saying properly goodbye. VM crash or System.exit called ?"

Question

I am working on a rather complex java project with many dependencies and many unit tests.

I am using java 1.6.0_65 on mac (mavericks) with maven 3.0.5 with maven-surefire-plugin:2.16 running in several forks. My problem is that running this setup with several forks causes a fork to exit with:

"The forked VM terminated without saying properly goodbye. VM crash or System.exit called ?"

running this with only one fork does not produce the problem (and everything passes)

There is some information out there about this problem including this StackOverflow question and this surefire bug (which seems to be solved by now)

I am aware that the "Answer" to this situation is to find what in my code calls System.exit() - Nothing i could find.

Or what causes my JVM to crash - there are no hs_pid crash reports.

My Question is what what kind of strategy can i use to find a locate this cause? To clarify, I am not interested in the answer noted above, but a way to find where it is originating from. (Or even better a different answer altogether to what might be causing this )

---

my Surefire configuration is: (but i did try other combinations)

<parallel>classes</parallel>
<threadCount>1</threadCount>
<forkCount>1C</forkCount>
<reuseForks>false</reuseForks>
<useSystemClassLoader>false</useSystemClassLoader>
<useManifestOnlyJar>true</useManifestOnlyJar>
<useFile>true</useFile>
<redirectTestOutputToFile>true</redirectTestOutputToFile>
<runOrder>alphabetical</runOrder>

---

Update #1 adding the relevant out put after running the maven goal with --debug (-X)

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.16:test (default-test) on project event-logger: ExecutionException; nested exception is java.util.concurrent.ExecutionException: java.lang.RuntimeException: The forked VM terminated without saying properly goodbye. VM crash or System.exit called ?
[ERROR] Command was/bin/sh -c cd /Users/nitzan/work/nitzan_5_parallel_tests/event-logger && /Library/Java/JavaVirtualMachines/1.6.0_65-b14-462.jdk/Contents/Home/bin/java org.apache.maven.surefire.booter.ForkedBooter /Users/nitzan/work/nitzan_5_parallel_tests/event-logger/target/surefire/surefire5107531798951225850tmp /Users/nitzan/work/nitzan_5_parallel_tests/event-logger/target/surefire/surefire_12561116468761732560tmp
[ERROR] -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.16:test (default-test) on project event-logger: ExecutionException; nested exception is java.util.concurrent.ExecutionException: java.lang.RuntimeException: The forked VM terminated without saying properly goodbye. VM crash or System.exit called ?
Command was/bin/sh -c cd /Users/nitzan/work/nitzan_5_parallel_tests/event-logger && /Library/Java/JavaVirtualMachines/1.6.0_65-b14-462.jdk/Contents/Home/bin/java org.apache.maven.surefire.booter.ForkedBooter /Users/nitzan/work/nitzan_5_parallel_tests/event-logger/target/surefire/surefire5107531798951225850tmp /Users/nitzan/work/nitzan_5_parallel_tests/event-logger/target/surefire/surefire_12561116468761732560tmp
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:213)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
    at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320)
    at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
    at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
    at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
    at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
    at org.codehaus.classworlds.Launcher.main(Launcher.java:47)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
Caused by: org.apache.maven.plugin.MojoFailureException: ExecutionException; nested exception is java.util.concurrent.ExecutionException: java.lang.RuntimeException: The forked VM terminated without saying properly goodbye. VM crash or System.exit called ?
Command was/bin/sh -c cd /Users/nitzan/work/nitzan_5_parallel_tests/event-logger && /Library/Java/JavaVirtualMachines/1.6.0_65-b14-462.jdk/Contents/Home/bin/java org.apache.maven.surefire.booter.ForkedBooter /Users/nitzan/work/nitzan_5_parallel_tests/event-logger/target/surefire/surefire5107531798951225850tmp /Users/nitzan/work/nitzan_5_parallel_tests/event-logger/target/surefire/surefire_12561116468761732560tmp
    at org.apache.maven.plugin.surefire.SurefirePlugin.assertNoException(SurefirePlugin.java:198)
    at org.apache.maven.plugin.surefire.SurefirePlugin.handleSummary(SurefirePlugin.java:188)
    at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:852)
    at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:720)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
    ... 25 more
Caused by: org.apache.maven.surefire.booter.SurefireBooterForkException: ExecutionException; nested exception is java.util.concurrent.ExecutionException: java.lang.RuntimeException: The forked VM terminated without saying properly goodbye. VM crash or System.exit called ?
Command was/bin/sh -c cd /Users/nitzan/work/nitzan_5_parallel_tests/event-logger && /Library/Java/JavaVirtualMachines/1.6.0_65-b14-462.jdk/Contents/Home/bin/java org.apache.maven.surefire.booter.ForkedBooter /Users/nitzan/work/nitzan_5_parallel_tests/event-logger/target/surefire/surefire5107531798951225850tmp /Users/nitzan/work/nitzan_5_parallel_tests/event-logger/target/surefire/surefire_12561116468761732560tmp
    at org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkPerTestSet(ForkStarter.java:316)
    at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:169)
    at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:958)
    at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:822)
    ... 28 more
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: The forked VM terminated without saying properly goodbye. VM crash or System.exit called ?
Command was/bin/sh -c cd /Users/nitzan/work/nitzan_5_parallel_tests/event-logger && /Library/Java/JavaVirtualMachines/1.6.0_65-b14-462.jdk/Contents/Home/bin/java org.apache.maven.surefire.booter.ForkedBooter /Users/nitzan/work/nitzan_5_parallel_tests/event-logger/target/surefire/surefire5107531798951225850tmp /Users/nitzan/work/nitzan_5_parallel_tests/event-logger/target/surefire/surefire_12561116468761732560tmp
    at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
    at java.util.concurrent.FutureTask.get(FutureTask.java:83)
    at org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkPerTestSet(ForkStarter.java:300)
    ... 31 more
Caused by: java.lang.RuntimeException: The forked VM terminated without saying properly goodbye. VM crash or System.exit called ?
Command was/bin/sh -c cd /Users/nitzan/work/nitzan_5_parallel_tests/event-logger && /Library/Java/JavaVirtualMachines/1.6.0_65-b14-462.jdk/Contents/Home/bin/java org.apache.maven.surefire.booter.ForkedBooter /Users/nitzan/work/nitzan_5_parallel_tests/event-logger/target/surefire/surefire5107531798951225850tmp /Users/nitzan/work/nitzan_5_parallel_tests/event-logger/target/surefire/surefire_12561116468761732560tmp
    at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:485)
    at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:352)
    at org.apache.maven.plugin.surefire.booterclient.ForkStarter.access$300(ForkStarter.java:85)
    at org.apache.maven.plugin.surefire.booterclient.ForkStarter$2.call(ForkStarter.java:288)
    at org.apache.maven.plugin.surefire.booterclient.ForkStarter$2.call(ForkStarter.java:283)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
    at java.lang.Thread.run(Thread.java:695)
[ERROR] 
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

Mana

does /Users/nitzan/work/nitzan_5_parallel_tests/event-logger/target/surefire/surefire_12561116468761732560tmp have any useful information ? – Milliner

the file contains the surefire configuration and classpath that surefire passes to the ForkedBooter. a bit tricy to isolate since it's deleted after the test, but nothing in the content seems useful. – Mana

There is a bug affecting all Surefire 2.x plugins that prevents forked VMs to start properly on all Linux computers where OpenJDK 8u181 is installed. See also bugs.debian.org/cgi-bin/bugreport.cgi?bug=911925 and issues.apache.org/jira/browse/SUREFIRE-1588 – Beaded

Thank you @THelper, one of the workarounds on the first ticket worked for me: <configuration> <useSystemClassLoader>false</useSystemClassLoader></configuration> – Paragraph

Answer 1

Steps:

(1) Run mvn with the -e and -X options to get more debug information.

(2) Look for "Error" in the output. In my case, when I ran the mvn command, part of the output included:

[ERROR] Command wascmd.exe /X /C "C:\dev\dev-tools\.....

(3) Execute the problematic command directly in the command shell.

In my case, executing

cmd.exe /X /C "C:\dev\dev-tools\....

from the command line resulted in an OutOfMemoryError.

Cinquecento

this does not answer the question, the question asks for a strategy for debugging "The forked VM terminated without saying properly goodbye". there are many possible reasons why this may be -> OutOfMemoryError is just one option out of many for this issue. – Mana

@NitzanVolman: I respectfully disagree with your comment. While kc2001's answer is terse, it worked for me, and it is a general strategy. In short, the strategy is: (1) Re-run mvn with -e and -X options; (2) Find the command that failed; (3) Re-run this command. If the output is still insufficient, consider running the command with -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5001, then connecting with a remote JVM debugger (IDE of your choice can do it). Bigger question: Why is the output from mvn so useless for these cases? – Liquescent

I updated the answer to make the steps more explicit, as suggested by Kevin. – Cinquecento

This strategy worked for me. In my case I found [ERROR] . . . Failed to get user permissions from end-point: https://[mycompanyurl]/users in the terminal output. I realized that I wasn't logged on to my corporate VPN. Logging on to the VPN resolved the error. – Purism

Answer 2

In my case the configuration forkedProcessExitTimeoutInSeconds for the maven-surefire-plugin helps. The default value are since maven-surefire-plugin:2.20.1 30 seconds. My project gots very time consuming test and so the forked JVM chrashes. Configure the plugin in the pom with the following property solves the problem.

<plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <configuration>
                <forkedProcessExitTimeoutInSeconds>120</forkedProcessExitTimeoutInSeconds>
        </configuration>
</plugin>

Hindrance

Make sense in combination with this post: https://mcmap.net/q/98632/-the-forked-vm-terminated-without-saying-properly-goodbye-vm-crash-or-system-exit-called For me, 120 was not high enough, but hey, it is a timeout value you do not want to run into. So I set it to 1200. With seems a a lot. And no unit tests run for that long. No where near actually. We have over 1000 tests in about 4 minutes total. So I would think 120 should have worked, – Chungchungking

Answer 3

If anyone is including a custom argLine argument, you must reconsider because it is likely the source of your issues with the memory allocation.

For Example (I used to have):

<argLine>XX:MaxPermSize=4096m ${argLine}</argLine>

Now I use hard specified values:

<argLine>-Xmx1024m -XX:MaxPermSize=256m</argLine>

For whatever reason, Applications that integrate with Surefire such as Jacoco, dont request enough memory to coexist with the testing that happens at build time.

(more information can also be found at this s.o. question (1))

(1)-maven jacoco: not generating code coverage report

Gilbertogilbertson

Answer 4

Writing here the strategy that i used to help others who are stuck with this problem.

It is possible to leverage the SecurityManager to throw an exception when System.exit() is called. you can then examine the stack trace to see exactly who called exit(). This is especially useful if the call to exit was made from code hidden inside one of the jars you are depending on and not from your own code.

private static void forbidSystemExitCall() {
    final SecurityManager securityManager = new SecurityManager() {
        public void checkPermission( Permission permission ) {
            if( permission.getName().startsWith("exitVM") ) {
                throw new RuntimeException("Something called exit ") ;
            }
        }
    } ;
    System.setSecurityManager( securityManager ) ;
}  

Mana

For thread per class strategy we ca add @BeforClass on this code placed in a common sub test class. Any clue to run this code in the same JVM than the one of Surefire for other loading strategy? – Wham

I don't understand your question. Is your intention to run the @BeforeClass as a part of the main maven build instead of form the surefire fork? if that is the case i don't think this is possible. – Mana

How do you install that for all tests? In my case, a random test out of ~1000 fails. – Klagenfurt

Answer 5

Following Maven Surefire documentation, you can execute the forked VM in debug mode, if it always fails. You can then debug your code until it exit.

Provided

thats a good approach if the problem is predictable. in case of a problem occurring randomly -> this approach is not practical – Mana

Answer 6

FWIW, I have experienced this error when the JVM ran out of memory during the maven build. On linux this was detected by OOM killer, which ended up generating kernel messages like Aug 28 20:53:27 ip-xxx-xxx-xxx-xxx kernel: [248686.775455] java invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0.

I guess on a mac you'd just want to monitor your memory usage with ActivityMonitor.

Poulin

Answer 7

What you may want to check is the argline setting for surefire or failsafe in build/pluging/plugin config in your pom. I had something improper in there that caused the forked vm to fail (Ironically, i put maven.failsafe.debug in there to help debug an earlier forking crash).

Amish

In my case argline was not getting set for failsafe plugin. – Brickbat

Answer 8

JVM options that can help:

-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=c:/dumps/

Note: You can use forward slashes.
Note 2: Make sure the folder exists and that the process has write permission. On recent windows systems, C:\ is write protected.
Note 3: Make sure you have enough free space to write the dump. The Java 9 documentation mentions that the system's temp folder will be used when the disk is full.

If there is no dump file, then you JVM didn't run out of memory.

The next option is -XX:ErrorFile= which allows to the JVM to log fatal errors.

-XX:+ShowMessageBoxOnError shows an error dialog if the JVM crashes.

Note: You can change the flags of a running JVM using the jinfo command.

Pass these options to Maven Surefire via the argLine option:

<build>
    <plugins>
        <plugin>
            <artifactId>maven-surefire-plugin</artifactId>
            <configuration>
                <!-- -XX:HeapDumpPath=C:\ -XX:+ShowMessageBoxOnError  -->
                <argLine>@{argLine} -XX:+HeapDumpOnOutOfMemoryError -Xmx1g -XX:HeapDumpPath=H:/dumps/ -XX:ErrorFile=H:/dumps/ -XX:+ShowMessageBoxOnError</argLine>
            </configuration>
        </plugin>
    </plugins>
</build>

The strange @{argLine} at the beginning allows other plugins like Jacoco to inject their options. For this to work, you need to add an empty property:

<properties>
    <argLine></argLine> <!-- Fallback when Jacoco isn't active. -->
</properties>

You can verify that it works when the error happens: Maven will then dump the whole command line used to start the forked JVM.

Klagenfurt

Answer 9

Changing the configuration of the plugin should resolve the problem:

<project>
  [...]
  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>3.0.0-M1</version>
        <configuration>
          <useSystemClassLoader>false</useSystemClassLoader>
        </configuration>
      </plugin>
    </plugins>
  </build>
  [...]
</project>

As suggested in this post and in the main documentation of the plugin.

Southward

Answer 10

If you are using Visual Studio Code

I did not do any changes in the pom.xml file or updated any dependency versions
Adding this line to settings.json of Visual Studio Code solved the problem.

"maven.executable.options": "-DforkCount=0",

Lymphangial

Answer 11

Consider updating the jacoco-maven-plugin to 0.8.3

e.g

<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>0.8.3</version>
<configuration>
<excludes>
<exclude>**/schemas/**/*</exclude>
</excludes>
<destFile>${sonar.jacoco.reportPaths}</destFile>
</configuration>

Xiphoid

Answer 12

I just delete all the maven repository and run maven clean install. Then the issue is gone.

Demented