Spring REST Service does not shutdown after an OutOfMemoryError

Question

I am experiencing what I consider a strange behaviour for a Java application. I am using Spring Boot 1.4.1 to develop a REST Service. Due to a bug in my code, an invocation to the service results in an OutOfMemoryError.

Surprisingly, the service responds to the request that generates the error with the following message:

{
  "timestamp": 1487862480405,
  "status": 500,
  "error": "Internal Server Error",
  "exception": "java.lang.OutOfMemoryError",
  "message": "No message available",
  "path": "/entity/exportCsv"
}

So far so good. What is more surprisingly is that the REST service does not shutdown after experiencing the Error. Someone said that after an Error the best thing to do is to properly log it and shutdown everything. The presence of an error should mean that the system is in an unrecoverable state.

Why does Spring MVC adopt such a strange policy in case of Error? Is it possible to force the exit from the application (the REST service) after an Error happened?

If you want to reproduce the above use case, just use the following code.

@RestController
@RequestMapping("/entity")
class Controller {
    @RequestMapping(value = "/exportCsv", method = RequestMethod.GET)
    ResponseEntity exportCsv() {
        if (true)
            throw new OutOfMemoryError();
        return null;
    }
}

Thanks to all.

EDIT: if you do think that catching an Error is a normal way to develop Java applications, please, try to have a look to these references:

• When to catch java.lang.Error?
• Why Catching Throwable or Error is bad?
• And many other...

Sanmiguel

I'm not sure what exactly you want to know. You seem to imply that Spring should shut down anything. – Russ

@zeroflagL I am asking if it is possible to force exit of the REST service in case of an OutOfMemoryError. Related to this, I am asking if someone knows why Spring implements the above policy. Thanks. – Sanmiguel

Answer 1

When running your application, you can specify the behavior of your application when an error like this happens with the following arguments:

• -XX:+HeapDumpOnOutOfMemoryError : this will create a dump which can be analysed afterwards. The dump will be located at the location given at -XX:HeapDumpPath=some_path.
• -XX:OnOutOfMemoryError=path_to_some_script.sh : this will run a script (it must be runnable by the same user which runs the application) when the application returns an error as an OutOfMemory
• -XX:OnError=path_to_some_script.sh : same as before, but for more generic exceptions.

Reference: http://www.oracle.com/technetwork/java/javase/clopts-139448.html

Karsten

Sorry, in which way this answer to my question? – Sanmiguel

Why does Spring MVC adopt such a strange policy in case of Error? Spring has a default behavior which can't do more when a JVM error happens; Is it possible to force the exit from the application (the REST service) after an Error happened? Yes, see my answer above. – Karsten

Spring works as any standard JVM application. That is an answer to your question. – Denadenae

So, your solution is to point to a script that will try to kill the Spring applcation in case of OutOfMemotyError? – Sanmiguel

The end of this script should be something like kill or restart. But you can perform whatever analyse/dump tasks before which can be useful to investigate the reasons of the error. – Karsten

@MichałMielec in this case Spring is not working as any other JVM application. In case of an Error a standard JVM application will stop. – Sanmiguel

You can catch any Error in a JVM application. Spring must do it and return a default message when the service is in a degraded mode. – Karsten

What you can do is not equals to what you should do. Many articles state that Errors must not be catch, because you have no guarantees that you will be able to recover the system from them. Look at this SO question for example When to catch java.lang.Error? – Sanmiguel

@SergioLema your solutions cannot work because Spring is intercepting the thrown OutOfMemoryError. So the error will never reach the JVM. Please, update your answer or delete it. – Sanmiguel

Spring Boot catch the OutOfMemoryError and let you know about it, but it also informs the JVM about it, so the JVM arguments I described in my comment are taken into account (I'm currently using them in a production application with Spring Boot and it works fine). – Karsten

Answer 2

I would say that in the world of web apps this is not suprising at all.

Imagine a RestController as a web page. If it contains an error, the web server nor the application that generates the page is expected to stop working. This error is usually a local or temporary problem related to your request. In these situations the web server should instead respond with HTTP status 500.

Spring even has an built-in error handling feature that lets you to handle each specific error differently. See this article by Paul Chapman for more info.

UPDATE: Regarding your inquiry about the OOM error handling: I think you missed how memory management and object allocation in JVM works. See Understand the OutOfMemoryError Exception.

Ducat

I don't agree. The behaviour you described is true for Exceptions. However, an OutOfMemoryError is not a "temporary problem. How the system will respond 500 to the request if there is no memory available at all? – Sanmiguel

Of course it can be temporary - imagine e.g. processing a huge input. There usually is memory available to handle the exception (the GC is still there). – Ducat

recovering from OOM is flimsy at best. Spring in particular seems to get stuck afterwards. It is generally expected for a web server to log and stop when hitting such errors, and be restarted by the managing infrastructure – Quadruple

Answer 3

java.lang.OutOfMemoryError won't stop the Spring Application. If you are running your application in a container then health-check hook would let container manager know that your application is not in right condition so it would automatically dispose the current pod and would create fresh one.

You should always give JAVA_TOOL_OPTIONS as environment variable to use the POD memory effectively. By default your Java Springboot app will use just 25% of maximum memory.

e.g. If you allocate 4 GB memory to your POD then your Java Springboot application will use max 1 GB of RAM and in turn you will not use the allocated resources properly. Always set JAVA_TOOL_OPTIONS environment variable for memory allocations.

Solicitous