java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED

Question

I have a service that is expected to execute requests at ~5 or more requests/min. This service depends on Apache AsyncHttpClient. After every few minutes, the clients hits some condition which causes java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED. All requests to the client start failing with same exception message. After service is restarted, this cycle repeats.

It is really hard to debug this problem as the request execution failure surprisingly does not cause a callback to the failed() method of the AsyncResponse.

From what I could gather, there has been a fix HTTPCORE-370 in HttpCore NIO which solved a similar problem in 4.3.2. I am using the following version -

commons-httpclient-3.1.jar
httpasyncclient-4.1.1.jar
httpcore-4.4.4.jar
httpcore-nio-4.4.4.jar

Yet seeing this problem.

Readily

Rather late, did you find a solution for this problem? – Activism

How can we fix this issue? – Beckham

Answer 1

I've been dealing with this same exception in my application, and I finally found a helpful suggestion from this post - http://httpcomponents.10934.n7.nabble.com/I-O-reactor-status-STOPPED-td29059.html

You can use #getAuditLog() method of the I/O reactor to find out exactly what exception caused it to terminate.

If you keep a reference to your ConnectionManager's IOReactor, you can call this method to get insight into the actual problem:

http://hc.apache.org/httpcomponents-core-4.4.x/httpcore-nio/apidocs/org/apache/http/impl/nio/reactor/AbstractMultiworkerIOReactor.html#getAuditLog()

Turns out I was doing something incredibly stupid in my own code. But I couldn't figure it out until I read the audit log.

Tiffanitiffanie

wo have same problem,can you share you reason? – Hindorff

My issue had to do with my code's business logic. It was a situation where I was trying to stop the same Google Guava StopWatch twice. What you find within the IOReactor's audit log will likely be something very specific to the code you wrote. – Tiffanitiffanie

none of your link is working – Borecole

Answer 2

In my case, using Elasticsearch high level client, this exception is due to esclient.indexAsync(indexRequest,RequestOptions.DEFAULT,null)

I fixed it by add an action listeners in all async requests like this

esclient.indexAsync(indexRequest,RequestOptions.DEFAULT, 
              new ActionListener<IndexResponse>() {

                    @Override

                    public void onResponse(IndexResponse response) {

                    }
                    @Override
                    public void onFailure(Exception e) {

                });

Haemic

With which version of Elasticsearch client is that happening? – Storfer

Answer 3

If you see OutOfMemoryError before this, try this

-XX:MaxDirectMemorySize=512M

See https://issues.apache.org/jira/browse/HTTPASYNC-104

Talented

Indeed, an OutOfMemoryError was the cause in my case. – Owner

Answer 4

We had encountered the same issue and after lots of digging we found that a proper IOReactorExceptionHandler needs to provided to the HttpAsyncClient to avoid this. It's unfortunate that it's not well covered in the documentation.

Below is a snippet of our code where a more robust client builder tries to add the exception handler. Note that IOExceptions would still stop the I/O reactor as they might imply underlying network communication failures. You may adjust according to your unique use cases.

 public RobustCloseableHttpAsyncClientBuilder withDefaultExceptionHandler() {
    return withExceptionHandler(
        new IOReactorExceptionHandler() {

          @Override
          public boolean handle(RuntimeException ex) {
            logger.error(
                "RuntimeException occurs in callback, handled by default exception handler and the I/O reactor will be resumed.",
                ex);
            return true;
          }

          @Override
          public boolean handle(IOException ex) {
            logger.error(
                "IOException occurs in callback, handled by default exception handler and the I/O reactor will be stopped.",
                ex);
            return false;
          }
        });
  }

Read this issue report in elasticsearch on Github for more exposure.

Pneumothorax

Thanks, I've implemented a similar way following github.com/elastic/elasticsearch/issues/… – Rowena