Elasticsearch Rest Client Still Giving IOException : Too Many Open Files

Question

This is a follow up to the solution which was provided to me on this previous post:

How to Properly Close Raw RestClient When Using Elastic Search 5.5.0 for Optimal Performance?

This same exact error message came back!

2017-09-29 18:50:22.497 ERROR 11099 --- [8080-Acceptor-0] org.apache.tomcat.util.net.NioEndpoint   : Socket accept failed

java.io.IOException: Too many open files
    at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) ~[na:1.8.0_141]
    at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) ~[na:1.8.0_141]
    at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) ~[na:1.8.0_141]
    at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:453) ~[tomcat-embed-core-8.5.15.jar!/:8.5.15]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_141]

2017-09-29 18:50:23.885  INFO 11099 --- [Thread-3] ationConfigEmbeddedWebApplicationContext : Closing org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@5387f9e0: startup date [Wed Sep 27 03:14:35 UTC 2017]; root of context hierarchy
2017-09-29 18:50:23.890  INFO 11099 --- [Thread-3] o.s.c.support.DefaultLifecycleProcessor  : Stopping beans in phase 2147483647
2017-09-29 18:50:23.891  WARN 11099 --- [Thread-3] o.s.c.support.DefaultLifecycleProcessor  : Failed to stop bean 'documentationPluginsBootstrapper'

    ... 7 common frames omitted

2017-09-29 18:50:53.891  WARN 11099 --- [Thread-3] o.s.c.support.DefaultLifecycleProcessor  : Failed to shut down 1 bean with phase value 2147483647 within timeout of 30000: [documentationPluginsBootstrapper]
2017-09-29 18:50:53.891  INFO 11099 --- [Thread-3] o.s.j.e.a.AnnotationMBeanExporter        : Unregistering JMX-exposed beans on shutdown
2017-09-29 18:50:53.894  INFO 11099 --- [Thread-3] com.app.controller.SearchController  : Closing the ES REST client

I tried using the solution from the previous post.

ElasticsearchConfig:

@Configuration
public class ElasticsearchConfig {

@Value("${elasticsearch.host}")
private String host;

@Value("${elasticsearch.port}")
private int port;

@Bean
public RestClient restClient() {
    return RestClient.builder(new HttpHost(host, port))
    .setRequestConfigCallback(new RestClientBuilder.RequestConfigCallback() {
        @Override
        public RequestConfig.Builder customizeRequestConfig(RequestConfig.Builder requestConfigBuilder) {
            return requestConfigBuilder.setConnectTimeout(5000).setSocketTimeout(60000);
        }
    }).setMaxRetryTimeoutMillis(60000).build();
}

SearchController:

@RestController
@RequestMapping("/api/v1")
public class SearchController {

    @Autowired
    private RestClient restClient;

    @RequestMapping(value = "/search", method = RequestMethod.GET, produces="application/json" )
    public ResponseEntity<Object> getSearchQueryResults(@RequestParam(value = "criteria") String criteria) throws IOException {

        // Setup HTTP Headers
        HttpHeaders headers = new HttpHeaders();
        headers.add("Content-Type", "application/json");

        // Setup query and send and return ResponseEntity...

        Response response = this.restClient.performRequest(...);
    }

    @PreDestroy
    public void cleanup() {
        try {
            logger.info("Closing the ES REST client");
            this.restClient.close();
        } 
        catch (IOException ioe) {
            logger.error("Problem occurred when closing the ES REST client", ioe);
        }
    }
}    

pom.xml:

<parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>1.5.4.RELEASE</version>
</parent>

<dependencies>
    <!-- Spring -->
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>

    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-test</artifactId>
        <scope>test</scope>
    </dependency>

    <!-- Elasticsearch -->
    <dependency>
        <groupId>org.elasticsearch</groupId>
        <artifactId>elasticsearch</artifactId>
        <version>5.5.0</version>
    </dependency>

    <dependency>
        <groupId>org.elasticsearch.client</groupId>
        <artifactId>transport</artifactId>
        <version>5.5.0</version>
    </dependency>

    <!-- Apache Commons -->
    <dependency>
        <groupId>org.apache.commons</groupId>
        <artifactId>commons-lang3</artifactId>
        <version>3.6</version>
    </dependency>

    <!-- Log4j -->
    <dependency>
        <groupId>log4j</groupId>
        <artifactId>log4j</artifactId>
        <version>1.2.17</version>
    </dependency>
</dependencies>

This makes me think that the RestClient was never explicitly closing the connection, in the first place...

And this is surprising since my Elasticsearch Spring Boot based Microservice is load balanced on two different AWS EC-2 Servers.

That exception occurreded like 2000 times reported by the log file and only in the end did the preDestroy() close the client. See the INFO from the @PreDestroy() cleanup method being logged at the end of the StackTrace.

Do I need to explicitly put a finally clause inside the SearchController and close the RestClient connection explicitly?

It's really critical that this IOException doesn't happen again because this Search Microservice is dependent on a lot of different mobile clients (iOS & Android).

Need this to be fault tolerant and scalable... Or, at the very least, not to break.

The only reason this is in the bottom of the log file:

2017-09-29 18:50:53.894  INFO 11099 --- [Thread-3] com.app.controller.SearchController : Closing the ES REST client

Is because I did this:

kill -3 jvm_pid

Should I keep the @PreDestory cleanup() method but change the contents of my SearchController.getSearchResults() method to reflect something like this:

@RequestMapping(value = "/search", method = RequestMethod.GET, produces="application/json" )
public ResponseEntity<Object> getSearchQueryResults(@RequestParam(value = "criteria") String criteria) throws IOException {
    // Setup HTTP Headers
    HttpHeaders headers = new HttpHeaders();
    headers.add("Content-Type", "application/json");

    // Setup query and send and return ResponseEntity...
    Response response = null;

    try {
        // Submit Query and Obtain Response
        response = this.restClient.performRequest("POST", endPoint, Collections.singletonMap("pretty", "true"), entity);
    } 
    catch(IOException ioe) {
        logger.error("Exception when performing POST request " + ioe);
    }
    finally {
        this.restClient.close();
    }
    // return response as EsResponse();
}

This way the RestClient connection is always closing...

Would appreciate if someone can help me with this.

Semblable

@Semblable please note that your exception is thrown by sun.nio.ch.ServerSocketChannelImpl.accept0() which suggests that - even though the Exception message is the same as in your first post - the cause of the exception is different and potentially nothing to do with Elasticsearch's RestClient. Perhaps you are serving too much traffic incoming to your service, or perhaps it is as simple as examining and upping ulimit in your ECS instances. – Apples

I agree with @diginoise, the problem doesn't seem to come from the RestClient. – Korwin

Answer 1

From my point of view, there are few thing that you are doing wrong but I will go directly to the solution.

I'm not going to write the full solution (in fact, I didn't execute or test anything), but the important is to understand it. Also, it is better if you move all related with data access to another layer. Anyway, this is only an example so the design is not perfect.

Step 1: Import the right library.

Practically the same as your example. I updated the example to use the last client library recommended in version 5.6.2

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>com.acervera</groupId>
  <artifactId>elastic-example</artifactId>
  <version>1.0-SNAPSHOT</version>
  <packaging>jar</packaging>

  <name>elastic-example</name>
  <url>http://maven.apache.org</url>

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <es.version>5.6.2</es.version>
  </properties>

  <parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>1.5.4.RELEASE</version>
  </parent>

  <dependencies>
    <!-- Spring -->
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-web</artifactId>
    </dependency>

    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-test</artifactId>
      <scope>test</scope>
    </dependency>

    <!-- Elasticsearch -->
    <dependency>
      <groupId>org.elasticsearch</groupId>
      <artifactId>elasticsearch</artifactId>
      <version>${es.version}</version>
    </dependency>

<dependency>
  <groupId>org.elasticsearch.client</groupId>
  <artifactId>elasticsearch-rest-high-level-client</artifactId>
  <version>${es.version}</version>
</dependency>

    <!-- Log4j -->
    <dependency>
      <groupId>log4j</groupId>
      <artifactId>log4j</artifactId>
      <version>1.2.17</version>
    </dependency>
  </dependencies>
</project>

Step 2: Create and shutdown the client in the bean factory.

In the bean factory, create and destroy it. You can reuse the same client.

import org.apache.http.HttpHost;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestHighLevelClient;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import javax.annotation.PostConstruct;
import javax.annotation.PreDestroy;
import java.io.IOException;

@Configuration
public class ElasticsearchConfig {


    // Here all init stuff with @Value(....)


    RestClient lowLevelRestClient;
    RestHighLevelClient client;

    @PostConstruct
    public void init() {
        lowLevelRestClient = RestClient.builder(new HttpHost("host", 9200, "http")).build();
        client = new RestHighLevelClient(lowLevelRestClient);
    }

    @PreDestroy
    public void destroy() throws IOException {
        lowLevelRestClient.close();
    }

    @Bean
    public RestHighLevelClient getClient() {
        return client;
    }

}

Step 3: Execute the query using the Java Transport Client.

Use the Java Transport Client to execute the query.

import org.elasticsearch.action.search.SearchRequest;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.client.RestHighLevelClient;
import org.elasticsearch.index.query.QueryBuilders;
import org.elasticsearch.search.SearchHit;
import org.elasticsearch.search.builder.SearchSourceBuilder;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;

import java.io.IOException;

@RestController
@RequestMapping("/api/v1")
public class SearchController {

    @Autowired
    private RestHighLevelClient client;

    @RequestMapping(value = "/search", method = RequestMethod.GET, produces="application/json" )
    public ResponseEntity<Tweet> getSearchQueryResults(@RequestParam(value = "criteria") String criteria) throws IOException {

        // This is only one example. Of course, this logic make non sense and you are going to put it in a DAO
        // layer with more logical stuff
        SearchRequest searchRequest = new SearchRequest();
        SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
        searchSourceBuilder.query(QueryBuilders.matchAllQuery());
        SearchResponse searchResponse = client.search(searchRequest);

        if(searchResponse.getHits().totalHits > 0) {
            SearchHit searchHit = searchResponse.getHits().iterator().next();

            // Deserialize to Java. The best option is to use response.getSource() and Jackson
            // This is other option.
            Tweet tweet = new Tweet();
            tweet.setId(searchHit.getField("id").getValue().toString());
            tweet.setTittle(searchHit.getField("tittle").getValue().toString());
            return ResponseEntity.ok(tweet);
        } else {
            return ResponseEntity.notFound().build();
        }

    }

}

Also, use a bean to build the response.

public class Tweet {

    private String id;

    private String tittle;

    public String getTittle() {
        return tittle;
    }

    public void setTittle(String tittle) {
        this.tittle = tittle;
    }

    public String getId() {
        return id;
    }

    public void setId(String id) {
        this.id = id;
    }

    // Here rest of bean stuff (equal, hash, etc) or Lombok
}

Step: 4

Enjoy Elasticsearch!

Notes: Java REST Client [5.6] » Java High Level REST Client

PS. It is necessary to refactor the example. It is only to understand the way.

Eury

In 5.6, they introduced the high-level REST client, but the low-level Java REST client is available since 5.0. The low-level REST client can also balance queries between all nodes. Finally, I would advise against using the transport client since Elastic is going to deprecate and finally remove the transport client and the Java API: elastic.co/blog/… – Korwin

@Korwin Anyway, the structure is the same. A thread-safe connector reused that must be opened and closed only one time when the application starts or finish. I will update the response this evening. Thx for your point! – Eury

Yes, but the thinking was "don't build something new on something that will go away soon" – Korwin

@Korwin 100% agree. I will update the response this evening. – Eury

I can't use the Transport Client - I need to use the low-level RestClient. Should I move the @PreDestory cleanup() method from my SearchController class to my ElasticsearchConfig class? Should I explicitly close the connection inside my SearchController using a try / catch / finally ? – Semblable

@Korwin - Can you see why the same exception from my original post is occurring again? I thought the PreDestroy cleanup() method would close the HTTP Connections but as you can see that only happens when you kill the app - The exceptions occur without ever going inside the PreDestroy cleanup method. Would really appreciate if someone can help me. – Semblable

Updated to the last RestClient version, as commented @Korwin – Eury

Can't you use the Java High Level REST Client? I updated the example. Check your pom because in your the example you are not using it. – Eury

@Semblable The cleanup() method will only be called when the app goes down and the bean is recycled. It will not be called after each request... and it doesn't need to be. I have the exact same settings in my app and there is no such errors. Are you holding long-lived connections with the clients of your /search API endpoint? – Korwin

@Semblable The best way to find your problem is creating a small proof of concept using, for example, my response code and execute a stress test (you need 5 minutes with JMeter). And share it with us in github. – Eury

@Korwin - "Are you holding long-lived connections with the clients of your /search API endpoint? " How do I check for this? Angel, why use the High Level Rest Client? So, the cleanup needs to go inside ElasticsearchConfig Java file? Can you modify the content in your post to not specify TransportClient? The code has changed but not really the description of the solution in your post. Why use both the HighLevelRestClient and Low Level Rest Client at the same time? I am confused? And I don't see the pom entry for the HighLevel Rest Client. Thanks both of you for trying to help. – Semblable

I'm not using the TransportClient, the pom has the elasticsearch-rest-high-level-client dependency and there is a link in the very bottom of the response to the client API documentation. In version used, the low level API is used to configure the connection. In the next release will be different and will be not necessary to use the low level API. So be sure that you are checking the documentation for the last release and not the one in the master. – Eury

I insist.Create a new small project with the ES stuff to isolate the project. Then, when you are sure that the ES code is ok, start to search in another place. Are you opening files to something like reading files or store data or maybe you created your own Log4J appender or you open your own sockets, etc and you DON'T CLOSE it? Also, check who opened more files. You can have the error in Tomcat, but maybe it is another process who is taking more resources. – Eury

Another point, this message is not the same that the in the another question. The previous error comes from the ES client when trying to open the connection socket. The new one comes from Tomcat, maybe when trying to open a new connection from a request. Use my code to be sure that you are using ES on the right way and then, search in another place. – Eury

@Eury - For your efforts, I awarded you the bounty. Will try this tomorrow or over the weekend and let you and Val know the results. Why did Val recommend putting the PreDestroy in SearchController whereas you recommended put in the ElasticsearchConfig Bean? – Semblable

Thanks. I think that is not good to have too many comments about different things in comments. If you want, we can have a chat in gitter or slack or chat.stackexchange.com or whatever to help you. After that, I can modify the answer to reflect your problem. My contact info is on my site and profile. – Eury

Answer 2

Are you sure that you don't initiate a new (thread pool) connection to the elasticsearch server on every HTTP request? I.e., in line

        Response response = this.restClient.performRequest(...);

Double-check the logs on the elasticsearch server after a single HTTP request. You should try implementing a Singleton pattern without the @Autowired annotation and see if the problem persists.

Birthmark

The low level RestClient is thread safe according to Elasticsearch's docs. What should I look for in the logs? – Semblable

He's using @Autowired already and the RestClient is created once in the configuration bean. – Korwin

Then, is the SearchController instantiated only once? If not, why don't you have restClient as static if it is a singleton? – Birthmark

@Semblable In the application logs, after an HTTP request you should see that a new connection thread pool is initiated. If not, then your application might leak resources in another place. – Birthmark

Answer 3

From your stacktrace, it appears that embedded tomcat(your application container) is not longer able to accept new connection due to too many open files error. From your code, elasticsearch rest client does not seems problematic.

Since you are re-using the single instance of RestClient while servicing your search request, there may not be more more than 30 (org.elasticsearch.client.RestClientBuilder.DEFAULT_MAX_CONN_TOTAL) open connections with ES cluster. So it is unlikely that RestClient it is causing the issue.

Other potential root cause may be your service's consumer are keeping connection open for longer time with your (tomcat) server or they are not closing connection properly.

Do I need to explicitly put a finally clause inside the SearchController and close the RestClient connection explicitly?

No. You shouldn't. Rest client should be closed while bringing down your service(in a @PreDestroy method as you are already doing correctly).

Tardy