How to Properly Close Raw RestClient When Using Elastic Search 5.5.0 for Optimal Performance?

Question

Am using a Spring Boot 1.5.4.RELEASE Microservice to connect to an ElasticSearch 5.5.0 instance using the low level Rest Client that ElasticSearch provides.

pom.xml

<parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>1.5.4.RELEASE</version>
</parent>

<dependencies>
    <!-- Spring -->
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>

    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-test</artifactId>
        <scope>test</scope>
    </dependency>

    <!-- Elasticsearch -->
    <dependency>
        <groupId>org.elasticsearch</groupId>
        <artifactId>elasticsearch</artifactId>
        <version>5.5.0</version>
    </dependency>

    <dependency>
        <groupId>org.elasticsearch.client</groupId>
        <artifactId>transport</artifactId>
        <version>5.5.0</version>
    </dependency>

    <!-- Apache Commons -->
    <dependency>
        <groupId>org.apache.commons</groupId>
        <artifactId>commons-lang3</artifactId>
        <version>3.6</version>
    </dependency>

    <!-- Jackson -->
    <dependency>
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-core</artifactId>
        <version>2.8.9</version>
    </dependency>

    <dependency>
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-databind</artifactId>
        <version>2.8.9</version>
    </dependency>

    <dependency>
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-annotations</artifactId>
        <version>2.8.9</version>
    </dependency>

    <!-- Log4j -->
    <dependency>
        <groupId>log4j</groupId>
        <artifactId>log4j</artifactId>
        <version>1.2.17</version>
    </dependency>

    <!-- JUnit -->
    <dependency>
        <groupId>junit</groupId>
        <artifactId>junit</artifactId>
        <version>4.11</version>
        <scope>test</scope>
    </dependency>

    <!-- Swagger -->
    <dependency>
        <groupId>io.springfox</groupId>
        <artifactId>springfox-swagger2</artifactId>
        <version>2.6.1</version>
        <scope>compile</scope>
    </dependency>

    <dependency>
        <groupId>io.springfox</groupId>
        <artifactId>springfox-swagger-ui</artifactId>
        <version>2.6.1</version>
        <scope>compile</scope>
    </dependency>
</dependencies>

Everything is setup correctly but after a bunch of hits, client apps were reporting an HTTP 500 error and this is what appeared in the log files:

java.io.IOException: Too many open files
        at sun.nio.ch.IOUtil.makePipe(Native Method) ~[na:1.8.0_141]
        at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:65) ~[na:1.8.0_141]
        at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36) ~[na:1.8.0_141]
        at java.nio.channels.Selector.open(Selector.java:227) ~[na:1.8.0_141]
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.<init>(AbstractMultiworkerIOReactor.java:142) ~[httpcore-nio-4.4.5.jar!/:4.4.5]
        at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.<init>(DefaultConnectingIOReactor.java:79) ~[httpcore-nio-4.4.5.jar!/:4.4.5]
        at org.apache.http.impl.nio.client.IOReactorUtils.create(IOReactorUtils.java:43) ~[httpasyncclient-4.1.3.jar!/:4.1.3]
        at org.apache.http.impl.nio.client.HttpAsyncClientBuilder.build(HttpAsyncClientBuilder.java:666) ~[httpasyncclient-4.1.3.jar!/:4.1.3]
        at org.elasticsearch.client.RestClientBuilder.createHttpClient(RestClientBuilder.java:202) ~[rest-5.5.0.jar!/:5.5.0]
        at org.elasticsearch.client.RestClientBuilder.build(RestClientBuilder.java:180) ~[rest-5.5.0.jar!/:5.5.0]
        at com.myapp.controller.SearchController.getSearchQueryResults(SearchController.java:94) ~[classes!/:1.0]

Inside SearchController (the second line after the // comment is line 94):

@RestController
@RequestMapping("/api/v1")
public class SearchController {

    @RequestMapping(value = "/search", method = RequestMethod.GET, produces="application/json" )
    public ResponseEntity<Object> getSearchQueryResults(@RequestParam(value = "criteria") String criteria) throws IOException {

        // Setup HTTP Headers
        HttpHeaders headers = new HttpHeaders();
        headers.add("Content-Type", "application/json");

        // Setup RestClient
        RestClient restClient = RestClient.builder(new HttpHost("localhost", 9200))
        .setRequestConfigCallback(new RestClientBuilder.RequestConfigCallback() {
            @Override
            public RequestConfig.Builder customizeRequestConfig(RequestConfig.Builder requestConfigBuilder) {
                return requestConfigBuilder.setConnectTimeout(5000).setSocketTimeout(60000);
            }
        }).setMaxRetryTimeoutMillis(60000).build();

        // Setup query and send and return ResponseEntity...

    }
}

Its obvious that what it was never closed after calling the restClient.performRequest() method...

So, I put this into my code:

Response response = null;
try {
   // Submit Query and Obtain Response
   response = restClient.performRequest("POST", endPoint,  Collections.singletonMap("pretty", "true"), entity);
}
catch (IOException e) {
   LOG.error("\n\n\tException: " + e + "\n\n");
   e.printStackTrace();
}
finally {
   restClient.close();
}

Read on Elastic Search's documentation that the RestClient class is thread-safe...

Also, read about the restClient.performRequestAsync() method but am somewhat inexperienced with threads and the description inside the documentation is vague.

Question(s):

1. Is my solution the best way to handle and close a bunch of socket resources?

2. Would appreciate if someone could show me a better way to use the low level RestClient with Elastic Search in sense that it won't cause the same issue with the socket resources not being freed resulting in an HTTP 500. Should I be using restClient.performRequestAsync? Could someone please provide an example?

Thank you for taking the time to read this...

Redroot

I think restClient.close() in finally block should do the trick. Isn't it working? – Stocker

Can you confirm that you're not using the spring boot starters for Elasticsearch? – Hotpress

The rest client should be created only once in a configuration bean, injected as a dependency into your controllers, and that's it. You don't need to create a new instance for each request. – Hotpress

@Val, I updated the post to give you the full pom.xml file dependencies which should answer your question regarding Spring Boot Starters. I am using Elastic Search's Raw Rest Client. Can you show me an example of the configuration bean via code? By the way, I am not creating a new instance (its the Builder pattern) I am not using the "new" Java keyword. I would really appreciate if you could provide code. There might be 1000s of mobile apps (both iOS & Android) which will be creating a single connection to this Spring Boot Microservice. So performance is critical. As is non-blocking. – Redroot

Where and when do you create your restClient. You say "inside SearchController", but it's not clear when. – Hotpress

@HatimStovewala, It hasn't failed yet with the restClient.close() in a the finally block, thanks. Like I told Val, that since the target clients are native apps that run on iOS and Android (and it might be thousands of connections) to this Spring Boot Microservice should I be using threads or restClient.performRequestAsync()? Sorry - need the best way to accommodate these numerous mobile clients. – Redroot

@Val, I just edited my post and displayed how I am using the raw Rest Client inside my SpringBoot Microservice. How can I setup it up to accommodate 1000s of single connections from mobile devices? Is this try / catch / finally good enough or do I need to use threads or the restClient.performRequestAsync() method call? If you can provide me code regarding the Configuration Bean along with dependency injection, I would be very grateful. – Redroot

Answer 1

It's is not a good practice to create a RestClient on every single request. You should create a single instance via a configuration bean like the one below:

@Configuration
public class ElasticsearchConfig {

    @Value("${elasticsearch.host}")
    private String host;

    @Value("${elasticsearch.port}")
    private int port;

    @Bean
    public RestClient restClient() {
        return RestClient.builder(new HttpHost(host, port))
        .setRequestConfigCallback(new RestClientBuilder.RequestConfigCallback() {
            @Override
            public RequestConfig.Builder customizeRequestConfig(RequestConfig.Builder requestConfigBuilder) {
                return requestConfigBuilder.setConnectTimeout(5000).setSocketTimeout(60000);
            }
        }).setMaxRetryTimeoutMillis(60000).build();
    }
}

And then in your SearchController class you can inject it like this (and also add a cleanup method to close the restClient instance when your container goes down):

@RestController
@RequestMapping("/api/v1")
public class SearchController {

    @Autowired
    private RestClient restClient;

    @RequestMapping(value = "/search", method = RequestMethod.GET, produces="application/json" )
    public ResponseEntity<Object> getSearchQueryResults(@RequestParam(value = "criteria") String criteria) throws IOException {

        // Setup HTTP Headers
        HttpHeaders headers = new HttpHeaders();
        headers.add("Content-Type", "application/json");

        // Setup query and send and return ResponseEntity...

        Response response = this.restClient.performRequest(...);

    }

    @PreDestroy
    public void cleanup() {
        try {
            logger.info("Closing the ES REST client");
            this.restClient.close();
        } catch (IOException ioe) {
            logger.error("Problem occurred when closing the ES REST client", ioe);
        }
    }

}    

Hotpress

Thank you for such a well designed approach! Would I still have to use restClient.close(); inside a finally { } or does the cleanup() method do that? The reason I got errors before was because I wasn't invoking restClient.close(). Should I include the finally after performRequest() and also keep the cleanup() method? – Redroot

No, you can keep the rest client open all the time and only close it in the cleanup method. Try it out an you'll see ;-) – Hotpress

It happend again, this time using your solution! It seemed like the RestClient never closed its connection to begin with. I created a new post with my findings here: #46560012 – Redroot

I'll have a look – Hotpress

@Hotpress one question, if the rest client should not be initialized on every request what if the connection established to the machine breaks due to some reason will all the subsequent requests then fail or the client will create a new connection. – Ligetti

@UmangPachaury note that this thread is over 7 years old, so I would expect things to have changed quite a bit on the client side in the meantime :-) – Hotpress