Spring WebClient reactor.netty.internal.shaded.reactor.pool.PoolAcquireTimeoutException

Question

I am using Spring WebClient to call a rest service. The code for the post call as mentioned below.

Mono<ClientResponse> response = client.post()
                        .uri(uriBuilder -> uriBuilder.build())
                        .headers(httpHeaders -> httpHeaders.setAll(getHeaders()))
                        .body(BodyInserters.fromPublisher(Mono.just(message), String.class))
                        .exchange();
                response.subscribe(clientResponse -> {
                   System.out.println(clientResponse.statusCode());
                });

I am getting the following exception after posting continuously for a while (after posting 2-3 million requests in 5 mins).

 [     parallel-3] r.c.s.Schedulers                         : Scheduler worker in group main failed with an uncaught exception

reactor.core.Exceptions$ErrorCallbackNotImplemented: reactor.netty.internal.shaded.reactor.pool.PoolAcquireTimeoutException: Pool#acquire(Duration) has been pending for more than the configured timeout of 45000ms
Caused by: reactor.netty.internal.shaded.reactor.pool.PoolAcquireTimeoutException: Pool#acquire(Duration) has been pending for more than the configured timeout of 45000ms
    at reactor.netty.internal.shaded.reactor.pool.AbstractPool$Borrower.run(AbstractPool.java:317) ~[reactor-netty-0.9.0.RELEASE.jar!/:0.9.0.RELEASE]
    Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
    |_ checkpoint ⇢ Request to POST https://myrest-service-c12-1-lb-125370128.us-west-2.elb.amazonaws.com/antenna [DefaultWebClient]
Stack trace:
        at reactor.netty.internal.shaded.reactor.pool.AbstractPool$Borrower.run(AbstractPool.java:317) ~[reactor-netty-0.9.0.RELEASE.jar!/:0.9.0.RELEASE]
        at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68) [reactor-core-3.3.0.RELEASE.jar!/:3.3.0.RELEASE]
        at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28) [reactor-core-3.3.0.RELEASE.jar!/:3.3.0.RELEASE]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:834) [?:?]

It looks like the pool is exhausted and I need to limit the number of requests. Could some one help me in solving this issue. Thanks in advance.

Centi

Why do you think it's because the pool is exhausted? From the exception message, the error can be caused by the server side as well. have you checked server side log? – Proscribe

There is no error in the server logs. I think because the client is pushing millions of messages, the Webclient is running out of connections. – Centi

This error stopped when I throttle the number of requests from the client end. – Centi

what kind of ConnectionProvider are you using when create your http client? at first I used a fixed connection provider and have the same issue with you. Now I change to an elastic provider which can get more connections dynamically. After changing, I don't have this PoolAcquireTimeoutException so maybe you can try that as well – Proscribe

I was simply using.. WebClient.builder().clientConnector(new ReactorClientHttpConnector(getHttpClient())).baseUrl(url).build(); I updated it to... ConnectionProvider elasticPool = ConnectionProvider.elastic("ElasticPool"); HttpClient httpClient = HttpClient.create(fixedPool); WebClient.builder().clientConnector(new ReactorClientHttpConnector(httpClient)).baseUrl(url).build(); It simply hanged after posting 5000+ messages. – Centi

When I used the fixed pool with higher number of connections, I experienced the same above issue... ConnectionProvider fixedPool = ConnectionProvider.fixed("fixedPool", 10000); After I changed the number of connections from 10000 to 1000, it started posting, but had the actual issue (PoolAcquireTimeoutException) came again. After experimented with different number of connections, I got settled with 1000 and request throttling. – Centi

Answer 1

Seems to be a bug in reactor-netty that has recently been fixed: https://github.com/reactor/reactor-netty/issues/1012

New