Spring webflux WebClient logs 'Connection reset by peer'

Question

I have this following code which uses WebClient to make HTTP calls.

        webClient.post()
                 .uri("/users/track")
                 .body(BodyInserters.fromObject(getUserTrackPayload(selection, customAttribute, partyId).toString()))
                 .header(CONTENT_TYPE, APPLICATION_JSON)
                 .retrieve()
                 .onStatus(httpStatus -> !CREATED.equals(httpStatus),
                           response -> response.bodyToMono(String.class)
                                               .flatMap(body -> buildErrorMessage(response.statusCode().value(), body, partyId,
                                                                                  customAttribute)
                                                   .flatMap(e -> Mono.error(new MyException(e)))))
                 .bodyToMono(Object.class)
                 .map(o -> (JsonObject)new Gson().toJsonTree(o))
                 .flatMap(body -> body.get("message") != null && body.get("message").getAsString().equalsIgnoreCase("success")
                                  && body.get("attributes_processed") != null && body.get("attributes_processed").getAsInt() == 1
                     ? Mono.just(body)
                     : buildErrorMessage(CREATED.value(), body.toString(), partyId, customAttribute)
                         .flatMap(e -> Mono.error(new MyException(e))));

I am getting the following logs the first time this code is called after some time (like 10 minutes). But, the call is succeeding with the right output.

io.netty.channel.unix.Errors$NativeIoException: syscall:read(..) failed: Connection reset by peer at io.netty.channel.unix.FileDescriptor.readAddress(..)(Unknown Source)
2019-03-19 03:11:45,625 WARN  [:::] [reactor-http-epoll-8] reactor.netty.http.client.HttpClientConnect : [id: 0x2e3252c0, L:/172.18.0.125:42956 - R:my-endpoint.com/151.101.53.208:443] The connection observed an error

Not sure why these logs are getting generated. When I was using SpringBoot 2.1.0, it was logging in ERROR level, now I upgraded to 2.1.3 version (reactor netty version - 0.8.5) and it is logging in WARN level. Should I be worried about these logs?

Martijn

Any luck with this? I'm kinda facing a similar issue. – Snocat

Answer 1

I have seen similar behavior when the client goes away (bounced) after connecting to it once and the next request fails - then retries.

Try disabling connection pooling when creating the WebClient. Something like this:

@Bean
public WebClient webClient() {
    return WebClient.builder()
             .clientConnector(connector())
             .build();
}

private ClientHttpConnector connector() {
    return new ReactorClientHttpConnector(HttpClient.from(TcpClient.newConnection()));
}

Alcus

will it not affect performance if we disabled connectionpool? – Concerto

Was anyone able to find a solution apart from creating a new pool? – Gabardine

Refer this thread: github.com/reactor/reactor-netty/issues/388 – Concerto

Answer 2

In my case these logs came with an exception in my code and the request I sent not happening, so it is something to be aware of and fix if possible.

In this github thread similar to the one Rajeev commented, there's a similar problem that's resolved by using the following connectionProvider:

connector:ConnectionProvider provider = ConnectionProvider.builder("fixed")
            .maxConnections(500)
            .maxIdleTime(Duration.ofSeconds(20))
            .maxLifeTime(Duration.ofSeconds(60))
            .pendingAcquireTimeout(Duration.ofSeconds(60))
            .evictInBackground(Duration.ofSeconds(120)).build();

this.webClient = WebClient.builder()
            .clientConnector(new ReactorClientHttpConnector(HttpClient.create(provider)))
            .build();

It worked for me, I no longer get this error and my code runs properly. This way you don't have to disable the connection pool.

Sussman