Reactive Redis (Lettuce) always publishing to single thread

Question

Im using Spring Webflux (with spring-reactor-netty) 2.1.0.RC1 and Lettuce 5.1.1.RELEASE.

When I invoke any Redis operation using the Reactive Lettuce API the execution always switches to the same individual thread (lettuce-nioEventLoop-4-1).

That is leading to poor performance since all the execution is getting bottlenecked in that single thread.

I know I could use publishOn every time I call Redis to switch to another thread, but that is error prone and still not optimal.

Is there any way to improve that? I see that Lettuce provides the ClientResources class to customize the Thread allocation but I could not find any way to integrate that with Spring webflux.

Besides, wouldn't the current behaviour be dangerous for a careless developer? Maybe the defaults should be tuned a little. I suppose the ideal scenario would be if Lettuce could just reuse the same event loop from webflux.

I'm adding this spring boot single class snippet that can be used to reproduce what I'm describing:

@SpringBootApplication
public class ReactiveApplication {
    public static void main(String[] args) {
        SpringApplication.run(ReactiveApplication.class, args);
    }
}

@Controller
class TestController {

    private final RedisReactiveCommands<String, String> redis = RedisClient.create("redis://localhost:6379").connect().reactive();

    @RequestMapping("/test")
    public Mono<Void> test() {
        return redis.exists("key")
            .doOnSubscribe(subscription -> System.out.println("\nonSubscribe called on thread " + Thread.currentThread().getName()))
            .doOnNext(aLong -> System.out.println("onNext called on thread " + Thread.currentThread().getName()))
            .then();
    }

}

If I keep calling the /test endpoint I get the following output:

onSubscribe called on thread reactor-http-nio-2
onNext called on thread lettuce-nioEventLoop-4-1

onSubscribe called on thread reactor-http-nio-3
onNext called on thread lettuce-nioEventLoop-4-1

onSubscribe called on thread reactor-http-nio-4
onNext called on thread lettuce-nioEventLoop-4-1

Adit

Answer 1

That's an excellent question!

The TL;DR;

Lettuce always publishes using the I/O thread that is bound to the netty channel. This may or may not be suitable for your workload.

The Longer Read

Redis is single-threaded, so it makes sense to keep a single TCP connection. Netty's threading model is that all I/O work is handled by the EventLoop thread that is bound to the channel. Because of this constellation, you receive all reactive signals on the same thread. It makes sense to benchmark the impact using various reactive sequences with various options.

A different usage scheme (i.e. using pooled connections) is something that changes directly the observed results as pooling uses different connections and so notifications are received on different threads.

Another alternative could be to provide an ExecutorService just for response signals (data, error, completion). In some scenarios, the cost of context switching can be neglected because of the removing congestion in the I/O thread. In other scenarios, the context switching cost might be more notable.

You can already observe the same behavior with WebFlux: Every incoming connection is a new connection, and so it's handled by a different inbound EventLoop thread. Reusing the same EventLoop thread for outbound notification (that one, that was used for inbound notifications) happens quite late when writing the HTTP response to the channel.

This duality of responsibilities (completing a command, performing I/O) can experience some gravity towards a more computation-heavy workload which drags performance out of I/O.

Additional resources:

• Investigate on response thread switching #905.

Brynnbrynna