can vert.x event bus replace the need for Kafka?

Question

I am evaluating the vert.x framework to see if I can reduce the Kafka based communications between my microservices developed using spring boot.

The question is: Can I replace

1. Kafka with vert.x event bus and
2. spring boot microservices with vert.x based verticles

Alee

Answer 1

To answer quickly, I would say it depends on your needs.

Yes, the eventbus can be a good way to handle natively communication between microservices verticles using an asynchronous and non-blocking paradigm.

But in some cases you could need:

• to handle some common enterprises patterns like replay mechanisms, persistence of messages, transactional reading
• to be able to process some kind of messages in a chronological order
• to handle communication between multiples kind of microservices that aren't all written with the same framework/toolkit or even programming language
• to handle reliability, resilience and failure recovery when all your consumers/microservices/verticles are died
• to handle dynamic horizontal scalability and monitoring of your consumers/microservices/verticles
• to be able to work with a single cluster deployed in multi-datacenters and multi-regions

In those cases I'd prefer to choose Apache Kafka over the native eventbus or an old fascioned JMS compliant system.

It's not forbidden to use both eventbus and kafka in the same microservices architecture according to your real needs. For example, you could have one kafka consumers group reading a kafka topic to handle scaling, monitoring, failure recovery and reply mechanism and then handle communication between your sub-verticles through the eventbus.

I'll clarify a little bit for the scalability and monitoring part and explain why I think it's more simple to handle that with Kafka over the native eventbus and cluster mode with vert.x : Kafka allow us to know in real time (through JMX metrics and the describe command):

• the "lag" of a topic which corresponds to the number of unread messages
• the number of consumers of each group that are listening a topic
• the number of partitions of a topic affected of each consumers
• i/o metrics

So it's possible to use an ElasticStack or Prometheus+Grafana solution to monitor those metrics and use them to handle a dynamic scalability (when you know that there's a need to increase temporarily the number of consumers for example according to the lag metric and the number of partitions and the cpu/ram/swap metrics of your hosts).

To answer the second question vert.x or SpringBoot my answer will be not very objective but I'd vote for vert.x for its performances on the JVM and especially for its simplicity. I'm a little tired of the Spring factory and its big layers of abstraction that hides a lot of issues under a mountain of annotations triggering a mountain of AOP.

Moreover, In the Java world of microservices, there's other alternatives to SpringBoot like the different implementations of Microprofile (thorntail project for example).

Hydrogenolysis

Answer 2

The event-bus is not persistent. You should use it for fast verticle-to-verticle communications, and more generally to dispatch events where you know that you can loose them if you have some crash.

Kafka streams are persistent, and you should send events there because either you want other (possibly non-Vert.x) applications to consume them, and/or because you want to ensure that these events are not being lost in case of failure.

A reactive (read "scalable and fault-tolerant") Vert.x application typically uses a combination of both the event-bus and some replicable messaging systems like AMQP / Kafka / etc.

Sloane

Answer 3

On the question:

Can I replace spring boot microservices with vert.x based verticles?

Yes, definitely, although the 2 have different programming models.

If you want a more progressive approach and use Spring for structuring your application while using Vert.x for resource efficiency over your I/O and event processing then you can mix them, see https://github.com/vert-x3/vertx-examples/tree/master/spring-examples for examples.

Sloane

from my POV using Spring in a verticle defeats the whole idea of micro services – Skeet

Answer 4

Take a look at the Quarkus framework: in the workshop section you'll find Vert.x and Apache Kafka combined!

Spacesuit

can you pls elaborate on this, does quarkus provide eventbus – Iconolatry

@letsdebugtheissue quarkus provide vert.x and make easier to use it with GraalVM, so yes it provide eventbus by providing vert.x. It also provide extension to make integrations with Kafka easier (and when you're using kafka, you won't need the eventbus anymore most of the time) – Hydrogenolysis