Kafka Consumers in Onion Architecture

Question

I am working with a project following DDD and has consumers to a Kafka Queue. My question is straight forward, where do the consumers reside in the Onion Architecture and Hexagonal Architecture? Are they event handlers or should they be a part of the infrastructure?

I am using Kafka Consumers to listen to change events of other aggregate roots and want to store the data in my current aggregate. Basically replicating the data from one microservice to another.

Lawsuit

Answer 1

The way I see it is:

1. Your aggregates are the Core

2. The message handlers are Use Cases, which use dependencies (like repository interfaces) and the core to execute the business use case.

3. There is infrastructure code that peeks messages from the queue and triggers use cases

With this approach, you can unit test your use cases without worrying about infrastructure and you can replace the whole messaging queue technology. The same approach works for handling API requests. In practice, the only difference is that with the API you can return a response synchronously, and with messaging you can't.

As a practical note, in .NET I use a library called Mediatr to implement and trigger the use cases. In java, I found PipelinR which looks similar at first sight. This type of approach allows you to implement all use cases the same way for all your synchronous and asynchronous usages.

Luetic

Exactly what I was thinking. Thanks – Lawsuit

Answer 2

Think this way: if the same message your application receives from a Kafka Queue should also be received with a RESTful endpoint, would it be any difference between the consumer and the RESTful endpoint from the PoV of their place in the architectural layers? no, it wouldn't, they'd part of the same layer because they do the same: they accept an external message, despite the fact that the communication channel & type is (very) different. According to The Onion Architecture : part 2:

SpeakerController is part of the user interface

and I'd say that the same (i.e. user interface) is for the RESTful endpoint and the Kafka Queue consumer. Both of them would contain no business logic at all but they'll delegate to an Application Service. If the message types are not exactly the same, there might be message-integrity-validation & conversion-to-DTO particular to each other before delegating to the target Application Service.

The ideea is that one could add more communication channels (e.g. command line, web sockets, etc) but as long as the use cases don't change than the entire Application Core remains unchanged because it doesn't depend on the User Interface but the opposite.

Garald