Difference Between Apache Kafka and Camel (Broker vs Integration)

Question

I am trying to understand the differences between something like Kafka and something like Camel. To my understanding Camel would provide much more abstraction for developers without having to worry about changing protocols/systems to some extent. How would Kafka not be able to handle most of what Camel can do now? I am reading through the documentation and it seems like Kafka has been updated/upgraded enough to slightly break away from being a message broker only. I guess my question would really come down to how does Kafka compare to Camel in regards to future proofing systems and where does Kafka fall short of Camel? I am under the impression that Kafka doesn't scale as well as a system grows.

Edit: This is strictly based around messages.The documentation surrounding Camel makes it very clear that it's based around Enterprise Integration Patterns, but the deeper I dive into Kafka documentation the same patterns can be implemented. Am I missing something?

Circumspection

Answer 1

Apache Kafka : Is a streaming processing platform. It is based on massively scalable publish subscribe message queue architecture. There are many other platforms which are based on JMS publish subscribe model, which could do the same(with some exceptions). Some of the most popular are Apache-Activemq, RabbitMq

Apache Camel : Is a message oriented middleware. It has implemented almost all the Enterprise Integration Patterns.

You can use Apache Camel with Apache Kafka. Or you can use Apache Kafka without Apache Camel also.

Infective

So, if a system requires you to have something along the lines of Kafka and you know in the future you will more than likely integrate something else, Camel tied in with Kafka as a component would be a more apt solution? – Circumspection

Personally, I would use camel in any enterprise project. It has implementations for many Design Patterns. It blends very well with any Publish Subscribe model. – Infective

Thank you for replying and your help. One more question, though. Is Kafka "competent" enough to easily handle transformations of messages? – Circumspection

I stumbled across my solution while reading up on NiFi. Go figure. Kafka isn't meant to handle dataflow challenges such as data enrichment. – Circumspection

Apache Kafka definitely does support both transactions and data enrichment. See confluent.io/blog/transactions-apache-kafka and confluent.io/blog/… – Tallahassee

Answer 2

They are two totally different things.

Think about Camel as an interface definition tool where you can define endpoints or channels where messages fly in. But they are abstract. Compare Camel with Spring Integration for instance.

Kafka can provide those messages, so it can implement those abstract channels or endpoints. But so can ActiveMQ and others.

Burglarize

Thank you for the reply. So Camel is meant to be centralized for current and future integration of new systems? I'm assuming you could do the same with Kafka to some extent, but it would require much more work and not be as abstract? – Circumspection

Answer 3

Kafka is a message broker. It is comparable with other message brokers like ActiveMQ, RabbitMQ, Azure Service Bus etc. Camel is an integration middleware. It is more comparable to Apache ServiceMix.

Carrell

Answer 4

Taking a look at the theory of an Event-Driven Architecture https://www.oreilly.com/library/view/software-architecture-patterns/9781491971437/ch02.html we could differentiate two different kinds of Event-driven topologies depending on whether we need an event mediator or not.

1. Message broker. In this category we find Kafka as it doesn't rely on a message mediator. Of course as written on previous answers, we could use Kafka together with a mediator depending on our needs.
2. Message mediator. In this category we find products like Camel. You may see it as a message controller.

Homoiousian