Apache Camel vs Apache Kafka [duplicate]

Question

As far as I know, Apache Kafka is asynchronous messaging platform, where as Apache Camel is a platform implementing the enterprise integration patterns.

So, what are the practical differences of Apache Camel and Apache Kafka? We planned to implement the system with Apache Camel, which is relatively easy, but our customer wanted the Apache Kafka instead without rational.

What would be the advantages of choosing Apache Kafka to implement a message queue functionality, which could be implemented with Apache Camel as well? I'm concerned Kafka would just introduce unnecessary overhead to project. Are we comparing apples and oranges?

What we need is straightforward API's to setup and use clustered message queues. Our initial plan was to use Camel to consume/produce on clustered JMS or ActiveMQ queues. How would Kafka make this task easier? The application itself would run on WebLogic server on either case.

The messaging would be point-to-point type, where there are multiple instances of same service running, but only one instance should process the message and emit the result according to load balancing policy. Message queues are also clustered, so neither failure of service instance or queue instance is SPOF.

Firstly

Answer 1

Camel and Kafka are totally different things. In many use cases, camel is just used as a client of kafka/activemq/... .

Kafka and activemq are similar, but also different things, refer What is the difference between Apache kafka vs ActiveMQ. Kafka has higher throughput, and data always on disk, so a little more reliable than activemq.

Kafka is usually used as real-time data streaming, and in general activemq is mainly used for integration between applications, the book says so. But in most real world cases ,kafka and activemq can replace each other easily.

Colporteur

Answer 2

It is very hard to compare those two. They are not covering the same areas of work, but exist some systems, where you can replace one by the other.

So very shortly.

Kafka is messaging platform with streaming ability to process messages Apache Kafka.

Camel is ETL framework it can transform messages/events/data from "any" (see endpoint list by Camel) input point and send it to "any" output Apache Camel - Enterprise Integration Patterns.

You may use Camel without Kafka at all, and vice versa. But there are of course possibilities to use succesfully both together.

• Case 1. You process mail and store in PostgreSQL DB. Kafka is useless here.
• Case 2. You process messages from ActiveMQ and send them to Kafka. You may use both.

Alkahest

So, is Kafka suited to communication, where many instances of a service listens for a message, but only one should process it (i.e. remove it from queue after receiving it). So basically this is point-to-point style of communication with clustered services and queues. – Firstly

Not only one consumer may process it. It depends on setup. There are both options: Kafka will act as load balancer (one message goes to one of many consumers) and Kafka will act as replicator (one messages goes to all from defined set of consumers). And it is not point-to-point, as Kafka needs broker. – Loganiaceous

Camel is not ETL. It is EIP implementation – Footboard

@Footboard how you define ETL (Extract, Transform and Load), that you said Camel is not ETL. And BTW I had stated Camel is ETL FRAMEWORK and not ETL itself. – Loganiaceous

@SewerynHabdank-Wojewódzki Camel being an EIP implementation can be used for ETL, but ETL is a very broad term.. And I don't think Camel is advertised as a ETL tool. – Footboard