Apache Camel vs Apache Nifi

Question

I am using Apache camel for quite long time and found it to be a fantastic solution for all kind of system integration related business need. But couple of years back I came accross the Apache Nifi solution. After some googleing I found that though Nifi can work as ETL tool but it is actually meant for stream processing.

In my opinion, "Which is better" is very bad question to ask as that depend on different things. But it will be nice if somebody can describe more about the basic comparison between the two and also the obvious question, when to use what.

It will help to take decision as per my current requirement, which will be the good option in my context or should I use both of them together.

Chekiang

Answer 1

The biggest and most obvious distinction is that NiFi is a no-code approach - 99% of NiFi users will never see a line of code. It is a web based GUI with a drag and drop interface to build pipelines.

NiFi can perform ETL, and can be used in batch use cases, but it is geared towards data streams. It is not just about moving data from A to B, it can do complex (and performant) transformations, enrichments and normalisations. It comes out of the box with support for many specific sources and endpoints (e.g. Kafka, Elastic, HDFS, S3, Postgres, Mongo, etc.) as well as generic sources and endpoints (e.g. TCP, HTTP, IMAP, etc.).

NiFi is not just about messages - it can work natively with a wide array of different formats, but can also be used for binary data and large files (e.g. moving multi-GB video files).

NiFi is deployed as a standalone application - it's not a framework or api or library or something that you integrate in to something else. It is a fully self-contained, realised application that is fully featured out of the box with no additional development. Though it can be extended with custom development if required.

NiFi is natively clustered - it expects (but isn't required) to be deployed on multiple hosts that work together as a cluster for performance, availability and redundancy.

So, the two tools are used quite differently - hopefully that helps highlight some of the key differences

Upraise

Thanks for the answer. I dont have much idea about Nifi, but as per the answer it seems that Nifi can do anything. Then for which scenarios one should prefer Camel, over Nifi? – Chekiang

Camel is a framework that you could integrate in to your own application to handle integrations with external message-oriented applications - so if you are building your own, self-contained application that works exclusively with messages, then Camel would be a good fit. NiFi is typically deployed as part of a wider architecture. – Upraise

Answer 2

It's true that there is some functional overlap between NiFi and Camel, but they were designed very differently:

• Apache NiFi is a data processing and integration platform that is mostly used centrally. It has a low-code approach and prefers configuration.
• Apache Camel is an integration framework which is mostly used in distributed solutions. Solutions are coded in Java. Example solutions are adapters, flows, APIs, connectors, cloud functions, and so on.

They can be used very well together. Especially when using a message broker like Apache ActiveMQ or Apache Kafka.

An example: A Java application is enhanced with Camel to send messages to Kafka. In NiFi the first step is consuming those messages from Kafka. Then in the NiFi flow, the message is changed in various steps. In the middle, the message is put on another Kafka topic. A Camel function (Camel K) in the cloud does various operations on the message, when it's finished it puts the messages on a Kafka topic. The message goes through a NiFi flow, which calls for an API created with Camel.

In a blog, I wrote in detail about the various ways to combine Camel and Nifi:

https://raymondmeester.medium.com/using-camel-and-nifi-in-one-solution-c7668fafe451

Lysimeter

Thanks for the answer, I appreciate "platform and framework" concept. – Chekiang