We are building a high-throughput low-latency stream processing application. We are using Apache Kafka both as a messaging platform and as a database.
It seems like the Kafka Streams and Alpakka Kafka frameworks do a lot in common, but Kafka Streams seems to be more "native" to Kafka, while Alpakka allows us to use the power of the Akka framework.
What are the key differences between the two frameworks?
The key difference is that Kafka streams require Kafka topics on the input side and on the output side. With Alpakka you can create processing pipeline for all kind of inputs and outputs not only Kafka topics. Also, akka streams DSL is more powerful (and complicated) than Kafka streams DSL.
By using the Alpakka various types of Sink and Flow (pipelines) can be attached with the build in APIs in Akka Stream. An example can be using Alpakka Kafka and Alpakka Cassandra at the same time to enable automatic publishing messages and saving to databases. The test toolkit in Akka can also make tests consistent across your system.
Well I have experience with both.
For Kafka Streams, achilles heel in my opinion is the datastore it uses, RockDB, which is necessary for KTables and internal State Stores. RockDB is great loading single Values but if you have iterate over things, after a Dataset size of 100 000 the performance degrades significantly. You can change the datastore to something but it is really not that well documented.
It also suffers from the disadvantages of the key/value databases, you can't only ask questions over primary key and there is not wildcard matching or anything and it has no means to implement CQRS pattern
Akka on the other hand is not that newbie friendly, specially if you don't have a grasp of stream processing concepts. But it is not coupled with single persistence of option, it has components to implement CQRS. And Akka developers made lots of brain storming to improve the performance Kafka for stream processing.
Another really important point, while some might say it is not relevant, Kafka Streams has no back pressure mechanism but Alpakka Kafka have it, which can be very critical in some production scenarios, you can convince yourself with this Netflix Blog.
In short, if you want to start quickly with stream processing, start with Kafka Streams but be ready to hit a wall and switch to Alpakka Streams.
If you need examples, I have two blogs about those topics, you can check it those, blog1 blog2
© 2022 - 2024 — McMap. All rights reserved.
