Apache Flink vs Twitter Heron?

Question

There are a lot of questions comparing Flink vs Spark Streaming, Flink vs Storm and Storm vs Heron.

The origin of this question is from the fact that both Apache Flink and Twitter Heron are true stream processing frameworks (not micro-batch, like Spark Streaming). Storm has been decommissioned by Twitter last year and they're using Heron instead (which is basically Storm reworked).

There are nice presentations by Slim Baltagi on Flink and Flink vs Spark: https://www.youtube.com/watch?v=G77m6Ou_kFA

Nice research by Ilya Ganelin on various streaming frameworks: https://www.youtube.com/watch?v=KkjhyBLupvs

Pretty interesting thoughts on Flink vs Storm: What is/are the main difference(s) between Flink and Storm?

But I haven't seen any comparison of new Storm/Heron vs Apache Flink.

Both of the projects are pretty young, both support using previously written Storm applications and many other things. Flink is more fitting into Hadoop ecosystem, Heron is more into Twitter based ecosystem stack.

Any thoughts?

Pancreatotomy

Did you have a look Twitter's paper about Heron? It describes the main differences between both systems. Keep in mind though, that Storm made large progress since that paper was published: dl.acm.org/citation.cfm?id=2742788 If you understand the differences between Storm and Heron and know something about Flink, you see how it differs from Flink. – Fondness

Answer 1

All of the points in the referenced article comparing Apache Flink and Apache Storm also apply to Twitter's Heron. Heron provides exactly the same type of semantics and functionality as Storm. Heron is really best understood simply as a re-implementation of Storm that better fits Twitter's operational requirements.

Tuddor

Thanks. And yes, that is true, but Heron is optimized, it may support features such as dynamic topology modification and scaling, which might be superior to Flink, which doesn't support that yet. Exactly the fact that Heron is a re-implemented Storm is interesting, because it adds new functionality that changes the perspective on existing frameworks. – Pancreatotomy

As of Storm 1.0, the system got mayor improvements; thus, not everything is true any more. The same holds for Flink... You should always compare different versions, or at least keep in mind, that SO answers date-out as those project move very fast. – Fondness

Sorry. To clarify that's exactly what I'm trying to say. Heron does not add any new major features related to semantics, processing guarantees, dynamic topology modification, dynamic scaling, etc. The improvements are operational in nature but no new major features over Storm and in some ways Storm actually has the more advanced features right now. Heron suits Twitter's operational needs quite well, though. – Tuddor

Answer 2

Heron, Stream processing engine developed by twitter and donated to Apache on 26th FEB 2018. As per Twitter, the throughput is 10–14x higher than that of Storm in all experiments, Similarly latency is 5-15x lower than Storm’s latency.

Other then throughput and latency it provides

• Easy debugging(Every task runs in process-level isolation).
• Handling spikes and congestion(using backpressure mechanism).
• Fully backward compatible with Storm which means only pom file changes required.

https://blog.twitter.com/engineering/en_us/a/2015/flying-faster-with-twitter-heron.html https://apache.github.io/incubator-heron/

Ionize

Links to external resources are encouraged, but please add context around the link so your fellow users will have some idea what it is and why it’s there. Always quote the most relevant part of an important link, in case the target site is unreachable or goes permanently offline. – Emancipation

@Emancipation Thanks for the suggestion. Already mentioned key points from my finding, will be adding more context to it. – Ionize

The links to twitter.github.io/heron are dead. – Olethea