What is/are the main difference(s) between Flink and Storm?

Question

Flink has been compared to Spark, which, as I see it, is the wrong comparison because it compares a windowed event processing system against micro-batching; Similarly, it does not make that much sense to me to compare Flink to Samza. In both cases it compares a real-time vs. a batched event processing strategy, even if at a smaller "scale" in the case of Samza. But I would like to know how Flink compares to Storm, which seems conceptually much more similar to it.

I have found this (Slide #4) documenting the main difference as "adjustable latency" for Flink. Another hint seems to be an article by Slicon Angle that suggest that Flink better integrates into a Spark or HadoopMR world, but no actual details are mentioned or referenced. Finally, Fabian Hueske himself notes in an interview that "Compared to Apache Storm, the stream analysis functionality of Flink offers a high-level API and uses a more light-weight fault tolerance strategy to provide exactly-once processing guarantees."

All that is a bit sparse for me and I do not quite get the point. Can someone explain what problem(s?) with stream processing in Storm is (are?) exactly solved by Flink? What is Hueske referring to by the API issues and their "more light-weight fault tolerance strategy"?

Che

Note that Apache Spark (the focus of the linked question) is not the same as Apache Storm (this question here) - so, no, this is by no means a duplicate. – Che

Answer 1

Disclaimer: I'm an Apache Flink committer and PMC member and only familiar with Storm's high-level design, not its internals.

Apache Flink is a framework for unified stream and batch processing. Flink's runtime natively supports both domains due to pipelined data transfers between parallel tasks which includes pipelined shuffles. Records are immediately shipped from producing tasks to receiving tasks (after being collected in a buffer for network transfer). Batch jobs can be optionally executed using blocking data transfers.

Apache Spark is a framework that also supports batch and stream processing. Flink's batch API looks quite similar and addresses similar use cases as Spark but differs in the internals. For streaming, both systems follow very different approaches (mini-batches vs. streaming) which makes them suitable for different kinds of applications. I would say comparing Spark and Flink is valid and useful, however, Spark is not the most similar stream processing engine to Flink.

Coming to the original question, Apache Storm is a data stream processor without batch capabilities. In fact, Flink's pipelined engine internally looks a bit similar to Storm, i.e., the interfaces of Flink's parallel tasks are similar to Storm's bolts. Storm and Flink have in common that they aim for low latency stream processing by pipelined data transfers. However, Flink offers a more high-level API compared to Storm. Instead of implementing the functionality of a bolts with one or more readers and collectors, Flink's DataStream API provides functions such as Map, GroupBy, Window, and Join. A lot of this functionality must be manually implemented when using Storm. Another difference are processing semantics. Storm guarantees at-least-once processing while Flink provides exactly-once. The implementations which give these processing guarantees differ quite a bit. While Storm uses record-level acknowledgments, Flink uses a variant of the Chandy-Lamport algorithm. In a nutshell, data sources periodically inject markers into the data stream. Whenever an operator receives such a marker, it checkpoints its internal state. When a marker was received by all data sinks, the marker (and all records which have been processed before) are committed. In case of a failure, all sources operators are reset to their state when they saw the last committed marker and processing is continued. This marker-checkpoint approach is more lightweight than Storm's record-level acknowledgments. This slide set and the corresponding talk discuss Flink's streaming processing approach including fault tolerance, checkpointing, and state handling.

Storm also offers an exactly-once, high-level API called Trident. However, Trident is based on mini-batches and hence more similar to Spark than Flink.

Flink's adjustable latency refers to the way that Flink sends records from one task to the other. I said before, that Flink uses pipelined data transfers and forwards records as soon as they are produced. For efficiency, these records are collected in a buffer which is sent over the network once it is full or a certain time threshold is met. This threshold controls the latency of records because it specifies the maximum amount of time that a record will stay in a buffer without being sent to the next task. However, it cannot be used to give hard guarantees about the time it takes for a record from entering to leaving a program because this also depends on the processing time within tasks and the number of network transfers among other things.

Nostoc

Thank you very much indeed! One open point maybe, if I may bother you once more: What is this "adjustable latency" issue about? This seems like it could be pretty relevant given that different application domains will have different requirements in this respect. Can you explain what this implies, at least in terms of Flink? – Che

Sure, I extended my answer and discussed the adjustable latency. Let me know if you have further questions. – Nostoc

Does Flink allow for "hot" changes to the DAG workflow, as one can implement, for example, using Erlang? IE. Can one change the DAG during runtime? – Science

Hot code swap is not possible. However, you can persist the state of an application as a savepoint. The savepoint can be used to start a modified application. This can be done while the original application is still running such that the output can be flipped at some point. Note that apps cannot be arbitrarily modified when resuming from an existing savepoint. – Nostoc

Interesting and huge advantage of Flink is the capability of running Apache Beam with even higher level API. It's one of the most rich and complete runner for Beam. – Hymnody

Answer 2

Adding to the answer of Fabian Hueske:

Flink improves on Storm additionally also in the following ways:

• Backpressure: Flink's streaming runtime is well behaved when different operators run at different speeds, because downstream operators backpressure upstream operators very well though the network layer's manages buffer pools.

• User-defined state: Flink allows programs to maintain custom state in your operators. That state can actually participate in the checkpointing for fault tolerance, providing exactly-once guarantees for custom user-defined state. See this example of a user-defined state machine inside an operator, that is consistently checkpointed together with the data stream.

• Streaming Windows: Stream windowing and window aggregations are a crucial building block for analyzing of data streams. Flink comes with a quite powerful windowing system that supports many types of windows.

Yellowlegs

Regarding your first point, Storm is well-behaved under backpressure as of 1.0 (released Apr 2016) – Spleenful

Storm backpressure can be mitigated using the "spout_max_pending" property. It sets a threshold for the max tuples that can be present in a spout that are pending acknowledgment. Spout will not consume any more tuples going forward until the ack happens. – Disentitle

Answer 3

Disclaimer: I am an employee of Cloudera, a major supporter of Storm and (soon) Flink.

Functional

A lot of good technical points have already been presented. A very short summary of highlights:

• Both Flink and Storm can do per-event processing
• Storm does not appear to support event-time out of the box
• Storm has not lifted SQL support out of the experimental stage

Non-Functional

• Many customers found Storm (too) hard to use
• Storm adoption slowed down, and the community of Flink now appears to be more active than Storm
• Flink still has some catching up to do (e.g. documented examples), but overall it has caught up in nearly every area you might think of

Conclusion

Cloudera has recently announced the deprecation of Storm (in HDP). And simultaneously Flink was announced as its successor.

So, if you have usecases on storm, they will of course continue to work. But for new usecases I would look into Flink or other streaming engines.

Naidanaiditch

Note that as of April there will also no longer be Storm support from Microsoft. – Naidanaiditch

Answer 4

Based on my experience of Storm and Flink. I feel these tools can solve the same problem with different approaches. Every feature of Flink mentioned by @Stephan Ewen can be matched by Storm with internal API (i.e., spolts and bolts) and Trident API now. Somebody claims that Trident is mini-batch style while I think most of the complex apps with state-related or aggregation could only depend on batch-processing with window style. So I just list some main differences here without saying which is better.

• Development style. computing-oriented (e.g., chainable operator) in Flink vs. data-stream-oriented (e.g., addSpolt()/addBolt()) in Storm.
• High-level API. Functions (e.g, Map, Window, Join in Streaming level) in Flink vs. Native Window and Trident in Storm.
• Guaranteed message processing (GMP. i.e., at-exactly-once). Checkpoint with Two-Phase Commit Connector (e.g., KafkaConsumer) in Flink vs. Tuple-tree with the external state machine or Trident in Storm.
• Fault tolerance. Marker-checkpoint in Flink vs. record-level-ACK in Storm.
• Internal architecture. Simple abstraction and relative parallelism (e.g., slot for each thread considered with CPU cores) in Flink vs. Multi-layer abstractions (e.g., slot for each JVM as worker in supervisor and each supervisor can have many workers) in Storm.

Foretoken