Why would someone run Spark / Flink on Tez?

Question

In the Tez paper from Saha et al., the following modular architecture of Hadoop 2 with Tez is shown:

Why would someone run Spark/Flink on Tez?

What are the advantages? Better utilization of YARN?

Tablet

Have you tried contacting the authors directly? – Madonia

Answer 1

If I understand correctly, running spark on tez could theoretically lead to better a better DAG. Which could for instance be applied to machine learning iterations.

The relevant paragraph is presented below.

We were able to encode the post-compilation Spark DAG into a Tez DAG and run it successfully in a YARN cluster that was not running the Spark engine service. User defined Spark code is serialized into a Tez processor payload and injected into a generic Spark processor that deserializes and executes the user code. This allows unmodified Spark programs to run on YARN using Spark’s own runtime operators ... Tez sessions also enable Spark machine learning iterations to run efficiently by submitting the per-iteration DAGs to a shared Tez session. This work is an experimental prototype and not part of the Spark project

That being said, it appears that this combination has never been implemented outside an experimental setting, so even if there are decent reasons for combining Tez with tools like Spark, that is not going to help any projects at this point.

Also, my personal expectation is that unless you have very specific workloads, I would be suprised if a Tez DAG significantly outperforms the normal Spark DAG.

Wormseed