What is the difference between AWS Elastic MapReduce and AWS Kinesis Data Analytics?

Question

I'm executing a Flink Job with this tools. I think both can do exactly the same with the proper configuration. Does Kinesis Data Analytics do something that EMR can not do or vice versa?

Amazon Kinesis Data Analytics is the easiest way to analyze streaming data, gain actionable insights, and respond to your business and customer needs in real time.

Amazon Elastic Map Reduce provides a managed Hadoop framework that makes it easy, fast, and cost-effective to process vast amounts of data across dynamically scalable Amazon EC2 instances. You can also run other popular distributed frameworks such as Apache Spark, HBase, Presto, and Flink in EMR.

Brosine

you got me.... but I think that Kinesis is for real-time jobs and mapreduce is for jobs where you can wait...not sure though. – Orit

I've used both briefly and I think it's really more about your ecosystem and requirements. EMR is more hands-on as it is just a managed Hadoop cluster with Flink pre-installed. Kinesis offers more of a managed Flink deployment, with less environment configuration needed. EMR == managed Flink-cluster, Kinesis == managed single Flink job. If you're just getting started w/ Flink, don't already use Hadoop, and have to use one of the AWS offerings, I'd choose Kinesis and then move to EMR if you need more configuration options as you grow. – Pushball

Answer 1

The major difference is maintainability and management from your side.

If you want more independent management and more control then I would say go for AWS EMR. Where its your responsibility to manage the EMR infrastructure as well as the Apache Flink cluster in it.

But if you want less control and more focus on application development and you need to deliver faster(tight deadline) then KDA is the way to go. Here AWS provides all the bells and whistles you need for running your application. This also easily sets up with AWS s3 as code source and provides a bare minimum Configuration Management using the UI.

It scales automatically as well.(Need to understand KCU though).

It provides the same Flink dashboard where you can monitor your application and AWS Cloudwatch integration for debugging your application.

Please go through this nice presentation and let me know it that helps.

Please let me know.

https://www.youtube.com/watch?v=c_LswkrwOvk

Croze

What is KCU in this context? – Echeverria

Answer 2

I will say one major difference between the two is that Kinesis does not provide a hosted Hadoop service unlike Elastic MapReduce (now EMR)

Walke

Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center. – Uranalysis

Answer 3

Got this same question also. This video was helpful in explaining with a real architecture scenario and AWS explanation here tries to explain how Kinesis and EMR can fit together with possible use cases.

Jeffjeffcoat

The second link is irrelevant: It's from 2014 and is for Kinesis Data Streams. – Asperity

And the first link is a general data architecture, not specific to the question on Flink on EMR vs Kinsis DA! – Schneider