what are the options for hadoop on scala

Question

We are starting a big-data based analytic project and we are considering to adopt scala (typesafe stack). I would like to know the various scala API's/projects which are available to do hadoop , map reduce programs.

Annettaannette

Answer 1

Definitely check out Scalding. Speaking as a user and occasional contributor, I've found it to be a very useful tool. The Scalding API is also meant to be very compatible with the standard Scala collections API. Just as you can call flatMap, map, or groupBy on normal collections, you can do the same on scalding Pipes, which you can imagine as a distributed List of tuples. There's also a typed version of the API which provides stronger type-safety guarantees. I haven't used Scoobi, but the API seems similar to what they have.

Additionally, there are a few other benefits:

• Scalding is heavily used in production at Twitter and has been battle-tested on Twitter-scale datasets.
• It has several active contributors both inside and outside Twitter that are committed to making it great.
• It is interoperable with your existing Cascading jobs.
• In addition to the Typed API, it has a a Fields API which may be more familiar to users of R and data-frame frameworks.
• It provides a robust Matrix Library.

Celebrated

Answer 2

I've had success with Scoobi. It's straightforward to use, strongly typed, hides most of the Hadoop mess (by doing thing like automatically serializing your objects for you), and totally Scala. One of the things I like about its API is that the designers wanted the Scoobi collections to feel just like the standard Scala collections, so you actually use them much the same way, except that operations run on Hadoop instead of locally. This actually makes it pretty easy to switch between Scoobi collections and Scala collections while you're developing and testing.

I've also used Scrunch, which is built on top of the Java-based Crunch. I haven't used it in a while, but it's now part of Apache.

Manassas

Answer 3

Twitter is investing a lot of effort into Scalding, including a nice Matrix library that could be used for various machine learning tasks. I need to give Scoobi a try, too.

For completeness, if you're not wedded to MapReduce, have a look at the Spark project. It performs far better in many scenarios, including in their port of Hive to Spark, appropriately called Shark. As a frequent Hive user, I'm excited about that one.

Ventose

Answer 4

The first two I would likely investigate are Scalding (which builds on top of Cascading) and Scoobi. I have not used either, though, but Scalding, in particular, looks like it provides a really nice API.

Arela

Answer 5

Another option is Stratosphere, It offers a Scala API that converts the Scala types to Stratosphere's internal data types.

The API is quite similar to Scalding but Stratosphere natively supports advanced data flows (so you don't have to chain MapReduce Jobs). You will have much better performance with Stratosphere than with Scalding.

Stratosphere does not run on Hadoop MapReduce but on Hadoop YARN, so you can use your existing YARN cluster.

This is the word count example in Stratosphere (with the Scala API):

val input = TextFile(textInput)
val words = input.flatMap { line => line.split(" ") }
val counts = words
  .groupBy { word => word }
  .count()
val output = counts.write(wordsOutput, CsvOutputFormat())
val plan = new ScalaPlan(Seq(output))

Eoin