jar containing org.apache.hadoop.hive.dynamodb

Question

I was trying to programmatically Load a dynamodb table into HDFS (via java, and not hive), I couldnt find examples online on how to do it, so thought I'd download the jar containing org.apache.hadoop.hive.dynamodb and reverse engineer the process.

Unfortunately, I couldn't find the file as well :(.

Could someone answer the following questions for me (listed in order of priority).

1. Java example that loads a dynamodb table into HDFS (that can be passed to a mapper as a table input format).
2. the jar containing org.apache.hadoop.hive.dynamodb.

Thanks!

Villous

Answer 1

It's in hive-bigbird-handler.jar. Unfortunately AWS doesn't provide any source or at least Java Doc about it. But you can find the jar on any node of an EMR Cluster:

/home/hadoop/.versions/hive-0.8.1/auxlib/hive-bigbird-handler-0.8.1.jar

You might want to checkout this Article:

• Amazon DynamoDB Part III: MapReducin’ Logs

Unfortunately, Amazon haven’t released the sources for hive-bigbird-handler.jar, which is a shame considering its usefulness. Of particular note, it seems it also includes built-in support for Hadoop’s Input and Output formats, so one can write straight on MapReduce Jobs, writing directly into DynamoDB.

Tip: search for hive-bigbird-handler.jar to get to the interesting parts... ;-)

Pontus

Answer 2

1- I am not aware of any such example, but you might find this library useful. It provides InputFormats, OutputFormats, and Writable classes for reading and writing data to Amazon DynamoDB tables.

2- I don't think they have made it available publically.

Jemina

Thanks for the information Tariq! (especially 2). Helped save a few hours of my life :) – Villous