Flatten tuple like a bag

Question

My dataset looks like the following:

( A, (1,2) )
( B, (2,9) )

I would like to "flatten" the tuples in Pig, basically repeating each record for each value found in the inner-tuple, such that the expected output is:

( A, 1 )
( A, 2 )
( B, 2 ) 
( B, 9 )

I know this is possible when the tuples (1,2) and (2,9) are bags instead.

Tinney

Answer 1

Your insight is good; it's possible by transforming the tuple in a bag. The schema we want to aim for is: {a: chararray,{(chararray)}} for example: (A,{(1),(2)})

Here is the solution to your problem:

A = LOAD 'data.txt' AS (a:chararray,b:(b1:chararray,b2:chararray));
B = FOREACH A GENERATE a, TOBAG(b.b1,b.b2);
C = FOREACH B GENERATE a, FLATTEN($1);

The magic part is the TOBAG operator.

Overkill

Nice! In this case the tuple has two elements (b.b1,b.b2). How do I do this when the size of the tuple may vary? – Laroche

I would also like to know how to do this with a variable length Tuple. – Radiograph

You can write your own UDF for that purpose. DataBag bag = BagFactory.getInstance().newDefaultBag(); for (int i = 0; i< inputTuple.size(); i++) { bag.add(TupleFactory.getInstance().newTuple(inputTuple.get(0))); } return bag; – Lynellelynett

A bit late, but use TOKENIZE! (pig.apache.org/docs/latest/func.html#tokenize) – Sikata

Answer 2

You can use DataFu's UDF TransposeTupleToBag (http://datafu.incubator.apache.org/docs/datafu/1.1.0/datafu/pig/util/TransposeTupleToBag.html) and then flatten the bag to get a row per item in the bag.

Bedim

Answer 3

I know this is an old thread but I could not get the above method to work. Thought I will share my findings.

input: (1-2-3, abc)
       (4-5-6, xyz)
desired output:
       (1, abc)
       (2, abc)
       (3, abc)
       (4, xyz)
       (5, xyz)
       (6, xyz)

Initially, I used STRSPLIT that generates a tuple resulting in the similar input as above but was unsuccessful.

output = FOREACH input GENERATE FLATTEN(TOBAG(STRSPLIT($0, '-'))), $1

This resulted in the output as:

 (1,2,3,abc)
 (4,5,6,xyz)

However, when I used tokenize and replace functions I got the desired output.

output = FOREACH input GENERATE FLATTEN(TOKENIZE(REPLACE($0,'-', ' '))), $1;

Rondon