Apache Pig - nested FOREACH over same relation

Question

I have a number of bags and I want to compute the pairwise similarities between the bags.

sequences = FOREACH raw GENERATE gen_bag(logs);

The relation is described as follows:

sequences: {t: (type: chararray, value:charray)}

The similarity is computed by a Python UDF that takes two bags as arguments. I have tried to do a nested foreach over the sequences variable, but I cant loop over the same relation twice. I've also tried to define the sequences twice, but I cant access the copy in the foreach. I'm also unsure how I can come up with a data structure that allows me to do things like this. How can I do this?

Gissing

Answer 1

It sounds like you could load up two copies of the source data, perform a cross-product, then iterate through calling your UDF on each pair. Something like:

sequences_A = FOREACH raw GENERATE gen_bag(logs);
sequences_B = FOREACH raw GENERATE gen_bag(logs);
all_pairs = CROSS sequences_A, sequences_B;
FOREACH all_pairs GENERATE myudf(first_bag, second_bag);

Outroar