Concat Avro files using avro-tools

Question

Im trying to merge avro files into one big file, the problem is concat command does not accept the wildcard

hadoop jar avro-tools.jar concat /input/part* /output/bigfile.avro

I get:

Exception in thread "main" java.io.FileNotFoundException: File does not exist: /input/part*

I tried to use "" and '' but no chance.

Avon

where are your input files? – Corset

@54l3d: I think the question was: Are they stored on the local file systems or HDFS ? – Blowup

@ClémentMATHIEU may be, they are on HDFS – Avon

Answer 1

I quickly checked Avro's source code (1.7.7) and it seems that concat does not support glob patterns (basically, they call FileSystem.open() on each argument except the last one).

It means that you have to explicitly provide all the filenames as argument. It is cumbersome, but following command should do what you want:

IN=$(hadoop fs -ls /input/part* | awk '{printf "%s ", $NF}')
hadoop jar avro-tools.jar concat ${IN} /output/bigfile.avro

It would be a nice addition to add support of glob pattern to this command.

Maggot

make sure to filter out "Found xxx items" from hadoop fs listing – Cupid

@EdiBice Updated the example. Thanks for the tip! – Cooking

Answer 2

Instead of hadoop jar avro-tools.jar one can run java -jar avro-tools.jar, since you don't need hadoop for this operation.

Cleodell