java.io.IOException: Invalid int encoding

Question

I'm writing a mapReduce Job to read and process an Avrofile. The input file is a Avro The output format is Avro

When I'm executing the Mapreduce Job, I'm getting the following exception in the reducer phase. As the reducer throws the IOException, I'm not able to capture and subside it in the reducer. The error stack trace in Hue looks

java.io.IOException: Invalid int encoding
at org.apache.avro.io.DirectBinaryDecoder.readInt(DirectBinaryDecoder.java:113)
at org.apache.avro.io.ValidatingDecoder.readInt(ValidatingDecoder.java:83)
at org.apache.avro.reflect.ReflectDatumReader.readInt(ReflectDatumReader.java:166)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:156)
at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:177)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:148)
at org.apache.avro.generic.GenericDatumReader.readArray(GenericDatumReader.java:206)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:150)
at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:177)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:148)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:139)
at org.apache.avro.hadoop.io.AvroDeserializer.deserialize(AvroDeserializer.

Upon googling, I noticed there was a apache JIRA ticket (https://issues.apache.org/jira/browse/AVRO-882). No updates.

I'm using AVRO-1.7.5 and following is the dependency maven

    <dependency>
        <groupId>org.apache.avro</groupId>
        <artifactId>avro</artifactId>
        <version>1.7.5</version>
    </dependency>

Any help would be much appreciated?. Thanks

Giarla

Have you solved the issue? – Cosmopolite

I'm using avro 1.8.1 and also have this issue. How did you solve it? – Chaps

Jira ticket is still open, as a workaround casting might be an option? – Shout

Answer 1

In my case this issue manifested itself because I was not using the correct schema to read the document.

Remember that if you are reading a Avro document which was written with a schema (writer schema) that is different than what you are reading with, you will need to construct the SpecificDatumReader with the constructor that take both the reader & writer schema.

new SpecificDatumReader<>(writerSchema, readerSchema)

instead of

new SpecificDatumReader<>(class)

Quintin