NullPointerException in Avro ReflectDatumWriter

Question

I have a specific issue with Avro serialization of Java Objects. I have POJO's generated from xsd schemas which I am then trying to serialize using avro to place on a kafka topic. Some of the xmlElements are optional

A test message looks like this:

@XmlRootElement(name = "message")
public class Testmessage {

  @XmlElement
  public String id;


  @XmlElement
  public String name;

  public Testmessage(String id, String name) {
    this.id = id;
    this.name = name;
  }

  public Testmessage() { }

  @Override
  public String toString() {
    return "Message{" +
        "id='" + id + '\'' +
        ", name=" + name +
        '}';
  }
}

And the method to serialize and place on the topic is:

public void sendMessage(Testmessage msg) throws Exception{

    DatumWriter<Testmessage> writer = new ReflectDatumWriter<Testmessage>(Testmessage.class); 
    ByteArrayOutputStream os = new ByteArrayOutputStream();

    Encoder encoder = EncoderFactory.get().binaryEncoder(os, null);
    writer.write(msg, encoder);
    encoder.flush();
    os.close();
    KeyedMessage<String, byte[]> data = new KeyedMessage<String, byte[]>(TOPIC_NAME, os.toByteArray());

    producer.send(data);

}

When I send both fields all works as expected. If I null one of the fields or leave it out I get NPE's from the write.

java.lang.NullPointerException: in Testmessage in string null of string in field id of    Testmessage
at org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:145)
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:58)

Any ideas? or point me in the right direction

Thanks!

Feil

Answer 1

It would appear that I ended managing to solve it myself after posting, a couple days of reading the internets later

    ReflectData reflectData = ReflectData.AllowNull.get();
    Schema schema = reflectData.getSchema(Testmessage.class);

    DatumWriter<Testmessage> writer = new ReflectDatumWriter<Testmessage>(schema); 

Appears to allow the use of nulls quite happily.

On to my next error! which is

org.apache.avro.UnresolvedUnionException: Not in union ["null",{"type":"record","name":"XMLGregorianCalendar","namespace":"javax.xml.datatype","fields":[]}]: 2014-10-22
at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:604)
at org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:151)
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:71)
at org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:143)
at org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:114)

Feil

ReflectData.AllowNull.get() is the way to go. Avro needs this extra flag to handle null values. I think this is so because the schema generated for such cases is much bigger than the objects without any null fields in the hierarchy. – Helico

Did you solve your "Not in union" error? If so, how? – Commentary

Yes as described below I converted to a string – Feil

Answer 2

Avro does not support all the data-types. Date and time support has been added very recently too (https://issues.apache.org/jira/browse/AVRO-739)

So this might be due to XMLGregorianCalendar not being supported as a data-type in Avro. Could you try using some other data-type (recommend using a String at first, before moving on to a more complex data type)?

Helico

Answer 3

I had the same error while testing my function in Junit, so I added also ReflectData.AllowNull to my function in the service then the test works fine.

Dogooder

Answer 4

Make sure that the values in the Java object that you're passing do comply with the Avro schema.

All the non-nullable fields should have a value, including List objects. If the field is of type List, initialize it with an empty list.

Stack trace for such an error would be similar to the following:

java.lang.NullPointerException: in <class-name> in string null of string in field id of <class-name> at org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:145) at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:58) at org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:114)

To find the exact field in the POJO which caused NullPointerException:

1. Add a debug point at the line where it throws the exception. According to the the above stack-trace it is the line 145 of ReflectDatumWriter.java.

2. Then navigate back the call stack (methods execution path) until you find the relevant field name. As in the picture, the name of the field is the value of the Schema.Field parameter passed to the writeField method.

Gaynellegayner