kryo.readObject cause NullPointerException with ArrayList

Question

I get a NullPointerException when I unserialize an ArrayList object using kryo.

Caused by: java.lang.NullPointerException   
at java.util.ArrayList.ensureExplicitCapacity(ArrayList.java:215)   
at java.util.ArrayList.ensureCapacity(ArrayList.java:199)   
at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:96)
at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)    at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)     
at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)

I can see that StdInstantiatorStrategy creates an ArrayList without calling its constructor leaving one of the fields uninitialized causing the exception.

The documentation says that the no argument constructor should be called first and if none is available, the StdInstantiatorStrategy should be used to do field by field initialization.

What am I doing wrong?

Carduaceous

Answer 1

With kryo version 2.24, calling

kryo.setInstantiatorStrategy(new StdInstantiatorStrategy());

overrides the default instantiator strategy which uses the class no argument constructor if present. The correct way to do this is to call:

((Kryo.DefaultInstantiatorStrategy) kryo.getInstantiatorStrategy()).setFallbackInstantiatorStrategy(new StdInstantiatorStrategy());

This is explained here https://github.com/EsotericSoftware/kryo

I think this has changed since version 2.21

Carduaceous

Answer 2

I was getting a similar null pointer exception and I needed to add a custom serializer from: https://github.com/magro/kryo-serializers

Ex:

kryo.register( Collections.EMPTY_LIST.getClass(), new CollectionsEmptyListSerializer() );

Abmho

Answer 3

I meet the same problem and finally solve it. In my case, I use protobuf object in Spark job. Spark kryo serializer couldn't serialize/deserialize protobuf object very well. We can use two methods to solve this problem.

1. Use protobuf default serialize/deserialize method instead of kryo serialize method. For example, you can convert your Spark rdd[YourProtobufObject] to rdd[ByteString], use pb.toByteString() do the serialize and use .parseFrom(xxByteString) do the deserialize. Actually, this method is not elegant, but it works.
2. Register your own protobuf class to kryo. The details as follows.

• First, add config to SparkConf. For example

conf.set("spark.serializer","org.apache.spark.serializer.KryoSerializer")
.set("spark.kryo.registrator","your.own.registrator.implement.MyKryoRegistrator")

• Second, create your own registrator implement. You can use Twitter opensource Chill project ProtobufSerializer dependency or mvnrepository, and use ProtobufSerializer directly. The maven dependency looks like this.

   <dependency>
       <groupId>com.twitter</groupId>
       <artifactId>chill_2.11</artifactId>
       <version>0.9.3</version>
   </dependency>
   <dependency>
       <groupId>com.twitter</groupId>
       <artifactId>chill-protobuf</artifactId>
       <version>0.9.3</version>
       <exclusions>
           <exclusion>
               <groupId>com.twitter</groupId>
               <artifactId>chill-java</artifactId>
           </exclusion>
       </exclusions>
   </dependency>
  

create your own kryo registrator implement named MyKryoRegistrator

class MyKryoRegistrator extends KryoRegistrator {
  override def registerClasses(kryo: Kryo): Unit = {
    kryo.register(classOf[YourProtobufObject], new ProtobufSerializer())
  }
}

• Third, run again your Spark job, it will be ok.

Dumbbell

Answer 4

Also pay attention to how you create the list, there are ways to create non-modifiable list, which may cause this kind of serialization trouble.

For example popular shortcut in tests Arrays.asList("a", "b") can be problematic.

Workaround: new ArrayList<>(asList("a", "b")).

Contravention