I have a JavaPairRDD<Integer, Integer[]> on which I want to perform a groupByKey action. The groupByKey action gives me a: org.apache.spark.shuffle.MetadataFetchFailedException: Missing ...

here is my code: val bg = imageBundleRDD.first() //bg:[Text, BundleWritable] val res= imageBundleRDD.map(data => { val desBundle = colorToGray(bg._2) //lineA:NotSerializableException: org.ap...

I'm trying to serialize some objects of custom classes and I got the following exception: com.esotericsoftware.kryo.KryoException: com.sleepycat.je.EnvironmentFailureException: (JE 5.0.73) Identi...

I am already compressing RDDs using conf.set("spark.rdd.compress","true") and persist(MEMORY_AND_DISK_SER). Will using Kryo serialization make the program even more efficient, or is it not useful i...

I read the akka serialization page, where they talk about serialization-bindings as serialization-bindings { "java.lang.String" = java "docs.serialization.Customer" = java "com.google.protobuf...

I'd like to pass an object from the driver node to other nodes where an RDD resides, so that each partition of the RDD can access that object, as shown in the following snippet. object HelloSpark ...

Two questions, the answer to the general one will guide me on how minimal I can make a MVCE. 1) How can I know to register WrappedArray up front, (and every other class in Scala I might use)? Is ...

I am using Hazelcast 3.6.1 to read from a Map. The object class stored in the map is called Schedule. I have configured a custom serializer on the client side like this. ClientConfig config = new...

I read that the Kryo serializer can provide faster serialization when used in Apache Spark. However, I'm using Spark through Python. Do I still get notable benefits from switching to the Kryo seri...

I have a class that implements a custom Kryo serializer by implementing the read() and write() methods from com.esotericsoftware.kryo.Serializer (see example below). How can I register this custom ...

When deserializing a serialized object (from a file) using Kryo, I get the following exception: java.lang.ExceptionInInitializerError (...) Caused by: com.esotericsoftware.kryo.KryoException: (.....

In Spark, there are 3 primary ways to specify the options for the SparkConf used to create the SparkContext: As properties in the conf/spark-defaults.conf e.g., the line: spark.driver.memory 4g...

Problem description We have a Hadoop cluster on which we store data which is serialized to bytes using Kryo (a serialization framework). The Kryo version which we used to do this has been forked f...

I'm struggling to understand what exactly Avro, Kryo and Parquet do in the context of Spark. They all are related to serialization but I've seen them used together so they can't be doing the same t...

Twitter-chill looks like a good solution to the problem of how to serialize efficiently in Scala without excessive boilerplate. However, I don't see any evidence of how they handle case classes. D...

I have a GenericUDF (see code below) that was running fine on Hadoop-1 and Hive-0.12. But when testing the same GenericUDF using Hive-0.13 + Hadoop-2, I am getting the below error. Vertex failed...

I am trying to serialize and deserialize objects of a custom class (say, SomeClass, having a default no-args constructor) to a byte[] array, using Kryo 2.19 and the default serializer (FieldSeriali...

I'm trying to use kryo serialization with: kryo.setRegistrationRequired(true); I keep getting the following error saying that a certain class is not registered: java.lang.IllegalArgumentExcept...

I am trying to serialize a List of List of some objects (of a customized class: List> ), using Kryo. list2D; // List<List<MyClass>> which is already produced. Kryo k1 = new Kryo(); O...

How do you deserialize immutable collection using Kryo ? Do I need to register something in addition to what I've done ? Here is my sample code import com.esotericsoftware.kryo.Kryo import com.es...

Lets say I have an array arr of objects of type A in memory, each of which has a reference field pointing to the same object B. Illustration: A_1 A_2 A_3 ... A_N | | | | | | V | \--->\--&gt...

I recently discovered the library kryonet, which is super awesome and fits my needs excellently. However, the one problem that I am having is developing a good strategy for registering all of the...