no valid constructor on spark

Question

This is my code:

class FNNode(val name: String)

case class Ingredient(override val name: String, category: String) extends FNNode(name)


val ingredients: RDD[(VertexId, FNNode)] = 
sc.textFile(PATH+"ingr_info.tsv").
      filter(! _.startsWith("#")).
      map(line => line.split('\t')).
      map(x => (x(0).toInt ,Ingredient(x(1), x(2))))

and there are no errors when I define these variables. However, when trying to execute it:

ingredients.take(1)

I get

org.apache.spark.SparkException: Job aborted due to stage failure: Exception while getting task result: java.io.InvalidClassException: $iwC$$iwC$Ingredient; no valid constructor
    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1431)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1419)

It seems this could be related to Serialization issues as per the answer here . However, I have no idea of how to solve this if it is indeed a Serialization issue.

I'm following along the code in this book by they way, so I would assume this should have at least worked at some point?

Bouchier

FNNode has to be a case class as well unless I think. – Cabinet

Unfortunately, I get this: "error: case class Ingredient has case ancestor $iwC.$iwC.FNNode, but case-to-case inheritance is prohibited. To overcome this limitation, use extractors to pattern match on non-leaf nodes" – Bouchier

Can you try, whether it helps to let FNNode extend Serializable? – Allheal

This code actually works fine for me (in local mode) :/ – Berkowitz

what spark/scala version are you using? – Bouchier

@TzachZohar These serialization errors do never occur in local mode (Unfortunatelly...), as the classes are not needed to send over the network and hence are not serialized at all. – Allheal

Answer 1

This fixed your issue for me:

class FNNode(val name: String) extends Serializable

Coalfish