Running an uber jar from sbt assembly results in error: Could not find or load main class

Question

I have a spark job packaged as an uber-jar using the sbt assembly plugin. The build.sbt specifies a runnable main to be the target of the resulting uber-jar

mainClass in assembly := Some("com.foo.Bar")

After the assembly is correctly created, running the intended command:

java -jar assembly.jar

results in

Error: Could not find or load main class com.foo.Bar

Using the an alternative method, like java -cp assembly.jar com.foo.Bar gives the same error message.

Then, I extracted the contents of the uber-jar in a new directory. I can see my com/foo/ directory and the Bar.class file. From the root of the extracted directory I tried:

java -cp . com.foo.Bar

and I get a correct result.

Further trying to find the reason of the error, I tried:

java -verbose -jar assembly.jar

I can see the java core classes being loaded, but I don't see any of my packaged classes being loaded.

What can possibly be wrong here?

Hardheaded

Answer 1

After an extensive investigation (read: pulling hairs out), it turns out that this behavior is the result of a rogue INDEX.LIST from one of the flattened jar files landing in the META-INF directory of the resulting uber-jar.

Following the JAR file spec, the INDEX.LIST, if present, dictates what packages from the Jar file are to be loaded.

To avoid this, we updated the mergeStrategy with a rule to avoid any pollution of the resulting META-INF directory:

case PathList("META-INF", xs @ _*) => MergeStrategy.discard

This fixed the issue and returned my sanity.

---

Update:

After some extra searching, it turns out that the default merge strategy takes proper care of INDEX.LIST. This answer applies when the customized merge strategy contains cases that handle the META-INF pathSpec

Hardheaded

This also fixes another related problem, I can use the fatjar to bring dependencies into spark-shell. I can run the import statements. But then I get a java.lang.NoClassDefFoundError when I try to actually use them. Maddening! – Soap

You saved my life. I was literally going insane. I don't want to imagine what you went through to discover it on your own. – Considering

I did not understand the "update", though first part did work. – Stephainestephan