Shading over third party classes

Question

I'm currently facing a problem with deploying an uber-jar to a Spark Streaming application, where there are congruent JARs with different versions which are causing spark to throw run-time exceptions. The library in question is TypeSafe Config.

After attempting many things, my solution was to defer to shading the provided dependency so it won't clash with the JAR provided by Spark at run-time.

Hence, I went to the documentation for sbt-assembly and under shading, I saw the following example:

assemblyShadeRules in assembly := Seq(
      ShadeRule.rename("org.apache.commons.io.**" -> "shadeio.@1")
      .inLibrary("commons-io" % "commons-io" % "2.4", ...).inProject
)

Attempting to shade over com.typesafe.config, I tried applying the following solution to my build.sbt:

assemblyShadeRules in assembly := Seq(
  ShadeRule.rename("com.typesafe.config.**" -> "shadeio.@1").inProject
)

I assumed it was supposed to rename any reference to TypeSafe Config in my project. But, this doesn't work. It matches multiple classes in my project and causing them to be removed from the uber jar. I see this when trying to run sbt assembly:

Fully-qualified classname does not match jar entry:
  jar entry: ***/Identifier.class
  class name: **/Identifier.class
Omitting ***/OtherIdentifier.class.
Fully-qualified classname does not match jar entry:
  jar entry: ***\SparkBaseJobRunner$$anonfun$1.class
  class name: ***/SparkBaseJobRunner$$anonfun$1.class

I also attempted using:

assemblyShadeRules in assembly := Seq(
  ShadeRule.rename("com.typesafe.config.**" -> "shadeio.@1")
           .inLibrary("com.typesafe" % "config" % "1.3.0")

This did finish the assemblying process of the uber JAR, but didn't have the desired run time effect.

I'm not sure I fully comprehend the effect shading has on my build process with sbt.

How can I shade over references to com.typesafe.config in my project so when I invoke the library at run-time Spark will load my shaded library and avoid the clash caused by versioning?

I'm running sbt-assembly v0.14.1

Radom

Answer 1

Turns out this was a bug in sbt-assembly where shading was completely broken on Windows. This caused source files to be removed from the uber JAR, and for tests to fail as the said classes were unavailable.

I created a pull request to fix this. Starting version 0.14.3 of SBT, the shading feature works properly. All you need to do is update to the relevant version in plugins.sbt:

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.3")

In order to shade a specific JAR in your project, you do the following:

assemblyShadeRules in assembly := Seq(
  ShadeRule.rename("com.typesafe.config.**" -> "my_conf.@1")
    .inLibrary("com.typesafe" % "config" % "1.3.0")
    .inProject
)

This will rename the com.typesafe.config assembly to be packaged inside my_conf. You can then view this using jar -tf on your assembly (omitted irrelevant parts for brevity):

***> jar -tf myassembly.jar
my_conf/
my_conf/impl/
my_conf/parser/

Edit

I wrote a blog post describing the issue and the process that led to it for anyone interested in a more in-depth explanation.

Radom

Is there a sample project that showcases this bug fix? – Alonsoalonzo

@sparkr No, but you can easily reproduce it for by using sbt-assembly 0.14.2. – Radom

Would you mind taking a look at the problem that I'm facing? Here is the link to the Stackoverflow post: #43988525 – Alonsoalonzo

Have you got a GitHub project that shows how to properly create a build.sbt? – Alonsoalonzo

@sparkr For shading? – Radom

Yes for shading! Did you get a chance a look into the question that I posted? – Alonsoalonzo

I'm doing sbt publish and want to package my dependency with the published jar and have it shaded. I tried adding assemblyShadeRules but didn't work when I publish. Any idea? – Shonna

@Avner Publishing an uber jar as dependency isn't something you usually want. Are you sure that's what you meant? – Radom

Not sure at all. I am currently not publishing an uber jar. The consumer of my jar is having problems so I though an "uber jar" might be able to mask those problems – Shonna

@YuvalItzchakov Maybe you can answer this question or know of another similar question #51380802 – Shonna