gradle force version of transitive dependency not working. No exclude, override or force seems to apply

Question

I have a conflict with a transitive dependency. Overriding, excluding or forcing does not help. What else can I do to get the right version of the library into the jar? The full code dan be. found https://github.com/geoHeil/gradle-dependency-resolution but the main parts of it are described below.

problem description

executing

./gradlew shadowJar

with geomesa dependencies (which pull in an outdated version of typesafe/lightbend configuration library) disabled:

dependencies {

    compile "com.github.kxbmap:configs_2.11:0.4.4"
    //compile "org.locationtech.geomesa:geomesa-hbase-spark-runtime_2.11:2.0.1"
}

and executing the jar

java -jar build/libs/gradleThing-all.jar                         

outputs

hello
my config is: Success(Job1Configuration(frequencyCounting))

When enabling the dependencies:

./gradlew shadowJar
java -jar build/libs/gradleThing-all.jar                         

it fails with

hello
Exception in thread "main" java.lang.Exception: Failed to start. There is a problem with the configuration: Vector([extract] com.typesafe.config.Config.hasPathOrNull(Ljava/lang/String;)Z)

which is related to an outdated version of the config library, how to resolve NoSuchMethodError on typesafe config?. Also confirmed via:

gradle dependencyInsight --dependency om.typesafe:config

> Task :dependencyInsight
com.typesafe:config:1.3.1 (conflict resolution)
   variant "runtime" [
      Requested attributes not found in the selected variant:
         org.gradle.usage = java-api
   ]
\--- com.github.kxbmap:configs_2.11:0.4.4
     \--- compileClasspath

com.typesafe:config:1.2.1 -> 1.3.1
   variant "runtime" [
      Requested attributes not found in the selected variant:
         org.gradle.usage = java-api
   ]
+--- org.locationtech.geomesa:geomesa-convert-avro_2.11:2.0.1
|    \--- org.locationtech.geomesa:geomesa-convert-all_2.11:2.0.1
|         +--- org.locationtech.geomesa:geomesa-tools_2.11:2.0.1
|         |    \--- org.locationtech.geomesa:geomesa-hbase-spark-runtime_2.11:2.0.1
|         |         \--- compileClasspath
...

How can I fix the transitive dependency to my desired version of 1.3.3?

my solutions

Trying to set:

compile("org.locationtech.geomesa:geomesa-hbase-spark-runtime_2.11:2.0.1") {
                exclude group: 'com.typesafe', module: 'config'
            }

and re running the jar fails again with the same issue.

Also an constraint https://docs.gradle.org/current/userguide/managing_transitive_dependencies.html#sec:dependency_constraints like:

implementation("com.typesafe:config")
    constraints {
        implementation("com.typesafe:config:1.3.3") {
            because 'previous versions miss a method https://mcmap.net/q/1916340/-how-to-resolve-nosuchmethoderror-on-typesafe-config'
        }
    }

or using force https://docs.gradle.org/current/userguide/managing_transitive_dependencies.html#sec:enforcing_dependency_version like :

implementation('com.typesafe:config:1.3.3') {
        force = true
    }

or like:

configurations.all {
    resolutionStrategy {
        force 'com.typesafe:config:1.3.3'
    }
}

does not give the right version.

result

All variants fail with the same error

What is wrong? There must be a way to force the desired version.

Selfinductance

Grade 4.8 RC2 does not seem to help github.com/gradle/gradle/issues/5364 – Selfinductance

What does dependencyInsight give after excluding through exclude group: 'com.typesafe', module: 'config' ? – Landed

gist.github.com/geoHeil/ca0d4670c1dcc13f1ed9c7429564c2a5 is returned, i.e. as expected only that dependency is visualized in the logs, however this is NOT the result I get in the jar, as otherwise the job would not have crashed. – Selfinductance

Also the ideas of discuss.gradle.org/t/… do not work. – Selfinductance

Answer 1

The problem you have is the exact opposite of what you are describing.

The different things you tried in your project all make sure that the version of com.typesafe:config is a 1.3.x.

However, when adding org.locationtech.geomesa:geomesa-hbase-spark-runtime_2.11:2.0.1 you pull in some dependency that already has the classes from com.typesafe:config but from the 1.2 line. And when you create your fat jar, that version overwrites the classes from the 1.3 line.

This can be seen by decompressing your fat jar and running:

$ javap com/typesafe/config/Config.class | grep hasPath
  public abstract boolean hasPath(java.lang.String);

showing that indeed the hasPathOrNull method is missing.

That shading issue is hinted at in https://issues.apache.org/jira/browse/SPARK-9441.

Given this, the easy path - if possible for you - would be to downgrade "com.github.kxbmap:configs_2.11 to a version that relies on com.typesafe:config:1.2.x

The other solution is to find out exactly which fat jar is including these and see if you can exclude it from your own fat jar.

Doreathadoreen

Why does not even the exclude working? Should that not exclude the dependency from the fat jar? – Selfinductance

My intention was to get rid of the 1.2 classes – Selfinductance

What would the process look like to find the exact jar? Should dependencyInsight be the tool of choice? – Selfinductance

Any attempt to modify the dependencies does not work because the problematic version does not appear as a dependency. The classes are already inside another dependency. – Doreathadoreen

Any chance to identify such a offending dependency with a tool? – Selfinductance

You can try one of the methods described in baeldung.com/gradle-fat-jar and leverage the duplicate strategy from the Jar task, by making it fail on duplicates: docs.gradle.org/current/dsl/… – Doreathadoreen