how to create fat jar which does not overwrite entries under META-INF/services

Question

I have the following gradle build config:

plugins {
id 'com.github.johnrengelman.shadow' version '1.2.3'
}

group 'abc'
version '1.0-SNAPSHOT'

apply plugin: 'java'
apply plugin: 'application'
mainClassName = "abc.Driver"


repositories {
    mavenCentral()
}

dependencies {
    compile (group: 'org.apache.hadoop', name: 'hadoop-client', version: '2.6.0')
}

sourceSets {
    main {
        java {
            srcDir './src'
        }
    }
}

jar {
    manifest {
        attributes(
                'Class-Path': configurations.compile.collect { it.getName() }.join(' '),
                'Main-Class': mainClassName
        )
    }
}

task fatJar(type: Jar) {
    manifest {
        attributes 'Implementation-Title': 'Gradle Jar File Example',
                'Implementation-Version': version,
                'Main-Class': mainClassName
    }
    baseName = project.name + '-all'
    from { (configurations.compile - configurations.provided).collect

        {
            //println it.getName()
            it.isDirectory() ? it : zipTree(it)
        }

    }
    {
        exclude "META-INF/*.SF"
        exclude "META-INF/*.DSA"
        exclude "META-INF/*.RSA"
    }
    with jar
}

The main method that I have is just the following piece of code:

public static void main(String[] args) {
    Iterable<ClientProtocolProvider> frameworkLoader =
            ServiceLoader.load(ClientProtocolProvider.class);
    for(ClientProtocolProvider cpp: frameworkLoader) {
        System.out.println(cpp.toString());
    }
}

When I run the main method from IDE as expected I get the following output:

org.apache.hadoop.mapred.YarnClientProtocolProvider@4783da3f
org.apache.hadoop.mapred.LocalClientProtocolProvider@300ffa5d

But when I run the gradle fat jar task and I create the fat jar, after running the main method using (java -jar) through terminal I just get:

org.apache.hadoop.mapred.LocalClientProtocolProvider@7f31245a

I found that when fat jar is created, the entries under META-INF/services are merged for all dependencies and hence I lose the declaration for YarnClientProtocolProvider which I need further in my code.

YarnClientProtocolProvider is declared in hadoop-mapreduce-client-jobclient.jar

LocalClientProtocolProvider is declared in hadoop-mapreduce-client-common.jar

Does any body know how to create a fat jar which does not merges entries under META-INF/services?!

Stoops

I had the same issue, but with Maven shade plug-in. This is post gave me the hint to solve the issue. Used ServicesResourceTransformer to fix the issue in maven-shade-plugin – Gluten

Answer 1

This should do the work

shadowJar {
    mergeServiceFiles() 
}

Stanwinn

this will overwrite the content of service provider config files in favor of a winner dependency jar, hence we lose some service provider declarations. – Stoops

Check this out: #32888466 – Stanwinn