Maven: Using Java 8 libraries in applications instrumented with retrolambda-maven-plugin and DEX-ed with android-maven-plugin

Asked 17/5, 2015 at 11:24 Answered 10/9, 2015 at 20:16

Solved android maven java-8 android-maven-plugin retrolambda

I have written a small set of minilibraries for my internal use. This is set is built using Maven. The libraries are targetted for "regular" Java, GWT and Android. Some of them are written in Java 8 because I hadn't any intention to run them on GWT or Android, thus the other libraries are written in old Java 6 to support these two. I have a plan of full migration of my libs to Java 8 (in terms of language features), and I succeeded to run Java 8 rewritten libraries on not yet released GWT 2.8.0. However, I cannot make the Java 8 rewritten libs to compile for Android applications. The issue is that Retrolambda (the retrolambda-maven-plugin plugin) seems to be able to process the current Maven module classes only, and fully ignores the dependency classes. Thus, the android-maven-plugin breaks the target application build with:

[INFO] UNEXPECTED TOP-LEVEL EXCEPTION:
[INFO] com.android.dx.cf.iface.ParseException: bad class file magic (cafebabe) or version (0034.0000)
[INFO]  at com.android.dx.cf.direct.DirectClassFile.parse0(DirectClassFile.java:472)
[INFO]  at com.android.dx.cf.direct.DirectClassFile.parse(DirectClassFile.java:406)
[INFO]  at com.android.dx.cf.direct.DirectClassFile.parseToInterfacesIfNecessary(DirectClassFile.java:388)
[INFO]  at com.android.dx.cf.direct.DirectClassFile.getMagic(DirectClassFile.java:251)
[INFO]  at com.android.dx.command.dexer.Main.processClass(Main.java:665)
[INFO]  at com.android.dx.command.dexer.Main.processFileBytes(Main.java:634)
[INFO]  at com.android.dx.command.dexer.Main.access$600(Main.java:78)
[INFO]  at com.android.dx.command.dexer.Main$1.processFileBytes(Main.java:572)
[INFO]  at com.android.dx.cf.direct.ClassPathOpener.processArchive(ClassPathOpener.java:284)
[INFO]  at com.android.dx.cf.direct.ClassPathOpener.processOne(ClassPathOpener.java:166)
[INFO]  at com.android.dx.cf.direct.ClassPathOpener.process(ClassPathOpener.java:144)
[INFO]  at com.android.dx.command.dexer.Main.processOne(Main.java:596)
[INFO]  at com.android.dx.command.dexer.Main.processAllFiles(Main.java:498)
[INFO]  at com.android.dx.command.dexer.Main.runMonoDex(Main.java:264)
[INFO]  at com.android.dx.command.dexer.Main.run(Main.java:230)
[INFO]  at com.android.dx.command.dexer.Main.main(Main.java:199)
[INFO]  at com.android.dx.command.Main.main(Main.java:103)
[INFO] ...while parsing foo/bar/FooBar.class

The retrolambda-maven-plugin is configured as follows:

<plugin>
    <groupId>net.orfjackal.retrolambda</groupId>
    <artifactId>retrolambda-maven-plugin</artifactId>
    <version>2.0.2</version>
    <executions>
        <execution>
            <phase>compile</phase>
            <goals>
                <goal>process-main</goal>
                <goal>process-test</goal>
            </goals>
        </execution>
    </executions>
    <configuration>
        <target>1.6</target>
        <defaultMethods>true</defaultMethods>
    </configuration>
</plugin>

Is it possible to configure the Retrolambda plugin to process the library classes as well so all dependencies? Or maybe I just could use another bytecode processing tool?

UPDATE #1

I think I'm wrong about the Retrolambda fail. Digging into it a little more, I've figured out that android-maven-plugin can be blamed for it because it picks the not instrumented JAR files from the Maven repository directly, and not from the target directory. Enabling verbose logging discovered this pseudo-code command invoked by the android-maven-plugin:

$JAVA_HOME/jre/bin/java
-Xmx1024M
-jar "$ANDROID_HOME/sdk/build-tools/android-4.4/lib/dx.jar"
--dex
--output=$BUILD_DIRECTORY/classes.dex
$BUILD_DIRECTORY/classes
$M2_REPO/foo1/bar1/0.1-SNAPSHOT/bar1-0.1-SNAPSHOT.jar
$M2_REPO/foo2/bar2/0.1-SNAPSHOT/bar2-0.1-SNAPSHOT.jar
$M2_REPO/foo3/bar3/0.1-JAVA-8-SNAPSHOT/bar3-0.1-JAVA-8-SNAPSHOT.jar

My idea is executing the maven-dependency-plugin plugin to obtain those three artifacts into the $BUILD_DIRECTORY/classes directory to let the retrolambda-maven-plugin instrument the dependencies. Let's say:

STEP 1: Copy the dependencies to the target directory

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-dependency-plugin</artifactId>
    <executions>
        <execution>
            <phase>process-classes</phase>
            <goals>
                <goal>unpack-dependencies</goal>
            </goals>
            <configuration>
                <includeScope>runtime</includeScope>
                <outputDirectory>${project.build.directory}/classes</outputDirectory>
            </configuration>
        </execution>
    </executions>
</plugin>

STEP 2: Instrument the copied dependencies using Retrolambda

Invoke retrolambda-maven-plugin

STEP 3: Compile the DEX file

Invoke android-maven-plugin excluding the dependencies that were copied to the target directory, because they are all supposed to be located in the target directory

But this fails too, because I can't find a way to exclude artifacts from being DEXed with the android-maven-plugin.

How do I suppress the artifacts from being fetched from the repository where they are stored in non-instrumented state?

My plugins configuration:

org.apache.maven.plugins:maven-dependency-plugin:2.10
net.orfjackal.retrolambda:retrolambda-maven-plugin:2.0.2
com.jayway.maven.plugins.android.generation2:android-maven-plugin:3.9.0-rc.3

UPDATE #2

The Simpligility team has released Android Maven Plugin 4.4.1 featuring the scenario described below. See more at the plugin changelog.

<plugin>
    <groupId>com.simpligility.maven.plugins</groupId>
    <artifactId>android-maven-plugin</artifactId>
    <version>4.4.1</version>
</plugin>

Example scenario can be found at http://simpligility.github.io/android-maven-plugin/instrumentation.html

Ibadan answered 17/5, 2015 at 11:24 Comment(1)

great question and solution! saved my day... – Soundproof 11/2, 2016 at 17:50

Four months later, after having some time for investigation, I have finally made it work. First of all, the crucial thing about the solution is patching android-maven-plugin. I have forked the original plugin at GitHub and added a few configuration filter options that allow to include or exclude artifacts by group IDs, artifact IDs, and their respective versions. This is very important to configure retrolambda-maven-plugin. The overall workflow is basically the same as I noted in the question. Here it is:

maven-compiler-plugin Enable Java 8 language features support. Optional, as the main objective is processing Java 8 dependencies.
maven-dependency-plugin Unpack ALL Java 8 dependencies to the current project build target directory for the further processing.
retrolambda-maven-plugin Process all the obtained class files using the Retrolambda plugin.
android-maven-plugin Compile the DEX and APK files with the fork of the original android-maven-plugin.

Installing the fork:

#!/bin/bash

# The last upstream merge revision
PLUGIN_COMMIT=a79e45bc0721bfea97ec139311fe31d959851476

# Clone the fork
git clone https://github.com/lyubomyr-shaydariv/android-maven-plugin.git

# Ensure proper revision
cd android-maven-plugin
git checkout $PLUGIN_COMMIT

# Build the forked plugin, no tests
mvn clean package -Dmaven.test.skip=true

# Clone plugin JAR
cd target
cp android-maven-plugin-4.3.1-SNAPSHOT.jar android-maven-plugin-4.3.1-SNAPSHOT-$PLUGIN_COMMIT.jar

# Clone and modify pom.xml
cp ../pom.xml pom-$PLUGIN_COMMIT.xml
sed -i "s/<version>4.3.1-SNAPSHOT<\\/version>/<version>4.3.1-SNAPSHOT-$PLUGIN_COMMIT<\\/version>/g" pom-$PLUGIN_COMMIT.xml

# Update the plugin descriptor
unzip android-maven-plugin-4.3.1-SNAPSHOT-$PLUGIN_COMMIT.jar META-INF/maven/plugin.xml
sed -i "s/<version>4.3.1-SNAPSHOT<\\/version>/<version>4.3.1-SNAPSHOT-$PLUGIN_COMMIT<\\/version>/g" META-INF/maven/plugin.xml
zip android-maven-plugin-4.3.1-SNAPSHOT-$PLUGIN_COMMIT.jar META-INF/maven/plugin.xml

# Install the plugin
mvn org.apache.maven.plugins:maven-install-plugin:2.5.2:install-file -DpomFile=pom-$PLUGIN_COMMIT.xml -Dfile=android-maven-plugin-4.3.1-SNAPSHOT-$PLUGIN_COMMIT.jar

Done. Next, register the fork in your pom.xml and configure the build plugins:

<!-- enable Java 8 for the current module -->

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-compiler-plugin</artifactId>
    <version>3.2</version>
    <configuration>
        <source>1.8</source>
        <target>1.8</target>
    </configuration>
</plugin>

<!-- unpack Java 8 dependency classes to the current module build directory -->

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-dependency-plugin</artifactId>
    <version>2.10</version>
    <executions>
            <execution>
                <phase>process-classes</phase>
                <goals>
                    <goal>unpack-dependencies</goal>
                </goals>
                <configuration>
                    <includeScope>runtime</includeScope>
                    <includeGroupIds>foo-group,bar-group,baz-group</includeGroupIds>
                    <outputDirectory>${project.build.directory}/classes</outputDirectory>
                </configuration>
        </execution>
    </executions>
</plugin>

<!-- Convert Java 8 to Java 6 -->

<plugin>
    <groupId>net.orfjackal.retrolambda</groupId>
    <artifactId>retrolambda-maven-plugin</artifactId>
    <version>2.0.6</version>
    <executions>
        <execution>
            <phase>process-classes</phase>
            <goals>
                <goal>process-main</goal>
                <goal>process-test</goal>
            </goals>
        </execution>
    </executions>
    <configuration>
        <defaultMethods>true</defaultMethods>
        <target>1.6</target>
    </configuration>
</plugin>

<!-- DEXify and build the APK excluding the Java 8 dependencies as they are already processed -->

<plugin>
    <groupId>com.simpligility.maven.plugins</groupId>
    <artifactId>android-maven-plugin</artifactId>
    <version>4.3.1-SNAPSHOT-a79e45bc0721bfea97ec139311fe31d959851476</version>
    <executions>
        <execution>
            <phase>package</phase>
        </execution>
    </executions>
    <configuration>
        <androidManifestFile>${project.basedir}/src/main/android/AndroidManifest.xml</androidManifestFile>
        <assetsDirectory>${project.basedir}/src/main/android/assets</assetsDirectory>
        <resourceDirectory>${project.basedir}/src/main/android/res</resourceDirectory>
        <sdk>
            <platform>19</platform>
        </sdk>
        <undeployBeforeDeploy>true</undeployBeforeDeploy>
        <proguard>
            <skip>true</skip>
            <config>${project.basedir}/proguard.conf</config>
        </proguard>
        <excludes>
            <exclude>foo-group</exclude>
            <exclude>bar-group</exclude>
            <exclude>baz-group</exclude>
        </excludes>
    </configuration>
    <extensions>true</extensions>
    <dependencies>
        <dependency>
            <groupId>net.sf.proguard</groupId>
            <artifactId>proguard-base</artifactId>
            <version>5.2.1</version>
            <scope>runtime</scope>
        </dependency>
    </dependencies>
</plugin>

It should work.

Update

Recently I patched the fork for the Proguard Maven mojo.

Update 2

The android-maven-plugin repository owner merged my commits, so the artifact filtering is scheduled for the next version.

Ibadan answered 10/9, 2015 at 20:16 Comment(17)

Any luck with GWT+Retrolambda? – Oozy 28/12, 2015 at 6:32

@Oozy these components are unrelated. GWT processes the original source code only, whilst Retrolambda processes byte code. GWT 2.8.0-SNAPSHOT can work with Java 8 lambda expressions pretty stable. – Ibadan 28/12, 2015 at 8:47

Thanks for your great job! I hope you help solve my issue with your suggestion: I have an aar android library which I am depended to. As you might guess, the build fails because maven-dependency-plugin can not unpack this type of artifacts: Unpacking [path]\someLibrary.aar to [path]\target\classes. Failed to execute goal org.apache.maven.plugins:maven-dependency-plugin:2.10:unpack-dependencies (default) on project aquilax-dist-droid-operator: Unknown archiver type: No such archiver: "aar". – Grimaud 19/1, 2016 at 15:33

@Grimaud Yes, I have faced with exactly the same issue. There is no known to me elegant way to resolve the issue, but if you're fine with maven-antrun-plugin, you can refer a question I asked earlier: #33940955 – Ibadan 19/1, 2016 at 15:37

@Grimaud I've also documented a sample AAR-scenario that also has been merged to the master branch. Thus, you won't need custom local plugin management in the nearest future. – Ibadan 19/1, 2016 at 15:42

@Lyubomyr Shaydariv, thank you for your reply. Yet another question I have, if you don't mind. Is there any way to exclude all the dependencies which are copied to current project's build/classes directory (using the patched android-maven-plugin)? – Grimaud 20/1, 2016 at 13:27

@Grimaud if I understand you correctly, then yes: you could use <skipDependencies>true</skipDependencies>. There are some more filtering options (where: 1) explicit rules override implicit; 2) includes override excludes; 3) concrete artifacts override concrete artifact types): skipDependencies (boolean), artifactTypeSet with child includes and excludes ('jar', 'aar', etc), and artifactSet with child includes and excludes as well. – Ibadan 20/1, 2016 at 14:11

@Grimaud here are the commits that bring new configuration options: skipDependencies: github.com/lyubomyr-shaydariv/android-maven-plugin/commit/… ; artifactTypeSet: github.com/lyubomyr-shaydariv/android-maven-plugin/commit/… , artifactSet: github.com/lyubomyr-shaydariv/android-maven-plugin/commit/… , and includes over excludes github.com/lyubomyr-shaydariv/android-maven-plugin/commit/… – Ibadan 20/1, 2016 at 14:12

@Grimaud and, since these all the changes have been merged to the upstream project, you can switch to the original project by cloning it as I ocasionally sync my fork with the upstream now. I recommend you to use the very last revision, because the a79e45bc0721bfea97ec139311fe31d959851476 revision I specified in the answer is somewhat broken (it does not work with ProGuard causing some issues -- fixed in ab2e6706b64dd31a33ece3de77f42c2b2e8606d7 and later). This is probably all I can share. – Ibadan 20/1, 2016 at 14:19

@Lyubomyr Shaydariv, Unfortunately the exclude tag does not work for me at all. I am receiving the same error with/without using <excludes>. I will apply you latest solution to see if it works. – Grimaud 20/1, 2016 at 16:27

@LyubomyrShaydariv, and Would you please let me know how/where I would see an example of using <artifactSet> with child excludes? I can not find any information on the web page (simpligility.github.io/android-maven-plugin). Is this form OK? <artifactSet> <excludes> <exclude>*:maven-core</exclude</excludes></artifactSet> Many thanks in advance – Grimaud 20/1, 2016 at 16:41

@Grimaud Unfortunately no. Wildcards are not supported. The general syntax of an include/exclude element content is [groupId[:artifactId[:versionId]]], where in-brackets text is optional. Thus you can filter by groupId, by groupId/artifactId, or by groupId/artifactId/version (empty values are simply ignored). You probably might find a bit more in the original PR discussion here: github.com/simpligility/android-maven-plugin/pull/632 – Ibadan 20/1, 2016 at 20:38

@Grimaud Sorry for poor documentation, fortunately it can be found in JavaDoc that's probably might be supplied along with the plugin and probably could be seen in an IDE (really, not sure). Ah, by the way, enable the -X switch in Maven command-line to get Maven verbose output and obtain the filtered dependencies list. Hope this switch helps. :) – Ibadan 20/1, 2016 at 20:53

@LyubomyrShaydariv I did take a look at the source code of the commit-links you provided previously, and that helped a lot (sorry to inform you late). Thank to your previous job and your present responsiveness, I managed to build my project :) BTW, the bash could be shortened to: #!/bin/bash [\n] git clone github.com/lyubomyr-shaydariv/android-maven-plugin.git[\n] cd android-maven-plugin[\n] mvn clean package -Dmaven.test.skip=true[\n] mvn org.apache.maven.plugins:maven-install-plugin:2.5.2:install-file -DpomFile=pom.xml -Dfile=target/android-maven-plugin-4.3.1-SNAPSHOT.jar – Grimaud 21/1, 2016 at 15:53

@Grimaud I'm glad I could help you out. :) – Ibadan 21/1, 2016 at 16:33

@LyubomyrShaydariv Hey, its me again. Using gradle, the same issue appears. In other word, one might to unpack dependencies to retrolambda them. I want to know if you have any probable experience with that too? – Grimaud 30/1, 2016 at 16:56

@Amin, no, I don't use Gradle. – Ibadan 30/1, 2016 at 17:54

Hot tags

Godot Unity Godot Help Programming Godot 4.X GUI GDScript 3D 2D Physics CSharp Godot 3.X VR XR Projects C++

Recommended topics

Hot tags