How to resolve a maven dependency with a name that is not compliant with the java 9 module system? [duplicate]

Question

I am trying to build a demo project in java 9 with maven that uses the dependency:

<dependency>
    <groupId>org.apache.spark</groupId>
    <artifactId>spark-mllib_2.10</artifactId>
    <version>2.2.0</version>
</dependency>

However when I run the jar tool to determine the automatic module name to use in my project's module-info.java I get the following error:

$ jar --file=spark-mllib_2.10/2.2.0/spark-mllib_2.10-2.2.0.jar --describe-module
Unable to derive module descriptor for: spark-mllib_2.10/2.2.0/spark-mllib_2.10-2.2.0.jar
spark.mllib.2.10: Invalid module name: '2' is not a Java identifier

It appears that the automatic module algorithm can't come up with a name that is valid java for this jar. Without adding the proper requires I get compile errors that the packages in spark mllib are missing such as:

package org.apache.spark.mllib.linalg does not exist

Is there anyway I can use this dependency in my project before it adds an official reserved module name or its own module-info?

Telescopic

From the look of it seems like the suggestion made here might help you as well. There isn't much progress visible from the owners on this one though. – Gravitative

A module name can be derived from most JAR files but there are some outliers with strange names that cannot be mapped to useful names. In this case, it would be good to submit a bug to the Apache Spark project asking them to choose a module name and add it to their JAR files with the "Automatic-Module-Name" attribute. Better still would be to publish explicit modules but they may not be ready to do that yet. – Gulosity

Answer 1

For a proper, long-term solution the Apache Spark project must either add module declarations (module.info.java) to their JARs or set the Automatic-Module-Name entry in their JAR's manifest.

If you can't wait for that, you can do the latter yourself quite easily:

1. create a file manifest.txt with the following content:

    Automatic-Module-Name: org.apache.spark.mlib2
   

2. Append that entry to the JAR's manifest:

    jar --update --file spark-mllib_2.10.jar --manifest=manifest.txt
   

3. Make sure it worked:

    jar --describe-module --file spark-mllib_2.10.jar
   

If you're planning to use this JAR in a real project, you would make it available to your colleagues, for which I would create a new version (maybe 2.10.patched-auto-name?), upload it to the company's Nexus and then edit the POMs to replace the original dependency with that one.

All in all, this is likely not worth the effort and you just might want to wait.

From the comments:

One thing to note is that the file you make the update with must end in a new line or the update silently fails.

Outboard

shouldn't that be manifest.mf? – Gravitative

No, the name of the file that you use for the manifest option doesn't matter. It could be foobar for all jar cares - it simply pulls the content out the file and adds it in the right place. – Outboard

This worked perfectly. The update file can be of any name since you are just appending the entries from it to the existing MANFEST.MF in the jar. One thing to note is that the file you make the update with must end in a new line or the update silently fails. I'll be adding the updated jar to a local repo as described here: https://mcmap.net/q/47205/-maven-add-a-dependency-to-a-jar-by-relative-path/8757510 – Telescopic

"One thing to note is that the file you make the update with must end in a new line or the update silently fails." as @AndySchmitt mentioned is important, I also gave up because of this silently failing – Jablonski

Thanks @Jablonski and AndySchmitt for that note. A mere 4.5 years later, I added it to the answer. 😬 – Outboard