How to resolve dependencies between modules within multi-module project?

Question

After working with Maven for a while, I am thrilled by the many features that Maven brings into the build architecture, particularly the dependency management. However, I have run into one issue again and again - how Maven resolves dependencies between multi-module projects. I am wondering if this is the big flaw of the current Maven implementation and/or if there is any satisfactory workaround.

Let's say I have a multi-module Maven project. The Parent pom contains three modules -- moduleA (jar), moduleB (jar), and moduleC(war). B depends on A and C depends on B. Simple enough? Now that I want to run the mvn dependency:go-offline at the parent project, which is supposed to resolve all the dependencies and bring them into the local .m2 directory. It fails because Maven complains that it cannot solve dependency for moduleA when it is acting on moduleB. Because all these modules belong to one groupId, I even try to use -DexcludeGroupIds=x.y.z to exclude these module dependencies, but it still fails at the same point.

I understand why Maven is complaining - moduleA is not built yet and thus there is no moduleA:jar artifact in my local or internal repository when go-offline goal is executed. But IMHO the plugin should treat these inter-module dependencies differently. In this case, it should simply ignore it. One might argues that I can simply do mvn clean install, which will install moduleA:jar into the local repository. After that, running mvn dependency:go-offline will work for sure. But that workaround defeats the purpose of this go-offline goal. This plugin allows us to resolve and pull dependencies into our local repository without building the whole project. I used dependency:copy-dependencies goal in another case and it has the same issue.

I also ran into similar issue in other scenarios: "mvn clean generate-source" could not resolve dependencies. When I ran mvn clean compile, everything works fine, but when I ran mvn clean generate-source, it fails because Maven cannot resolve inter-module dependency. In that case, the was caused by @requiresDependencyResolution in the antrun plugin.

Since both antrun plugin and dependency plugin are very popular in the Maven world, I am sure I am not the only one who have run into this issue. Anyone finds any solution/workaround?

Ramayana

I've voted to close this question as "not constructive". It may be eloquently written, however I can't see a concrete question in there. Asking for a discussion over whether Maven is "correct" in its current behaviour is not a good fit for the Q&A model of this site. – Sanders

Duncan, I am also searching for a workaround other than do a mvn clean install first. Maybe I did not word the title properly. My intent is to find a solution/workaround, not to argue if maven behavior is correct or not. – Ramayana

@DuncanJones I modified the title and question a little to clarify my intent. Thanks for your comment. – Ramayana

@DuncanJones This issue is a pain when trying to create a multi-stage docker build where the first layer will retrieve all the required external dependencies so they will be cached in subsequent invocations (unless one of the pom files changes). – Michaelemichaelina

Answer 1

I doubt such functionality would ever be possible with Maven. Whilst your projects share a common parent and depend upon each other, Maven cannot possibly know where to find these projects in order to build them. It also cannot determine whether the projects just need to be built, or whether you've specified the wrong version number for your dependency.

Sanders

After some research, I reached the same conclusion. The only way that I can make the mvn dependency:go-offline to work is to have all the artifacts of modules either in the .m2 folder or in a maven enterprise repository. – Ramayana

putting the artifact in .m2 is "mvn install" And if you don't want to install, "mvn package dependency:go-offline" does the job. Petar created issues.apache.org/jira/browse/MDEP-516 issue to look for a better alternative or add this to the plugin's documentation – Greenhead

Answer 2

Maven has the concept of a "reactor" where artifacts that have been built in a single run (e.g. maven package) are available for dependency resolution during the build. For example, if your dependency graph yields the build order moduleA moduleB moduleC, and you do mvn package, Maven will build moduleA, package its artifact and add it to the reactor, then build moduleB, package it and add it to the reactor, then the same for moduleC. This means moduleB has access to moduleA's artifact for dependency resolution, and moduleC has access to moduleA and moduleB. This only works if artifacts are actually built, i.e. when you run the package goal.

The problem is that when you don't run the package goal because you're not interested in the artifacts (as for your dependency:go-offline example), artifacts for modules that have been processed don't get built and thus not added to the reactor. I find this annoying as well; I think Maven should look at the POM files in its list of modules to build and look there as well; but it doesn't.

In short, the solution to your problem is to do mvn package dependency:go-offline . This will not install artifacts in your local repository (which I believe is very bad practice) but it will put them in the reactor for the duration of the build, meaning that Maven will be able to resolve dependencies from your moduleB to the moduleA that has already been built. The downside is every module will be tested and packaged, which is a lot of work when all you wanted is to do dependency:go-offline.

Either way, hope this helps.

Financier

This was very helpful. I feel that if more Maven noobs could realize this beforehand, they'd have an easier time understanding Maven and what it aims to accomplish – Anticlerical

Re: "The downside is every module will be tested and packaged, which is a lot of work when all you wanted is to do dependency:go-offline." Don't forget about -DskipTests. 😀 – Pergola

Answer 3

This has been finally been resolved with Maven Dependency Plugin version 3.1.2.

You can make sure it's used by pinning the version in your pom.xml:

<build>
    <pluginManagement>
        <plugins>
            <plugin>
                <artifactId>maven-dependency-plugin</artifactId>
                <version>3.1.2</version>
            </plugin>
       </plugins>
    </pluginManagement>
</build>

Robert

I did and I still get "Could not resolve dependencies for project ...". Do I need to consider anything? – Menology

Difficult to say without more details. This fixed the issue for me several times. Doesn't necessarily fix all problems for everyone, though. Ask a new question with more details? – Robert

Answer 4

I have created a JIRA ticket with a sample project here:

https://issues.apache.org/jira/browse/MDEP-516

Please vote for it.

Gaze

Answer 5

You explained why it doesn't work so you understand the issues. The problem for you is that it stops when it can't find A.jar but that will only happen when you get to building B. So there is a sort of, sometimes useful strategy.

You have to mess with A by itself. Just build A. Use your plan of loading dependencies and then building it.

Once it builds, you can move on to doing the same thing with B and then C. Step by step.

One thing to remember here is that its sometimes ok to build B with an old snapshot of A in the local repo. You only need the new snapshot of A build in the repo if there are signature changes or new stuff required by B.

There are some discussions here too: Maven Modules + Building a Single Specific Module

One final not that usually these sort of questions come up when people have builds that take too long. There are several ways to make builds go faster:

1. Get faster hardware. The build computer, the disk storage or the network speed are typical components that are cheaper to upgrade than waste the time taken in slow builds.
2. Make the build go faster by not building stuff that doesn't need rebuilding. (For example, I had a build that rebuilt all the generated code every time. I added some stuff into the build that kept it from doing that except when dependencies to the generated code changed.)
3. Speed up the tests. Sometimes this means breaking the tests into two parts. Part 1 is fast tests and part 2 is slow tests. Run the fast tests on every build and the slow tests before any checkin of code or release of artifacts.
4. Break a multi-module build into 2 or more separate builds and use human intelligence to decide when to rebuild things. This works well when some jars are stable and don't change much any more.
5. Fill in your own method to make the build go faster.

Cuss

Lee, I like your tips about getting build faster. But to solve my main issue, building one module at time one by one seems really cumbersome. I would rather execute mvn clean install first on all modules, so that I do not need to deal with one module a time. Anyway, thanks for the input. – Ramayana

Answer 6

That's going to be supported by dependency:go-offline goal starting from Maven Dependency Plugin v3.1.2. Related jira ticket MDEP-204 and patch 23b7ca8790ae14175ed8e3a20c75c6274efe5ad8 with fix.

Footwall