Avoiding maven repository version collision when using feature branches

Question

Question: How do you handle feature branches for maven multi project builds?

Jenkins builds and deploys these branches to keep build overhead on developers to a minimum but develop and feature branches cannot build the same maven version or we risk mismatch between artifacts and source.

We have a script to change parent version in child poms and version in root pom. While this segregates the branches in maven space, it results in extra work when merging.

I think nexus pro staging feature might help us avoid this requirement and make each feature branch use a specific repo which we easily drop after branch deletion/merge.

Again: how to handle the problem of multiple branches and maven?

Rosinarosinante

We have a script to change parent version in child poms and version in root pom Why not use the maven-release-plugin:branch goal? It will create the branch automatically and will update the version, that you can set to x.x-my-branch or something similar. – Keirakeiser

Thanks @Tunaki. Is merging still a problem since the poms were modified? – Rosinarosinante

Yeah, with a different version, it will be a pain... see also #3555660 (or #12197691) – Keirakeiser

We use mvn versions:set -DnewVersion=1.2-BRANCH-SNAPSHOT in the branch to keep it distinct. – Minify

Answer 1

How about the following approach:

• Use the buildnumber-maven-plugin to fetch information from git and populate specific Maven properties (we are interested specifically in the scmBranch property (that is, the current git branch)
• Use the build-helper-maven-plugin to check whether we are in a feature branch or not (via a regex, excluding well-known branches like master, develop, etc.) and populate (or not) a new Maven property, say branch.classifier
• Use the maven-jar-plugin to set a classifier on the generated artifacts, based on what the previous step set, that is, using the new branch.classifier property: if empty, no classifier will be applied (default behavior, applied to the develop branch, for example); otherwise a classifier named after the current branch will be dynamically applied.

Here is a minimal example:

<build>
    <plugins>
        <plugin>
            <groupId>org.codehaus.mojo</groupId>
            <artifactId>buildnumber-maven-plugin</artifactId>
            <version>1.4</version>
            <executions>
                <execution>
                    <phase>validate</phase>
                    <goals>
                        <goal>create</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
        <plugin>
            <groupId>org.codehaus.mojo</groupId>
            <artifactId>build-helper-maven-plugin</artifactId>
            <version>1.10</version>
            <executions>
                <execution>
                    <id>regex-property</id>
                    <goals>
                        <goal>regex-property</goal>
                    </goals>
                    <configuration>
                        <name>branch.classifier</name>
                        <value>${scmBranch}</value>
                        <regex>(^develop)|(^master)|(^release.*)</regex>
                        <replacement></replacement>
                        <failIfNoMatch>false</failIfNoMatch>
                    </configuration>
                </execution>
            </executions>
        </plugin>
        <plugin>
            <artifactId>maven-jar-plugin</artifactId>
            <version>3.0.2</version>
            <configuration>
                <classifier>${branch.classifier}</classifier>
            </configuration>
        </plugin>
    </plugins>
</build>

The snippet below is basically populating the scmBranch property dynamically, then setting the branch.classifier to its value only if different than develop, master or release*, then setting it as a classifier.

Main advantages of this approach:

• No pom changes will be applied, hence no merging issues at all
• No clashes on Nexus working on the same version of the project but on different branches: the classified artifact will have different Maven coordinates, that is, the GAV (groupId, artifactId, version) becomes unique GAVC (+classifier)
• That's actually a meaningful usage of the classifier attribute of an artifact:

  The classifier allows to distinguish artifacts that were built from the same POM but differ in their content.

• The generated artifact will be dynamically different in Nexus, according to its source branch, hence having implicit traceability: no intervention from developers (no error prone, implicit convention), no intervention from CI job (easier maintenance), completely transparent
• Using classifiers, will be easier to use the artifacts generated by a branch as a maven dependency (e.g. in case of library project): I want to use the dependency currently under development on branch xxx

Examples
Hence, you would have the following artifacts generated:

• When working on develop: e.g. project-1.0.0-SNAPSHOT.jar (empty classifier, hence not applied, as handled by the regex)
• When working on featureA: e.g. project-1.0.0-SNAPSHOT-featureA.jar
• When working on hotfix-JIRA123: e.g. project-1.0.0-hotfix-JIRA123.jar
• When working on release-sprint42: that's up to you, I added this case to not apply the branch name, simply because in these cases I prefer to esplicitely set a special classifier, RC<number>, for release candidates, but that's a matter of conventions/taste/habits, you can apply the same approach on this branch as well, as long as no clashes will be created on Nexus. Also note: when using JIRA/Stash/Git integration, the release branch name is normally something like release/v0.1.0, where the / character may cause issues in some OS (still something fixeable via further regex replacing though, if really required).
• When working on master: hey, no one should work on master :) the case is there just as a double check, but that's actually not required

---

Warnings on this approach:

• As explained in the discussion below via comments, if the concerned Maven project is already using classifiers and even more via inter-modules dependencies (e.g. dependendies on test scope classes from another module), then this approach should be carefully tested, since it might have some drawbacks
• The publication of the <artifactId>.pom files containing branch classifier can get into conflicts with the mainline build (i.e. overriding it)

Burkett

thanks. that's very slick. I'll experiment with a small project and see what I can do. The tricky bits will be the tycho eclipse RCP section of my build, but I can probably use resource plugin to handle tag to property replacement in those files. – Rosinarosinante

@PeterKahn FYI: in the company where I work, we actually use it right now in our delivery pipeline and works fine, no conflicts in Nexus and developers can also import as maven dependencies the artifacts generated by a certain branch (rare case, but now possible) without any need of changing the project version nor issues on merging it. – Burkett

What about sources, test-sources, javadoc and other classifiers which normally use a specific name? Do those continue to work on command line and in IDEs? I'd expect sources and test-sources wouldn't work and would pull/use the default for the version not the branch specific. – Rosinarosinante

@PeterKahn indeed using the sample configuration above it would apply the behavior you described, but you could also configure the maven-source-plugin to use the classifier option, instead of the default sources value to have then sources${branch.classifier}, for instance. – Burkett

right but then wouldn't eclipse break? Or would it get the change via m2eclipse? – Rosinarosinante

@PeterKahn good point, I haven't tested it, but I guess it would indeed not resolve it properly indeed – Burkett

A_Di-Matteo I think I got it, please see answer below. Tried to @ you but it didn't seem to work as expected – Rosinarosinante

This results in rival poms pushed into local repo and nexus/artefactor repos. Mainline without branchSpecificClassifiers, featureBranch with. Therefore, if FB builds last, then all develops on team get its pom files download with bogus dependencies. I don't think works at all. There seems to be no way to accomplish the goal iv maven. Looks like making git smart about pom merges or generating poms on the fly are the only options. – Rosinarosinante

Why with bogus dependencies? What do you mean? This approach is used by one team since one year without any issue and works fine. – Burkett

During build of any module, mvn publishes artifactID.pom containing the dependencies. In this approach, mainline and feature branch builds publish the same artifactId.pom files containing rival data for dependencies. This results in failed builds for both as the two share the same definition of required dependencies. So, if feature branch builds last it publishes poms with dependencies containing branchClassifier which get used by both feature branch and by mainline. Seems like version or setting up per branch distributionMng repo are only seperators – Rosinarosinante

But why dependencies also have classifiers? Dependencies don't need to have the classifiers as long as you build from the parent pom (recommended approach anyway). Sounds like you went further but you are now blaming the original simple approach. I will test further this approach anyway, thanks for these valuable feedbacks! – Burkett

Here's why Part I: We have some many custom classifier (dist - for tar containing the distribution kit with libs and extra files, doc - containing the docs for a module). Our installer depends on these dist and doc artifacts to unpack-dependencies and then build the installer. At present, we list them in our dependency management section and in the installer's pom. – Rosinarosinante

Here's why part II: We also have modules depend on classifier tests from other modules. So, in that case I must list the dependency scope test, classifier tests in order for the 2nd module to use utility methods from the first. I guess we could refactor to migrate the test utilities into their own module and split each module to produce just one artifact - migrating the dist and doc into separate modules. That might enable this approach to work. – Rosinarosinante

I think we found another way, requiring fewer pom changes. We plan to use plugins and nexus staging repos to isolate the build giving it a different maven.local.repo and remote repo per branch based on branch name. Using something like this github.com/egineering-llc/gitflow-helper-maven-plugin. I'll add an answer to this question when I've completed the requirements / design / testing – Rosinarosinante

@PeterKahn as far as I understand from your explanation, you actually had further requirements and levels of complexity, not described in the original questions and of course not taken into consideration in the initial answer, that's why I found your dowvote a bit controversial, but I'm not gonna troll on this, that's not a problem anyway. Concerning the potential solution, please let me know about it, since I'm really interested into it. Thanks! – Burkett

Good point. Really a neitral no vote for me. I agree, there's much more complexity here. I'll post my attach when I've competed it. I need to write a plugin to interact with nexus and create repos when required. So, I'll post something in a week or so. – Rosinarosinante

Curious to know how the maven-jar-plugin would work in conjunction with maven-deploy-plugin. Deploy plugin seems to ignore the branch.classifier property. – Aileneaileron

Answer 2

This does not work Results in warnings throughout the build and GC error when run from top parent.

Ideally, we want to use version as to differentiate feature branch from mainline because it is the normal maven way and classifier manipulation can result in all kinds of issues.

Since maven can use environment variables for properties and we already initialize build environment with a script (we also have git hook scripts that can set environment variables from branch names) we can use env to control the version.

<groupID>my.project</groupId>
<artifactID>database</artifactId>
<version>1.2.0${env.BRANCHMODIFIER}-SNAPSHOT</version>

If on develop our scripts set BRANCHMODIFIER to ""
If on feature/JIRA-30495 our scripts set BRANCHMODIFIER to ".30495"

How does this work in eclipse or Intellij? No clue as of yet.

Rosinarosinante

A_Di-Matteo our complex use of classifiers became a mess quickly. Using #nexus #staging lead to local repos with what looked like develop artifact but which were not. Adding an artifactId modifier was likewise messy. Manipulating version seems the cleanest approach so far. – Rosinarosinante

This sort of works but results in warnings an a heap problem when run from top of tree. – Rosinarosinante

This approach may have issue when performing builds from local machines, right? – Burkett

Answer 3

We use a similar technique as Peter Kahn, modifying the version of the branch before building. We have three steps in our "Pre Steps":

1. Execute shell: echo VERSION=$(echo ${GIT_BRANCH} | sed 's_^.*\/__') > env.properties
2. Inject environment variables: env.properties
3. Invoke top level Maven targets: versions:set -DgenerateBackupPoms=false -DnewVersion=${VERSION}-SNAPSHOT

I am quite sure that this can be done with two or even one and only step as well, bit the principe behind it will be the same.

The reason why we don't change the version in the pom.xml files in the branch directly is indeed merging. With SVN this was possible (merging with --accept-mine-conflict. With GIT this does not exist anymore, so we stopped changing versions and created this pre-build steps.

Industrials

Answer 4

For Maven ge 3.5.0 try this https://maven.apache.org/maven-ci-friendly.html This is the recommended Maven solution. The only problem (but unusual) can be the numeric versions resolution of the maven dependencies. But this only apears, if you use different SNAPSHOT dependencies of the the module, which is a bad idea anyway.

Mobility