How to manage multiple maven projects in gitlab

Question

We have multiple (about 75) maven projects, which we have pushed to Gitlab individually in following manner:
GROUP1 - 4 projects
GROUP2 - 2 projects
GROUP3 - 10 Projects
.
.
GROUP10 - 6 Projects

---

We now want to implement CI tool like Jenkins. But creating and managing all 75 projects in jenkins individuality is very time consuming. So that we have 2 methods in mind.

1. git submodules: In this case we might get many complications, as individual developer is forking the multiple project and works in his forked repo.
2. Single git Project: In this case if every developer will fork the main repo, then disk space required is much and the performance will also get degrade. But it will be better than git submodules to manage, as build can be simplified with maven multiple module way.

So according to you, is there any other approach we can think about, or can we go with Single git project?

Swipple

Answer 1

I also manage over 50 Gitlab Maven Projects and use Gitlab CI to build each project and deploy the artifact to our central artifact repository. Gitlab CI does not support re-building consumers of Maven artifacts when a downstream artifact changes. They have a concept of "Triggering" but this simply does not work well when you have so many projects.

I usually push changes in one at a time and wait for the previous commit to fully build and publish to the central repository before pushing in the next commit.

This was very time consuming so as an alternative we did build an internal tool to solve this. I made the tool public a few days ago and it's in the Alpha stage, but you are free to try it if you'd like.

It's called Gitlab Pipes. The documentation has a quick start to get you started quickly.

Durham

Answer 2

Another approach, suggested in "Checkout multiple git repos into same Jenkins workspace" is to use a Jenkins pipeline, in which you can checkout multiple git repositories.

Which means: you can script that checkout, avoiding both submodules and monorepo.

Using the GitLab group API, you can get the list of projects in a given group, which means you could consider one job per group.
Or even one job for all groups.

Argumentative

Last I checked Multi-checkouts will lead to the inability to apply labels back to source control....unless this issue has been fixed...??? – Mccrary

@AndrewGray Not fixed, but you still can script a post-build step applying those label in each subfolders/repos of the workspace. – Argumentative

Answer 3

We do similar but for C++ projects. Imagine you have an xml file which lists all needed repositories and their versions (i.e. branch or tag), which looks like:

<?xml version="1.0" encoding="UTF-8"?>
<manifest>
  <projects>
    <project name="foo-project"
             remote="https://github.com/my-org/foo"
             path="foo"/>
    <project name="bar"
             remote="https://bitbucket.com/other-org/bar"
             path="bar"/>
  </projects>
</manifest>

Now you need a tool to manipulate these files. An old one would be git-repo (https://gerrit.googlesource.com/git-repo/), a new one by Google is jiri (https://github.com/fuchsia-mirror/jiri).

Now, considering you need to build all projects for every group and deploy them, use https://wiki.jenkins.io/display/JENKINS/Repo+Plugin for Jenkins.

So, you'll finish up with 10 manifests stored in your repository, each containing the projects you'll need to build.

Example: you have a project (manifest, group) A which includes subprojects 1,2,3,4. When you change subproject 3, it will fetch the manifest A and will rebuild/redeploy all 4 projects from this group.

Smacking

Answer 4

I believe you did the breakup of your 75 project in group based on certain thought process hence my suggestion is to have have group-wise Jenkins jobs it will give better control also. if intent is to have one job then call these group-wise Jenkins job in sequence.

we are running 7 different Jenkins Jobs for 7 different groups in single Bit-bucket repository.

Armandinaarmando