Gitlab diff between Merge Request Target branch and Source Branch in pipeline

Question

My use involves finding a diff between Merge Request and Master or Release branch (whichever branch I want to diff against) the MR will have CI_MERGE_REQUEST_TARGET_BRANCH set.

CHANGED_DIRS=$(git diff --name-only ${CI_MERGE_REQUEST_TARGET_BRANCH_NAME}...${CI_COMMIT_REF_NAME} | xargs -L1 dirname | uniq | cut -d'/' -f1 |tail -n+2 | uniq)

But it seems that gitlab doesn't recognize the Merge request target branch diff and gives this output.

fatal: ambiguous argument 'master...source': unknown revision or path not in the working tree.

How do I find the diff (which are directories). Appreciate help !

Degeneration

Did you run git fetch first? – Nesselrode

no - but git fetch what ? By default gitlab first init the repo and does the git add remote and checkout the branch (that is merge request branch) . Do you mean I should git fetch origin master first before diff ? – Degeneration

get fetch by itself, without arguments, it will fetch all branches. By default gitlab first init the repo and does the git add remote and checkout the branch (that is merge request branch) No, kind of, gitlab-ci ends up on detached head without branch checkout, but it depends on configuration. There's even a default number of commits checked in project configuration, I think default is 50. – Nesselrode

ok still got the same error , I am surprised how it works locally but not in gitlab ci , strange ``` * [new branch] master -> origin/master fatal: ambiguous argument 'origin/master...whatever': unknown revision or path not in the working tree. ``` – Degeneration

But yes git fetch did got all branches correctly. so locally I assume it must have the same structure as my local – Degeneration

Try git checkout master and git pull also. – Nesselrode

git fetch && git checkout ${CI_MERGE_REQUEST_TARGET_BRANCH_NAME} && git pull , this worked. – Degeneration

Answer 1

This is not really a Git issue, but rather a GitLab-CI issue. However, these particular issues crop up in multiple CI/CD systems, always with the same general reason:

• Making a full clone of a large repository is slow.
• CI systems often start by cloning a repository.
• Hence, CI systems often start by making a less-than-full clone of the repository.
• Git offered just two ways to do this in the past:¹ shallow clones and single-branch clones. In fact, they tend to go together: making a shallow clone, with git clone --depth number, also makes a single-branch clone unless you forcibly override Git's default by adding --no-single-branch as well.

Now, the clone command in any given CI/CD system may be controllable, or may be done by the CI/CD system before any software knobs that are available to you-as-a-developer are provided. If you can affect the command, that's usually the way to go, but if not, you may still be able to correct the problem by running additional Git commands (this again depends on the CI/CD system):

• git remote set-branches --add allows you to add various branches to an otherwise single-branch clone;
• git fetch --depth number, git fetch --deepen number, and git fetch --unshallow are three ways to alter the depth of an existing shallow clone, or remove the depth restrictions entirely.

If your CI/CD system makes a shallow, single-branch clone, and you need a deeper clone with more remote-tracking names, you can get that using git remote and/or git fetch.

As I haven't used GitLab's CI system, I cannot provide a recipe here, but based on comments, it sounds like they use --depth 50 by default. If you have a depth knob, setting it to zero (0) is the usual way to disable the --depth argument, which also disables the single-branch-ness.

Do note that regardless of how the system clones your repository, you will generally have only one branch name, or perhaps even none (detached HEAD) when cloning via tag. A full clone will have a full set of remote-tracking names, while a single-branch clone will have only one remote-tracking name until you use git remote to adjust this and run a subsequent git fetch to add more remote-tracking names.

With GitHub Actions, the v2 checkout makes a shallow single-branch clone by default, while the v1 checkout makes a full clone.

---

¹There is now a third method, the so-called partial clone with promisor remotes. This is arguably the right way for a CI system to do the trick, but partial clones are not very good yet, for a lot of reasons, so this probably has to wait for improvements to the implementation.

Being

Thanks @Being , I did exactly the depth on fetch so now I do git fetch --depth 30 for my master/release branches. The problem here was more on Gitlab exactly as in your Merge Requests(i.e. Pull requests) they mark this as a new refs/merge_requests/:id/ AND if you want to do git diff merge_request....master it doesn't recognize the master because it think there is only one branch , so on the container where the CI is running I had to locally fetch and clone the master/release branch to get a proper diff for me to work . But your answer is perfect on clone. – Degeneration

I create NEW branch B from branch A. gitlab CICD yaml $CI_COMMIT_REF_NAME="B" (TARGET), but how-to get the SOURCE name branch (A) – Kenny

Answer 2

As someone who already mentioned, this is because GitLab-CI does a shallow clone and not a full clone. However, rather than engineering a way to do a full clone or using remote comparisons, I use an inbuilt variable in GitLab-CI : CI_MERGE_REQUEST_DIFF_BASE_SHA.

To get the files use : git diff --name-only ${CI_MERGE_REQUEST_DIFF_BASE_SHA}

CI_MERGE_REQUEST_DIFF_BASE_SHA - The base SHA of the merge request diff. Ref

Platelayer

$ MODIFIED_FILES=$(git diff --name-only ${CI_MERGE_REQUEST_DIFF_BASE_SHA}) # collapsed multi-line command fatal: bad object d57392043f07a1fb5b3da95a15f38dc215f17ca0 – Superstratum

Answer 3

I solved this error by changing the command from

git diff --name-only ${CI_MERGE_REQUEST_TARGET_BRANCH_NAME}...${CI_COMMIT_REF_NAME}

to

git diff --name-only remotes/origin/${CI_MERGE_REQUEST_TARGET_BRANCH_NAME}...remotes/origin/${CI_COMMIT_REF_NAME}.

It seems that the gitlab runner doesn't have any local branches. So we have to tell it to compare the difference between remote branches.

Jewish

Answer 4

My repo is fairly big, so I cannot get away with setting an arbitrary depth level or by 'unshallowing' the git clone.

I managed to solve it using the following strategy:

1. Get the base commit hash from GitLab's CI_MERGE_REQUEST_DIFF_BASE_SHA as mentioned by others
2. Check if the feature branch contains the 'base commit' using merge-base
3. If not, 'deepen' the branch using --deepen=<depth> and go back to step 3
4. Repeat steps 2 and 3 for the main branch (in my case master) — you likely want to use rev-list --first-parent here instead of merge-base to reduce the likelihood of finding an unintended shorter path between the two branches via another branch
5. Use merge-tree to simulate what an actual merge will look like and feed that into diff (optional)

It's a bit more elaborate than a simple git fetch or git diff against remote branches, but I found it to be more robust this way.

It ends up producing exactly the same diff that GitLab's UI is showing, and you can also detect merge conflicts in your script and handle them as you please.

I detailed the full solution in this answer to a related question.

Intelligible