Cannot push from gitlab-ci.yml

Question

With my colleagues, we work on a C++ library that becomes more and more important each day. We already built continuous integration utilities through the gitlab-ci.yml file that let us:

• Build & Test in Debug mode
• Build & Test in Release mode
• Perform safety checks like memory leaks using Valgrind and checking if there is any clear symbol in our library we don't want inside it
• Generate documentation

All kind of stuff that made us choose GitLab !

We would like to profile our whole library and push the benchmarks in a separate project. We have already done something like for out documentation using the SSH key method but we would like to avoid this this time.

We tried a script like this:

test_ci_push:
  tags:
    - linux
    - shell
    - light
  stage: profiling
  allow_failure: false
  only:
    - new-benchmark-stage
  script:
    - git clone http://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.mycompany.home/developers/benchmarks.git &> /dev/null
    - cd benchmarks
    - touch test.dat
    - echo "This is a test" > test.dat
    - git config --global user.name "${GITLAB_USER_NAME}"
    - git config --global user.email "${GITLAB_USER_EMAIL}"
    - git add --all
    - git commit -m "GitLab Runner Push"
    - git push http://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.mycompany.home/developers/benchmarks.git HEAD:master
    - cd ..

We also tried a basic git push origin master to push our updated files but each time we got the same answer:

remote: You are not allowed to upload code for this project.
fatal: unable to access 'http://gitlab-ci-token:[email protected]/developers/benchmarks.git/': The requested URL returned error: 403

Both projects are under the same site and I have the rights to push in both. Where am I doing anything wrong here ?

Clara

Answer 1

The gitlab ci token is more like the deploy key in github.com, so it only has read access to the repository. To actually push you will need to generate a personal access token and use that instead.

First you need to generate the token as shown here in the gitlab documentation. Make sure you check both the read user and api scopes. Also this only works in GitLab 8.15 and above. If you are using an older version and do not wish to upgrade there's an alternative method I could show you but it is more complex and less secure.

In the end your gitlab-ci.yml should look something like this:

test_ci_push:
  tags:
    - linux
    - shell
    - light
  stage: profiling
  allow_failure: false
  only:
    - new-benchmark-stage
  script:
    - git clone http://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.mycompany.home/developers/benchmarks.git &> /dev/null
    - cd benchmarks
    - echo "This is a test" > test.dat
    - git config --global user.name "${GITLAB_USER_NAME}"
    - git config --global user.email "${GITLAB_USER_EMAIL}"
    - git add --all
    - git commit -m "GitLab Runner Push"
    - git push http://${YOUR_USERNAME}:${PERSONAL_ACCESS_TOKEN}@gitlab.mycompany.home/developers/benchmarks.git HEAD:master
    - cd ..

Alex

I want to use my user token to do a git push while in CI. I have generated an API access token, and I use it in addition of my git https url. That doesn't work, it says that I don't have push permissions while I am the owner. How can I do ? (Without SSH) – Begrudge

Each access token has scopes attached to it, the access token needs the api scope otherwise you can't do anything with it – Alex

I have checked all the scopes, including api scope and I still can't push to my repository. Are you certain that it is possible using a token. I was using ssh manually configured before because I thought it was the one solution – Begrudge

Thankyou! That push step is all I needed to realize how to push my package.json changes for npm versioning! – Antalya

Just a minor annotation to your script code: You don't need to touch test.dat first. When redirecting stdout to a file using >, this file is automatically created. – Lavelle

Please use the ${CI_REPOSITORY_URL}. You can also try ${GITLAB_USER_LOGIN} instead of ${YOUR_USERNAME} – Mcglothlin

doesn't work when the gitlab server uses 2FA as is common in enterprise environment. – Chary

Answer 2

While the previous answers are more or less fine, there are some important gotya's.

  before_script:
    - git config --global user.name "${GITLAB_USER_NAME}"
    - git config --global user.email "${GITLAB_USER_EMAIL}"
  script:
    - <do things>
    - git push "https://${GITLAB_USER_LOGIN}:${CI_GIT_TOKEN}@${CI_REPOSITORY_URL#*@}" "HEAD:${CI_COMMIT_TAG}"

For one, we only need to set the username/email to please git.

Secondly having it in the before script, is not super crucial, but allows for easier reuse when doing 'extend'.

Finally, pushing https is 'fine' but since we're not using a stored ssh key, we should avoid anything that can reveal the token. For one, while gitlab won't print the token in this command, git will happily inform us that the new upstream is set to https://username:thetokeninplaintexthere@url So there's your token in plain text, so don't use -u to set an upstream.

Also, it's not needed, we are only doing a single push.

Further more, when determining the URL, I found that using the exist CI_REPOSITORY_URL to be the most reliable solution (when moving repo's for example or whatnot). So we just replace the username/token in the URL string.

Aramanta

can you tell us what @${CI_REPOSITORY_URL#*@} exactly is? How is the replacement working and where do I find a reference to this syntax? – Leroy

The @ is actually part the HTTP protocol, saying you want to login with username:password@hostname. The variable: ${CI_REPOSITORY_URL} is the variable. "#*@" is regular expression (regex), but I also don't know what it does. Alternatively, you can also use: ${CI_SERVER_HOST}/${CI_PROJECT_PATH}.git after the @-symbol. – Mcglothlin

Instead of using the commit tag. I'm actually using "HEAD:${CI_DEFAULT_BRANCH}" – Mcglothlin

This solved my issue but then I went with using a private runner and using ssh key of the runner – Croaky

#*@ is substring removal, will remove everything from begin to @ – Achondrite

I was wondering as well, why this ${CI_REPOSITORY_URL#*@} was used. The comment from @miaonster put me on the right track: ${CI_REPOSITORY_URL} expands to something like https://gitlab-ci-token:[masked]@example.com/gitlab-org/gitlab-foss.git as you can see from the gitlab docs – and since we only need everything after the "@", this is done ${CI_REPOSITORY_URL#*@} and we get example.com/gitlab-org/gitlab-foss.git. – Nunnery

Answer 3

You could also provide user and password (user with write access) as secret variables and use them.

Example:

before_script:
 - git remote set-url origin https://$GIT_CI_USER:$GIT_CI_PASS@$CI_SERVER_HOST/$CI_PROJECT_PATH.git
 - git config --global user.email '[email protected]'
 - git config --global user.name 'MyUser'

You have to define GIT_CI_USER and GIT_CI_PASS as secret variables (you could always create dedicated user for this purpose).

With this configuration you could normally work with git. I'm using this approach to push the tags after the release (with Axion Release Gradle Pluing - http://axion-release-plugin.readthedocs.io/en/latest/index.html)

Example release job:

release:
  stage: release
  script:
    - git branch
    - gradle release -Prelease.disableChecks -Prelease.pushTagsOnly
    - git push --tags
  only:
   - master

Repugnant

I updated this answer with $CI_SERVER_HOST instead of gitlab.com. – Mcglothlin

Is $GIT_CI_USER the same as user.name 'MyUser' ? – Quid

Answer 4

I'm using the following GitLab job:

repo_pull_sync:
  image: danger89/repo_mirror_pull:latest
  rules:
    - if: '$CI_PIPELINE_SOURCE == "schedule"'
    - if: $REMOTE_URL
    - if: $REMOTE_BRANCH
    - if: $ACCESS_TOKEN
  before_script:
    - git config --global user.name "${GITLAB_USER_NAME}"
    - git config --global user.email "${GITLAB_USER_EMAIL}"
  script:
    - git checkout $CI_DEFAULT_BRANCH
    - git pull
    - git remote remove upstream || true
    - git remote add upstream $REMOTE_URL
    - git fetch upstream
    - git merge upstream/$REMOTE_BRANCH
    - git push "https://${GITLAB_USER_LOGIN}:${ACCESS_TOKEN}@${CI_REPOSITORY_URL#*@}" "HEAD:${CI_DEFAULT_BRANCH}"

I'm using my own danger89/repo_mirror_pull docker image based on alpine, check this GitHub repository for more info.

This GitLab job pull upstream changes from the predefined remote repository + branch (see variables below), and merge them locally in CI/CD and pushes them in GitLab again.

Basically I created a repository pull mirror (which is officially not available for free on GitLab CE, only a push mirror is supported in GitLab).

• Create in GitLab a Project Access Token first in GitLab. Via: Settings->Access Tokens. Check 'api' as the scope.
• Create a new schedule, via: CI/CD->Schedules->New schedule. With the following 3 variables:
  • REMOTE_URL (example: https://github.com/project/repo.git)
  • REMOTE_BRANCH (example: master)
  • ACCESS_TOKEN: (see Access Token in the first step! Example: gplat-234hcand9q289rba89dghqa892agbd89arg2854, )
• Save pipeline schedule

Again, see also: https://github.com/danger89/repo_pull_sync_docker_image

Regarding the question, see the git push command above, which allows you to push changes back into GitLab using GitLab (project) access token.

Mcglothlin

Answer 5

Check if GitLab 15.9 (February 2023) can help.

First, the documentation "GitLab CI/CD job token" does mention:

The token has the same permissions to access the API as the user that caused the job to run.

A user can cause a job to run by taking action like pushing a commit, triggering a manual job, or being the owner of a scheduled pipeline.
Therefore, this user must be assigned to a role that has the required privileges.

Second, GitLab 15.9 (for Saas gitlab.com or self-managed instances) proposes:

Control which projects can access your project with a CI/CD job token

The CI/CD job token stored in the CI_JOB_TOKEN CI/CD variable makes authenticating API calls within jobs more intuitive, enabling advanced automation.
For example, the token can be used with bot automation projects that run pipelines in other projects.
The token is short-lived, but has the same permissions as the user that triggered the pipeline.

In an effort to make its usage even more secure we are adding a setting that lets you define a list of trusted projects that you allow to use job tokens to access your project.
This added layer of security means that only these projects will be able to access your project’s API with a job token.
In the bot automation example, it lets you control which bot projects can interact with your own project.

This setting is currently disabled by default in existing projects to avoid impacting pipelines, but we strongly recommend enabling it in all your projects.
It is enabled by default for all new projects.

See Documentation and Issue.

Projects added to the allowlist can make API calls from running pipelines by using the CI/CD job token.

So if you need to push to a separate project, add to its inbound access list your current project. And your pipeline (using your current project) will be able to push to your separate project, using the default CI_JOB_TOKEN token.

---

And the allowlist is refined, with GitLab 17.0 (May 2024):

Add a group to the CI/CD job token allowlist

Introduced in GitLab 15.9, the CI/CD job token allowlist prevents unauthorized access from other projects to your project. Previously, you could allow access at the project level from other specific projects only, with a maximum limit of 200 total projects.

In GitLab 17.0, you can now add groups to a project’s CI/CD job token allowlist. The maximum limit of 200 now applies to both projects and groups, meaning a project allowlist can now have up to 200 projects and groups authorized for access. This improvement makes it easier to add large numbers of projects associated with a group.

See Documentation and Issue.

Jacindajacinta

I found this answer doesn't work for me (not sure why). For the same repository, my understanding from this answer is you should be able to commit and push back to the repo in a CI job, but I just get a 403 here: "remote: You are not allowed to upload code". Read-only access seems to be ok, but not write-access. – Baur

@Baur Do you have a GitLab instance 15.9? And did you add your project to the inbound list of the target project you want to push to? – Jacindajacinta

I have this feature enabled on 15.9 and the target repository is itself (so I didn't need to add it). I did make sure the switch was enabled to limit the feature as described. Do you know how to confirm the permissions of CI_JOB_TOKEN? – Baur

I suspect this is still an open issue given the discussion here (gitlab.com/groups/gitlab-org/-/epics/3559#note_846650238). – Baur

@Baur For testing, try and add the project to itself (to its own inbound list). Regarding permission, try curl --header "JOB-TOKEN: $CI_JOB_TOKEN" "https://gitlab.example.com/api/v4/projects/1/members/me", which should return your user ID, username, name, state, access level and expires at date for that project (knowing that, to push to a repository, you need to have at least Developer role for that project). – Jacindajacinta

I'm a maintainer on the project. I cannot add the project to itself (it's already on the inbound list and cannot be deleted or added) and the permissions show the access level is right. I still get a 403 unfortunately, so it must be something about write access. I can confirm read-access is fine tho. – Baur

@Baur OK: asking a separate question might yield more result, especially if you detail the specifics of your current setup. – Jacindajacinta