Getting GitLab CI to clone private repositories

Question

I have GitLab & GitLab CI set up to host and test some of my private repos. For my composer modules under this system, I have Satis set up to resolve my private packages.

Obviously these private packages require an ssh key to clone them, and I have this working in the terminal - I can run composer install and get these packages, so long as I have the key added with ssh-add in the shell.

However, when running my tests in GitLab CI, if a project has any of these dependencies the tests will not complete as my GitLab instance needs authentication to get the deps (obviously), and the test fails saying Host key verification failed.

My question is how do I set this up so that when the runner runs the test it can authenticate to gitlab without a password? I have tried putting a password-less ssh-key in my runners ~/.ssh folder, however the build wont even add the key, "eval ssh-agent -s" followed by ssh-add seems to fail saying the agent isn't running...

Dedrick

As of 2017, the highest-ranked answer is out-of-date. Marco's answer using GIT_SUBMODULE_STRATEGY is correct. I added this feature specifically to avoid the mess of dealing with injecting SSH keys. – Dumpy

@JonathonReinhart But isn't this solution a securty issue? I could clone any private GitLab repository as long as I know the relative URL? The runner could zip the cloned content and send it via email. – Wildon

@Wildon No it is not an issue. As of GitLab 8.12, CI jobs run as the user that triggered the pipeline, with a token that has a reduced set of privileges. The CI job can only access the same repositories as the person who pushed the code. – Dumpy

Answer 1

I'm posting this as an answer since others weren't completely clear and/or detailed IMHO

Starting from GitLab 8.12+, assuming the submodule repo is in the same server as the one requesting it, you can now:

1. Set up the repo with git submodules as usual (git submodule add git@somewhere:folder/mysubmodule.git)

2. Modify your .gitmodules file as follows

    [submodule "mysubmodule"]
      path = mysubmodule
      url = ../../group/mysubmodule.git
   

   where ../../group/mysubmodule.git is a relative path from your repository to the submodule's one.

3. Add the following lines to gitlab-ci.yml

    variables:
      GIT_SUBMODULE_STRATEGY: recursive
   

   to instruct the runner to fetch all submodules before the build.

Caveat: if your runner seems to ignore the GIT_SUBMODULE_STRATEGY directive, you should probably consider updating it.

(source: https://docs.gitlab.com/ce/ci/git_submodules.html)

Microwatt

I don't favour this approach because I don't think Gitlab should be forcing me to configure submodules in a specific way. I prefer to use Gitlab tokens and keep the configuration in .gitlab-ci.yml. – Kampala

Does this solution work if you want to run git submodule update --init --recursive locally? – Uzzial

In particular, how can one use this .gitmodulesand .gitlab-ci.yml file to run gitlab-runner locally? – Uzzial

It doesn’t work for repositories on another server, that cannot have a relative path. – Tourniquet

@Uzzial Yes if you locally run git submodule update --recursive --remote it works. In my case all my submodules are in gitlab under same group though. – Metzger

That doesn't seem to work anymore, at least not with gitlab-runner 13.3.1. The runner does seem to use that relative path poorly (ie, from the current directory the command is ran from instead of from the repository url). – Snicker

In case the sumodule origin project in Gitlab is private, you need to add the a username and a username's access token to the URL in the .gitmodules file in order for the update to succeed: url = https://<username>:<access_token>@gitlab/../group/mysubmodule.git – Homo

Answer 2

See also other solutions:

• git submodule permission (see Marco A.'s answer)
• job token and override repo in git config (see a544jh's answer)

---

Here a full howto with SSH keys:

General Design

• generating a pair of SSH keys
• adding the private one as a secure environment variable of your project
• making the private one available to your test scripts on GitLab-CI
• adding the public one as a deploy key on each of your private dependencies

Generating a pair of public and private SSH keys

Generate a pair of public and private SSH keys without passphrase:

ssh-keygen -b 4096 -C "<name of your project>" -N "" -f /tmp/name_of_your_project.key

Adding the private SSH key to your project

You need to add the key as a secure environment variable to your project as following:

• browse https://<gitlab_host>/<group>/<project_name>/variables
• click on "Add a variable"
• fill the text field Key with SSH_PRIVATE_KEY
• fill the text field Value with the private SSH key itself
• click on "Save changes"

Exposing the private SSH key to your test scripts

In order to make your private key available to your test scripts you need to add the following to your .gitlab-ci.yml file:

before_script:
  # install ssh-agent
  - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
  # run ssh-agent
  - eval $(ssh-agent -s)
  # add ssh key stored in SSH_PRIVATE_KEY variable to the agent store
  - ssh-add <(echo "$SSH_PRIVATE_KEY")
  # disable host key checking (NOTE: makes you susceptible to man-in-the-middle attacks)
  # WARNING: use only in docker container, if you use it with shell you will overwrite your user's ssh config
  - mkdir -p ~/.ssh
  - echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config

Code Snippet comes from GitLab documentation

Adding the public SSH key as a deploy key to all your private dependencies

You need to register the public SSH key as deploy key to all your private dependencies as following:

• browse https://<gitlab_host>/<group>/<dependency_name>/deploy_keys
• click on "New deploy key"
• fill the text field Title with the name of your project
• fill the text field Key with the public SSH key itself
• click on "Create deploy key"

Moskva

@Ridermansb Thanks for the link. The answer comes from a personal doc. I forgot the origin of the code snippet. It is worth to mention it. Thanks again for having pointed out. – Moskva

Instead of echoing to disable StrictHostKeyChecking (and risking accidentally clobbering a real system's ssh config), add a gitlab CI variable like KNOWN_HOSTS that has the known host(s). See this for details: docs.gitlab.com/ee/ci/ssh_keys/#verifying-the-ssh-host-keys – Brumley

Can I use a name other than SSH_PRIVATE_KEY? – Fiume

becko, you can change the name of the var SSH_PRIVATE_KEY. If you do, be sure to update it everywhere. – Moskva

Security warning: That documentation doesn't use a passphrase on the private key. – Ladybird

Answer 3

If you don't want to fiddle around with ssh keys or submodules, you can override the repo in git's configuration to authenticate with the job token instead (in gitlab-ci.yml):

before_script:
  - git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.example.com/group/repo.git".insteadOf [email protected]:group/repo.git

Eads

I have encountered a problem with this - the git config global persists on next jobs on my runner. It completely breaks the CI. Have you had this problem? I can not clone any resources from my gitlab because the CI_JOB_TOKEN is stale. – Abscind

@Abscind I use the Docker runner so I haven't encountered that problem. – Eads

Me too. Though I solved it today :) Its because of npm... By default npm install modifies package-lock.json. When you set in your CI to only fetch changes instead of cloning, this modified packge-lock.json file persists between jobs and npm cannot install anything... – Abscind

This solution is the most elegant. It works with dockerized GitLab Community Edition 11.6.4 – Micropathology

You can also omit the group and repo to use the job token for all repos on the gitlab instance. See gist.github.com/Kovrinic/ea5e7123ab5c97d451804ea222ecd78a – Imperceptive

please make sure you replace the colon on [email protected]:group/repo to gitlab.com/group/repo. took me a while. – Riser

Why and how does GitLab allows CI_JOB_TOKEN to authenticate to a private repo? How does it know that given repo shouldnt be hidden but accessible to the repo where the CI is running? – Anibalanica

Answer 4

The currently accepted answer embeds Gitlab-specific requirements into my .gitmodules file. This forces a specific directory layout for local development and would complicate moving to another version control platform.

Instead, I followed the advice in Juddling's answer. Here's a more complete answer.

My .gitmodules files has the following contents:

[submodule "myproject"]
    url = [email protected]:mygroup/myproject.git

In my gitlab-ci.yml I have the following:

build:
  stage: build
  before_script:
    - git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}@git.myhost.com/".insteadOf "[email protected]:"
    - git submodule sync && git submodule update --init

The trailing / and : are critical in the git config line, since we are mapping from SSH authentication to HTTPS. This tripped me up for a while with "Illegal port number" errors.

I like this solution because it embeds the Gitlab-specific requirements in a Gitlab-specific file, which is ignored by everything else.

Kampala

To save someone else from making the same silly pattern-matching mistake, for projects hosted on gitlab.com, git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.com/".insteadOf "[email protected]:" – Uzzial

@Gus, can you post a working example of a ci config file? I posted a similar question here: #58040683 but can’t make this work at all – Tourniquet

Answer 5

I used deploy tokens to solve this issue, as setting up SSH keys for a test runner seems a little long winded.

git clone http://<username>:<deploy_token>@gitlab.example.com/tanuki/awesome_project.git

The deploy tokens are per project and are read only.

Ready

Answer 6

One way to solve this without changing the git repository's structure is to perform the following steps:

1. get ssh host keys

Get the ssh host keys of the server that you are running on. For gitlab.com:

1. run ssh-keyscan gitlab.com > known_hosts
2. check that ssh-keygen -lf known_hosts agrees with the fingerprints reported here.
3. copy the content of the known_hosts and paste it on a variable called SSH_KNOWN_HOSTS on the repository.

This step is only needed once.

2. configure the job to use ssh

before_script:
    - git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.com".insteadOf "[email protected]:"
    - mkdir -p ~/.ssh
    - chmod 700 ~/.ssh
    - echo "$SSH_KNOWN_HOSTS" > ~/.ssh/known_hosts
    - chmod 644 ~/.ssh/known_hosts

The "ssh://[email protected]" bit may be different if you are trying to do git clone [email protected]: or pip install -e git+ssh://[email protected]/...; adjust it accordingly to your needs.

At this point, your CI is able to use ssh to fetch from another (private) repository.

3. [Bonus DRY]

Use this trick to write it generically:

.enable_ssh: &enable_ssh |-
    git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.com".insteadOf "ssh://[email protected]"
    mkdir -p ~/.ssh
    chmod 700 ~/.ssh
    echo "$SSH_KNOWN_HOSTS" > ~/.ssh/known_hosts
    chmod 644 ~/.ssh/known_hosts

and enable it on jobs that need it

test:
    stage: test
    before_script:
        - *enable_ssh
    script:
        - ...

Midiron

the [3] worked for me when I needed to get private repositories using gitlab and go get in particular the git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.com".insteadOf "https://gitlab.com" (it's a bit modified) – Heterogenesis

Answer 7

1. If your CI runner is running on a container model, you need to use the deploy key. doc: https://docs.gitlab.com/ee/user/project/deploy_tokens/#git-clone-a-repository

git clone https://<username>:<deploy_token>@gitlab.example.com/tanuki/awesome_project.git

2. Create your deploy token
3. Add your token in CI pipeline Variable
4. make sure your container has the git and change the git URL by insteadOf

  image: docker:latest
  before_script:
    - apk add --no-cache curl jq python3 py3-pip git
    - git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.example.come/".insteadOf '[email protected]:'

for replace URL: https://docs.gitlab.com/ee/user/project/working_with_projects.html#authenticate-git-fetches

Greyso

Answer 8

I had a scenario where I had to use my ssh key in 3 different scripts, so I put the ssh key stuff in a single shell script and called it first, before the other 3 scripts. This ended up not working, I think due to the ssh-agent not persisting between shell scripts, or something to that effect. I ended up actually just outputting the private key into the ~/.ssh/id_rsa file, which will for sure persist to other scripts.

.gitlab-ci.yml

script:
    - ci/init_ssh.sh
    - git push # or whatever you need ssh for

ci/init_ssh.sh

# only run in docker:
[[ ! -e /.dockerenv ]] && exit 0

mkdir -p ~/.ssh
echo "$GITLAB_RUNNER_SSH_KEY" > ~/.ssh/id_rsa
chmod 400 ~/.ssh/id_rsa
echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > /.ssh/config

It works like a charm!

Fungistat

Hi @user3246077! Thank you so much for the answer. I found a mistake, in the last line the ~ symbol to reference the user's home is missing. Can you fix it? – Endemic

Answer 9

If you are using an alpine-based image (maybe docker:latest or docker:dind), your before_script might look like this:

before_script:
    - apk add --no-cache openssh-client git
    - mkdir -p /.ssh && touch /.ssh/known_hosts
    - ssh-keyscan gitlab.com >> /.ssh/known_hosts
    - echo $SSH_KEY | base64 -d >> /.ssh/id_rsa && chmod 600 /.ssh/id_rsa
    - git clone [email protected]:mygroup/myproject.git

Citified

Answer 10

Gitlab 15.9.0 introduces an update to the pre-defined variable CI_JOB_TOKEN. Now you can control other projects' access to your private repository, see the release note and documentation.

Once access is granted, you can clone private repositories by adding this line to your job's scripts or before_scripts.

git clone https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.example.com/<namespace>/<project>

Unfortunately, this still does not play nicely with the submodule integration with Gitlab CI/CD. Instead, I do this in my projects.

# .gitlab-ci.yml
default:
  before_script:
  - git config --global url."https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.example.com/<namespace>/<project>.git".insteadOf '[email protected]/<namespace>/<project>.git'
  - git submodule update --init --recursive

or like this

# .gitlab-ci.yml
default:
  before_script:
  - |
    cat << EOF > ~/.gitconfig
    [url "https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.example.com/<namespace>/<project>.git"]
        insteadOf = [email protected]/<namespace>/<project>.git
    EOF
  - git submodule update --init --recursive

And this is what my .gitmodules would look like

[submodule "my-submodule"]
    path = modules/<project>
    url = [email protected]/<namespace>/<project>.git
    branch = main

Hope this help!

Gmt

Answer 11

Adding this to .gitlab-ci.yml did the trick for me. (as mentioned here: https://docs.gitlab.com/ee/user/project/new_ci_build_permissions_model.html#dependent-repositories)

before_script:
    echo -e "machine gitlab.com\nlogin gitlab-ci-token\npassword ${CI_JOB_TOKEN}" > ~/.netrc

(I tried setting up SSH_PRIVATE_KEY as mentioned in one of the answers above, it won't work)

Cutcheon

Answer 12

A variant of Duncan Jones' answer worked for me, the difference is that my modules are using HTTPS, like so:

[submodule "some-submodule"]
    path = submoduleDir
    url = https://gitlab.com/project-name/some-dir/submodule-app.git

Therfore I had to rewrite the domain gitlab.com to the domain + login/password: gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.com, which gives:

build:
  stage: build
  before_script:
    - apk update && apk add git # I had install git (using docker in docker)
    - git config --global url."gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.com".insteadOf "gitlab.com"
    - git submodule sync && git submodule update --init

Snicker

Answer 13

Seems there is finally a reasonable solution.

In short as of GitLab 8.12 all you need to do is use relative paths in the .submodules, and the git submodule update --init will simply work

Benedicto

This is an incomplete version of Marco A's answer above. – Gyrocompass