How do I pass credentials to pull a submodule in a Gitlab CI script?

Question

I have several projects, each in their own repository, that import a common library which has its own repository as well. So, the .gitmodules file includes the library with the full name:

Submodule 'xx/yy' (https://gitlab.com/xx/yy.git) registered for path 'xx/yy'

but this doesn't work:

Fatal: could not read Username for 'https://gitlab.com': No such device or address

the CI script is very simple:

image: mcr.microsoft.com/dotnet/core/sdk:3.0.100-preview9-alpine3.9

variables:
  GIT_SUBMODULE_STRATEGY: recursive

stages:
    - build

before_script:
    - "cd xx"
    - "dotnet restore"

build:
    stage: build
    script:
      - "cd xx"
      - "dotnet build"

The old answer was: GitLab pull submodules inside CI

but things have changed and we can, according to the docs, have submodules that don't have a relative path, as written here: https://docs.gitlab.com/ce/ci/git_submodules.html

Blackandwhite

I am not sure and never used it, but if I take a look to the documentation you have to do a sync in your before script - git submodule sync --recursive - git submodule update --init --recursive – Doti

To get credantials to your gitlab-ci.yaml you can set variables in the web interface and use it in the yaml. There are also some default variables starting with ci. I use it to authenticate to my docker registry, but I did not see one for login to the server docs.gitlab.com/ee/ci/variables – Doti

I will look into that, thanks! – Blackandwhite

Answer 1

tldr; like this:

# .gitlab-ci.yml

stages:
  - build

job1:
  stage: build
  before_script:
    - git config --global credential.helper store
    - git config --global credential.useHttpPath true
    - |
        git credential approve <<EOF
        protocol=https
        host=gitlab.com
        path=my-group/my-submodule-repo.git
        username=${CI_DEPENDENCY_PROXY_USER}
        password=${CI_DEPENDENCY_PROXY_PASSWORD}
        EOF
    - git submodule update --init --recursive

  script:
    - echo "Let's start the build..."

Explanation

The stages: - build and job1: stage: build declarations are boilerplate --- they inform the gitlab ci machinery that there exists one stage (named build) and one job that "belongs" to this stage.

The before_script part details things that need to happen early in the job --- everything thereunder must complete before script is started.

The git config --global credentials.helper tells git to use the credentials helper named "store". By default, this is a cleartext file located at ~/.git-credentials containing newline-delimited username-password-decorated URIs, each corresponding to a given git remote added by the user.

The git config --global credentials.useHttpPath tells git not to ignore the path attribute for any call (explicit or otherwise) to git credential. This is not strictly necessary, but rather good practice when, for example, you have multiple git remotes on the same host.

The git credential approve reads standard input (expressed as a heredoc) and passes the given credential to the credential.helper, namely store, to be written into ~/.git-credentials.

The git submodule update --init --recursive populates the existing (but as yet incomplete) superproject worktree with the content referenced by .gitmodules.

Assumptions

This aforementioned example makes the following assumptions:

• The superproject .gitmodules contains a reference to the submodule remote https://gitlab.com/my-group/my-submodule-repo.git.
• The git submodule remote is private; i.e., access thereto requires credentials in the form of a username and password ("personal access token" in gitlab parlance)
• You want to authenticate to this remote using the Gitlab CI Dependency Proxy credentials described here: https://about.gitlab.com/blog/2020/12/15/dependency-proxy-updates/ .
• You don't want to re-write the .gitmodules to use relative URLs as suggested here: https://docs.gitlab.com/ee/ci/git_submodules.html#configure-the-gitmodules-file .

References:

• https://docs.gitlab.com/ee/ci/yaml/#before_script
• https://docs.gitlab.com/ee/ci/git_submodules.html#using-git-submodules-in-your-ci-jobs
• https://docs.gitlab.com/ee/ci/yaml/README.html#git-submodule-strategy
• https://gitlab.com/gitlab-org/gitlab/-/issues/208770
• https://git-scm.com/docs/gitcredentials
• https://git-scm.com/docs/git-credential-store
• https://git-scm.com/docs/git-credential

Liquate

Is there a version of this where .gitmodules is used? In my case I have multiple submodules (all can be accessed via the same credentials) that are already inside the gitmodules config file. My super-repo is there just to add automated packaging (PyPI) without having developer access to the referenced repos. – Rajah

@Rajah I can imagine re-writing the pre_script block to support dynamic discovery of the git remotes within .gitmodules and then, for each, invoking git credential approve. Although, if every submodule resides under gitlab.com, it's easier to git config credentials.useHttpPath false. – Liquate

Thanks. I fixed it. All I had to do is (yes, in the before_script) add: git config --global credential.helper store followed by echo "https://${CI_REGISTRY_USER}:${CI_JOB_TOKEN}@gitlab.cc-asp.fraunhofer.de" > ~/.git-credentials. After that it's your standard init, sync and update recursively using the git submodule arguments. Thanks. :) – Rajah

Works perfectly! I used it without useHttpPath and path= which suited my use case better. – Cahra

Answer 2

Accepted answer didn't work for me (using a self-hosted GitLab v16 with two private repositories in a single group). In addition, the whole shebang with customizing git feels odd, though it might be useful in several different use cases for sure.

While looking for a solution, I also found that official documentation on using submodules in CI jobs. But it wasn't solving my issue.

There is one line that kept me thinking, though:

If you use the CI_JOB_TOKEN to clone a submodule in a pipeline job, the user executing the job must be assigned to a role that has permission to trigger a pipeline in the upstream submodule project.

Frankly, I still don't have a clue what this means in particular and I'd love to see more information on that in the official documentation. However, clicking on the link labelled CI_JOB_TOKEN I found another page with promising information.

Eventually, I succeeded with:

1. either using absoulute HTTPS URL in .gitmodules file (in case you don't use SSH-based logins at all) or add variable GIT_SUBMODULE_FORCE_HTTPS to enforce automatic mapping of SSH URLs into HTTPS URLs,

2. go to the repo of the submodule,

3. open Settings > CI/CD from left-hand side menu and expand Token Access section,

4. make sure Allow access to this project with a CI_JOB_TOKEN switch is enabled and

5. paste the pulling repository's path without .git extension into that field right below that switch and hit Add Project.

After that, everything's working fine with a .gitlab-ci.yml as simple as this:

variables:
  GIT_SUBMODULE_STRATEGY: recursive

build:
  script:
    # just assume submodule is available and build your stuff here
    - ...

Dorton

This works like a charm. In my case I had to paste the project relative path like group/project, maybe because I use my own instance of Gitlab. – Teahouse

Downside is a library that ought to be submoduled from lots of applications must whitelist them all. Sort of tedious and one of the sharpest edges in GitLab, but I'd rather do this than the other answer. – Noma

Answer 3

See Getting GitLab CI to clone private repositories, and in particular the answers based on insteadOf.

The advantage to this approach is that it's more independent of GitLab, in case you need to move to another CI/CD system later (not so tied to their YAML options). It also lets you clone only certain submodules of submodules rather than all of them (GitLab doesn't have YAML for this, so you'll need to use git).

You'll still need to follow the instruction in Thomas Urban's answer to allow access from the other projects (if they're in GitLab).

Langrage

Answer 4

Unfortunately @therealneil's answer does not work anymore, as username and token are put in the url. So here is a working version based on his work:

script:
    # Remove username and password from url
    - git remote set-url origin $(echo "$CI_REPOSITORY_URL" | sed "s#^\(.*://\).*@#\1#")
    # Activate password cache. You can extend the duration using --timeout
    # see https://git-scm.com/docs/git-credential-cache
    - git config --global credential.helper cache
    - |
      git credential approve <<'EOT'
      protocol=https
      host=gitlab.com
      username=gitlab-ci-token
      password=$AUTHTOKEN
      EOT
    - git submodule update --init `cat etc/submodules.test`

My solution also uses the cache instead of the store credential helper, which does not leave traces of your token on the disk.

P.S.: I tried to set the host using host=${CI_SERVER_HOST}, but that didn't work. Hints are welcome!

Heisel