Pip install a private repo from Gitlab with Personal Access Token on Gitlab-CI

Question

I'm using Gitlab to hosts two private Python packages. One, say package B, depends on the other, say A.

I would like to setup correctly package B, so I have a setup.py with

install_requires=[
    'packagea @ git+https://gitlab.com/me/packagea.git',
    ...
]

as well as a requirement.txt containing

git+https://gitlab.com/me/[email protected]

And that works well when a user installs package B with pip install git+https://gitlab.com/me/packageb.git, or when a user clones package B and then run pip install -r requirements.txt.

However, I would like to setup continuous integration on package B. My gitlab-ci.yml looks like

image: python:3.7
before_script:
  - pip install -r requirements.txt

pylint:
  script:
    - pylint **/*.py

This fails because the Gitlab-CI runner does not have any username/password to authenticate and pull package A repo. I've read that I could generate a Personal Access Token for the CI runner to authenticate, but this would mean that setup.py and requirements.txt contain the token. This seems ugly to me (I'd like the other users to still use their own username/password).

How can I achieve this?

Blandishments

Answer 1

In case anyone is like me and gets lost trying to address this pipeline issue, the solution that helped me was this Gitlab issue and the discussion below.

To summarize:

In .gitlab-ci.yml add these lines to your before_script section:

before_script
  - ...
  - echo "machine gitlab.com" > ~/.netrc
  - echo "login gitlab-ci-token" >> ~/.netrc
  - echo "password ${CI_JOB_TOKEN}" >> ~/.netrc
  - if [ -f "requirements.txt" ]; then pip install -r requirements.txt ; fi

In your requirements.txt file, specify the private dependency with the name, url, and egg (the egg must be the package name). e.g.

thispackageisonpip>=1.2.3
soisthisone==1.0.0
...
privatepackage @ git+https://gitlab.com/user/repository.git@tag#egg=privatepackage

where tag is e.g. 'v1.3.3' (note that there must be a tagged version to check out).

The private package should now be checked out successfully by the pipeline. I haven't been able to get this to work where requirements are specified in install_requires of setup.py, but at least this is workable.

Brote

Thanks for the trick! Going throw the discussion you mentioned, it’s not clear if this is actually a feature or a side effect of the implementation… but in the meantime, that’s pretty neat! Thanks! – Blandishments

Answer 2

In place of personal access token, I figured it out with the generated token when a CI Pipeline is triggered.

In this way, I solved this problem by using the user from pipeline (gitlab-ci-token) and also the token from pipeline (CI_JOB_TOKEN).

Then, inside the Dockerfile, I added the following commands before the pip install -r requirements.

... 

ARG CI_JOB_TOKEN 
RUN sed -i 's/<GITLAB_URL>/gitlab-ci-token:${CI_JOB_TOKEN}@<GITLAB_URL>/g' requirements.txt

...

After that I build the container in the .gitlab-ci.yml with the following command.

docker build . --build-arg CI_JOB_TOKEN=${CI_JOB_TOKEN} -t <REGISTRY_URL>:<TAG>

I hope that can help someone!

Vizier

Answer 3

Since package A is a private repo, you will need permissions to access it in any scenario.

You might consider using GitLab Deploy Keys. That way, any user (including the ci) with the deploy key can read or write to the repo (depending on the access you give the key). This ties the key to the repo, rather than to your user account.

It is important to note that this deploy key gives access to the private repo, so it is a secret and should be treated as such.

Secrets management is a complex, subjective topic with solutions varying significantly based on many factors, including how secure you need your project to be, ease of use, and team size. A common way to avoid storing passwords in plaintext is to use custom environment variables. If you are looking for recommendations on how to set up a project this way, this post might be a good place to start.

Clavicorn

Thanks @Lisa! I'm still having troubles understanding how to use this deploy key with my CI. I know that you can use it by changing the URL to package A, but this means having the key hard-coded in setup.py and requirements.txt. – Blandishments

I've updated my solution with more resources on secrets management – Clavicorn

Thanks a lot Lisa for your detailed answer. That’s a legit workaround, but it definitely requires to manually install each dependency (with the proper token) in the gitlab-ci.yaml script. That means maintaining the list of dependencies at multiple places, not to mention the creation and definition of tokens for each of those dependencies 😑. – Blandishments

Answer 4

With command-line access, e.g.

pip3 install -U git+https://gitlab.mycompany.com/foo/bar.git

using a personal access token, when you are prompted for a username, you enter any (non-empty) string. Then, when you are prompted for a password for that username, enter your personal access token.

Costumier