Upload to pypi from Gitlab Pipelines

Question

I'm trying to upload a package to pypi using a Gitlab CI job, but I cannot make it work :/ Anyone has a working example?

What I have tried so far in my .gitlab-ci.yaml (from my local machine all of them are working):

1. Twine with a .pypirc file

   - echo "[distutils]" >> ~/.pypirc
   - echo "index-servers =" >> ~/.pypirc
   - echo "    pypi" >> ~/.pypirc
   - echo "" >> ~/.pypirc
   - echo "[pypi]" >> ~/.pypirc
   - 'echo "repository: https://upload.pypi.org/legacy/" >> ~/.pypirc'
   - 'echo "username: ${PYPI_USER}" >> ~/.pypirc'
   - 'echo "password: ${PYPI_PASSWORD}" >> ~/.pypirc'
   - python3 setup.py check sdist bdist  # This will fail if your creds are bad.
   - cat ~/.pypirc
   - twine upload dist/* --config-file ~/.pypirc
   

2. Same as before but with $VARIABLE

   [...]
   - 'echo "username: $PYPI_USER" >> ~/.pypirc'
   - 'echo "password: $PYPI_PASSWORD" >> ~/.pypirc'
   [...]
   

3. Two options before but using python setup.py ... upload

4. twine upload dist/* -u $PYPI_USER -p $PYPI_PASSWORD
5. twine upload dist/* wiht TWINE_USERNAME and TWINE_PASSWORD environment variables.

... and always get a 403 Client Error: Invalid or non-existent authentication information. I'm running out of options...

Something

How did you define PYPI_USER and PYPI_PASSWORD? – Tawnyatawsha

Oops! I forgot about that. All variables I talk about are set via Settings > CI/CD > Secret variables (as protected, for the same branches I am running these jobs). – Something

Note that using CI_JOB_TOKEN is now (Sept. 2020, GitLab 13.4) possible. See my revised answer below. – Kweiyang

Answer 1

I got this working, using a modified version of your code:

pypi:
  stage: upload
  script:
  - pip install twine
  - rm -rf dist
  - echo "[distutils]" >> ~/.pypirc
  - echo "index-servers =" >> ~/.pypirc
  - echo "    nexus" >> ~/.pypirc
  - echo "" >> ~/.pypirc
  - echo "[nexus]" >> ~/.pypirc
  - echo "${PYPI_REPO}" >> ~/.pypirc
  - echo "${PYPI_USER}" >> ~/.pypirc
  - echo "${PYPI_PASSWORD}" >> ~/.pypirc
  - python3 setup.py check sdist bdist  # This will fail if your creds are bad.
  - python setup.py sdist bdist_wheel
  - twine upload -r nexus dist/*.tar.gz

The difference is I didn't use the "'" and got rid of the colons in the yaml; instead I set the values of the secrets as e.g., username: myuser

Roorback

(Sorry for the delay in check your answer) I've managed it to work, but making Gitlab variables "protected" instead of selecting an environment for them. When using environment dropdown, I get "SECURE" for the value of the variable instead of the actual one. :/ – Something

use "envsubst" to simplify your script – Czarist

Answer 2

I am simply using the TWINE_USERNAME and TWINE_PASSWORD variables, it worked out of the box.

This is the relevant part in my gitlab-ci.yml (replace the image with your desired one and of course change all the other stuff like stage, cache etc. to your needs):

pypi:
    image: docker.km3net.de/base/python:3
    stage: deploy
    cache: {}
    script:
        - pip install -U twine
        - python setup.py sdist
        - twine upload dist/*
    only:
        - tags

And add the environment variables in GitLab under Settings->CI/CD->Variables (https://your-gitlab-instance.oerg/GIT_NAMESPACE/GIT_PROJECT/settings/ci_cd):

Here is the successful pipeline:

Kenric

EDIT: This configuration works but when you got "403 Invalid or non-existent authentication information." than change password to avoid some special chars like $ and or %. Best regards. – Overlie

Thanks for sharing this great guide. Works like a charm. Quick note for anyone else trying this: The option "protected" does not work, but "masked" does. – Fleam

there's no such stage release and twine does not see environment variables in this configuration somehow – Snappish

Note: I was able to get this to work with both "protected" and "masked" – Astragalus

Answer 3

Note that GitLab 12.10 (April 2020) will offer in its premium or more edition, a simpler way, using CI_JOB_TOKEN (See below the second part of this answer, with GitLab 13.4, Sept. 2020)

Build, publish, and share Python packages to the GitLab PyPI Repository

Python developers need a mechanism to create, share, and consume packages that contain compiled code and other content in projects that use these packages. PyPI, an open source project maintained by the Python Packaging Authority, is the standard for how to define, create, host, and consume Python packages.

In GitLab 12.10, we are proud to offer PyPI repositories built directly into GitLab! Developers now have an easier way to publish their projects’ Python packages. By integrating with PyPI, GitLab will provide a centralized location to store and view those packages in the same place as their source code and pipelines.

In March, we announced that the GitLab PyPI Repository and support for other package manager formats will be moved to open source.
You can follow along as we work to make these features more broadly available in the epic.

See Documentation and Issue.

---

And with GitLab 13.4 (September 2020)

Use CI_JOB_TOKEN to publish PyPI packages

You can use the GitLab PyPI Repository to build, publish, and share python packages, right alongside your source code and CI/CD Pipelines.

However, previously you couldn’t authenticate with the repository by using the pre-defined environment variable CI_JOB_TOKEN.
As a result, you were forced to use your personal credentials for making updates to the PyPI Repository, or you may have decided not to use the repository at all.

Now it is easier than ever to use GitLab CI/CD to publish and install PyPI packages by using the predefined CI_JOB_TOKEN environment variable.

See Documentation and Issue.

---

You also have GitLab 16.2 (July 2023), which proposes:

Import PyPI packages with CI/CD pipelines

Have you been thinking about moving your PyPI repository to GitLab, but haven’t been able to invest the time to migrate?
In this release, GitLab is launching the first version of a PyPI package importer.

You can now use the Packages Importer tool to import packages from any PyPI-compliant registry, like Artifactory.

See Documentation and Issue.

Kweiyang

Answer 4

If problems with EOF appears, make sure to change Settings/Repository/Tags to be protected, so they will work again. I've posted here a more complete description.

Onofredo

Answer 5

You can also upload python package to a private Pypi server in one line (I am using it with gilab-ci):

1. Set environment variables PYPI_SERVER, PYPI_USER and PYPI_PASSWORD through Gitlab CI settings

2. Call

   twine upload --repository-url ${PYPI_SERVER} --username $PYPI_USER --password $PYPI_PASSWORDD $artifact
   

Note: I had to use twine from PIP (pip3 install twine) and not from my Ubuntu package as the version 10 of twine seems to have a bug (zipfile.BadZipFile: File is not a zip file).

Longley

Answer 6

You can also look into using dpl: Here's how I'm doing it:

pip:
  stage: upload
  script:
    - apt-get update -qy
    - apt-get install -y ruby-dev
    - gem install dpl
    - python setup.py sdist
    - dpl --provider=pypi --user=$PIP_USERNAME --password=$PIP_PASSWORD --skip_existing=true
  only:
  - master

You can set $PIP_USERNAME and $PIP_PASSWORD in the variables section for your project: settings -> CI/CD -> Variables

Judaism

Answer 7

I know this is an old question, but if you're using poetry (I'm testing with version 1.1.11) you can do it quite easily, like this:

poetry config repositories.my_private_repo [URL_TO_YOUR_PYPI_REPO]
poetry config http-basic.my_private_repo [USERNAME] [PASSWORD]
poetry build
poetry publish --repository my_private_repo

On develop branches, you can add the --dry-run argument to poetry publish so it won't actually get uploaded

Tuner