How to deploy Python packages to Gitlab Package Registry with Poetry?

Question

I'm trying to set up Poetry to deploy packages to our internal Gitlab Package Registry. According to other sources online the repository ID should be https://gitlab.com/api/v4/projects/<project id>/packages/pypi, but no matter what I try, Poetry returns

[UploadError]
HTTP Error 404: Not Found

Anyone got this working ?

Couteau

Answer 1

If you are attempting to deploy from GitLab CI, GitLab automatically creates a user and token combination that can be used to authenticate in the CI context under the user gitlab-ci-token and the password in the $CI_JOB_TOKEN variable.

All you need to do that is poetry specific is set the config values for poetry to know the package registry exists, then pass this for authentication. All of which can be done just in the CI config/script.

  script:
    - poetry build
    - poetry config repositories.gitlab "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/pypi"
    - poetry config http-basic.gitlab gitlab-ci-token "$CI_JOB_TOKEN"
    - poetry publish --repository gitlab

If you are deploying from outside of GitLab CI, then you would need that access token and to provide values as used in the script above.

Defence

According to the GitLab Docs the PyPi gitlab url should be ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/pypi – Dram

Answer 2

I actually got this working myself, and the above url is correct. My problem was that I tried to publish to a group (with the group id) and not to an actual project (aka repository).

So here is how to do it:

1. Add the repository to you poetry.toml

   [virtualenvs]
   in-project = true
   [repositories]
   [repositories.my-gitlab]
   url = "https://gitlab.com/api/v4/projects/<your project id>/packages/pypi"
   

2. Generate a token in gitlab that can read and write to package repository.

3. Publish the package

   poetry publish --repository my-gitlab -u <token-username> -p <token-password>
   
   

Couteau

You could use https://gitlab.com/api/v4/projects/${env.CI_PROJECT_ID}/packages/pypi to avoid hardcoding the project ID. – Benignity

"My problem was that I tried to publish to a group ..." Then why do groups have a section for packages. if you can't publish to them? – Heyde

@LawrenceI.Siden You can access all published packages from nested projects in a group registry without needing to know the particular project the package is in. Note: I've not done this in the API, just from the group package registry on the group webpage, so not sure any other limitations. – Nut

Answer 3

Here is a full GitLab CI/CD script:

# Use the official Python Docker image
image: python:latest

before_script:
  # Install Poetry for dependency management
  - pip install poetry
  
  # Install project dependencies from pyproject.toml
  - poetry install
  
  # Activate the Poetry-created virtual environment
  - source "$(poetry env info --path)/bin/activate"

publish:
  script:
    # Build the package (creates .whl and .tar.gz files)
    - poetry build
    
    # Set GitLab as the package repository
    - poetry config repositories.gitlab ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/pypi
    
    # Ignoring certificates can be needed for self-hosted GitLab instances
    # - poetry config certificates.gitlab.cert false
    
    # Publish the package to the GitLab repository
    - poetry publish --repository gitlab -u gitlab-ci-token -p ${CI_JOB_TOKEN}

Annihilator