How can I publish Python packages to CodeArtifact using Poetry?

Question

Trying to publish a Poetry package to AWS CodeArtifact. It supports pip which should indicate that it supports poetry as well since poetry can upload to PyPi servers.

I've configured the domain like so:

export CODEARTIFACT_AUTH_TOKEN=`aws codeartifact get-authorization-token --domain XXXX --domain-owner XXXXXXXXXXXX --query authorizationToken --output text`
poetry config repositories.the_aws_repo https://aws:$[e-mail]/pypi/XXXX/simple/
poetry config pypi-token.the_aws_repo $CODEARTIFACT_AUTH_TOKEN

But I'm getting 404 when trying to publish the package:

❯ poetry publish --repository the_aws_repo -vvv

No suitable keyring backend found
No suitable keyring backends were found
Using a plaintext file to store and retrieve credentials
Publishing xxx (0.1.5) to the_aws_repo
 - Uploading xxx-0.1.5-py3-none-any.whl 100%

  Stack trace:

  7  ~/.poetry/lib/poetry/_vendor/py3.8/clikit/console_application.py:131 in run
      129│             parsed_args = resolved_command.args
      130│
    → 131│             status_code = command.handle(parsed_args, io)
      132│         except KeyboardInterrupt:
      133│             status_code = 1

  6  ~/.poetry/lib/poetry/_vendor/py3.8/clikit/api/command/command.py:120 in handle
      118│     def handle(self, args, io):  # type: (Args, IO) -> int
      119│         try:
    → 120│             status_code = self._do_handle(args, io)
      121│         except KeyboardInterrupt:
      122│             if io.is_debug():

  5  ~/.poetry/lib/poetry/_vendor/py3.8/clikit/api/command/command.py:171 in _do_handle
      169│         handler_method = self._config.handler_method
      170│
    → 171│         return getattr(handler, handler_method)(args, io, self)
      172│
      173│     def __repr__(self):  # type: () -> str

  4  ~/.poetry/lib/poetry/_vendor/py3.8/cleo/commands/command.py:92 in wrap_handle
       90│         self._command = command
       91│
    →  92│         return self.handle()
       93│
       94│     def handle(self):  # type: () -> Optional[int]

  3  ~/.poetry/lib/poetry/console/commands/publish.py:77 in handle
      75│         )
      76│
    → 77│         publisher.publish(
      78│             self.option("repository"),
      79│             self.option("username"),

  2  ~/.poetry/lib/poetry/publishing/publisher.py:93 in publish
      91│         )
      92│
    → 93│         self._uploader.upload(
      94│             url,
      95│             cert=cert or get_cert(self._poetry.config, repository_name),

  1  ~/.poetry/lib/poetry/publishing/uploader.py:119 in upload
      117│
      118│         try:
    → 119│             self._upload(session, url, dry_run)
      120│         finally:
      121│             session.close()

  UploadError

  HTTP Error 404: Not Found

  at ~/.poetry/lib/poetry/publishing/uploader.py:216 in _upload
      212│                     self._register(session, url)
      213│                 except HTTPError as e:
      214│                     raise UploadError(e)
      215│
    → 216│             raise UploadError(e)
      217│
      218│     def _do_upload(
      219│         self, session, url, dry_run=False
      220│     ):  # type: (requests.Session, str, Optional[bool]) -> None

My AWS IAM user has permission to do this since I gave it the relevant permissions in the repo.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::XXXXXXXXXXXX:user/ShayN"
            },
            "Action": [
                "codeartifact:AssociateExternalConnection",
                "codeartifact:CopyPackageVersions",
                "codeartifact:DeletePackageVersions",
                "codeartifact:DeleteRepository",
                "codeartifact:DeleteRepositoryPermissionsPolicy",
                "codeartifact:DescribePackageVersion",
                "codeartifact:DescribeRepository",
                "codeartifact:DisassociateExternalConnection",
                "codeartifact:DisposePackageVersions",
                "codeartifact:GetPackageVersionReadme",
                "codeartifact:GetRepositoryEndpoint",
                "codeartifact:ListPackageVersionAssets",
                "codeartifact:ListPackageVersionDependencies",
                "codeartifact:ListPackageVersions",
                "codeartifact:ListPackages",
                "codeartifact:PublishPackageVersion",
                "codeartifact:PutPackageMetadata",
                "codeartifact:PutRepositoryPermissionsPolicy",
                "codeartifact:ReadFromRepository",
                "codeartifact:UpdatePackageVersionsStatus",
                "codeartifact:UpdateRepository"
            ],
            "Resource": "*"
        }
    ]
}

What am I missing?

Pappas

Answer 1

The problem is the /simple/ at the end of the repo url. This part should only be added when pulling from that repo, not when publishing to it. If you look closely to the documentation of AWS CodeArtifact on how to publish with twine, you'll see that it's also not there.

This works:

# This will give the repo url without the /simple/ part
# Example: https://<my-domain>-<domain-owner-id>.d.codeartifact.<region>.amazonaws.com/pypi/<my-repo>/
# Note the lack of the "aws:auth-token@" part
export CODEARTIFACT_REPOSITORY_URL=`aws codeartifact get-repository-endpoint --domain my-domain --domain-owner domain-owner-id --repository my-repo --format pypi --query repositoryEndpoint --output text`

# This will give the token to access the repo
export CODEARTIFACT_AUTH_TOKEN=`aws codeartifact get-authorization-token --domain my-domain --domain-owner domain-owner-id --query authorizationToken --output text`

# This specifies the user who accesses the repo
export CODEARTIFACT_USER=aws

# Now use all of these when configuring the repo in poetry
poetry config repositories.<my-repo-name-for-poetry> $CODEARTIFACT_REPOSITORY_URL
poetry config http-basic.<my-repo-name-for-poetry> $CODEARTIFACT_USER $CODEARTIFACT_AUTH_TOKEN

Note that the authentication token will expire when your AWS login session ends. Hence, you'll have to set the http-basic.<my-repo-name-for-poetry> with the new token every time it expires.

FYI, I had the same problem and it took me hours to figure this out. But in the end, more carefully reading the documentation should have helped me.

Internist

Answer 2

For anyone trying to get Poetry running on Windows Powershell you can use environment variables to get things working. Since windows commands cannot handle the long AWS authentication tokens, it only works if you use environment variables.

Hopefully these steps and command examples help:

Private Repository Configuration (Windows)

Pushing Packages to "myprivaterepo"

1. Log in to AWS or configure AWS CLI

    $Env:AWS_ACCESS_KEY_ID={{ awsaccesskeyidhere }}
    $Env:AWS_SECRET_ACCESS_KEY={{ secretaccesskeyhere }}
    $Env:AWS_SESSION_TOKEN={{ abcdefghijklmnopqrstuvwxyz }}

2. Get a CodeArtifact Authentication Token

    $Env:CODEARTIFACT_AUTH_TOKEN=aws --region us-east-1 codeartifact get-authorization-token --domain {{ repo_domain }} --domain-owner {{ aws_account }} --query authorizationToken --output text

You can check if this worked by running echo $Env:CODEARTIFACT_AUTH_TOKEN. If it worked, you'll see a long string of text. Also make sure the AWS region is the one you're actually using.

3. Set the Poetry Environment Variables

This part was not quite as obvious for me. The key here is to use the UPPERCASE name of your repository in the environment variable name, and it should be the same name in the next step as well. More information in the Poetry documentation for using environment variables.

    $Env:POETRY_HTTP_BASIC_MYPRIVATEREPO_USERNAME=echo aws
    $Env:POETRY_HTTP_BASIC_MYPRIVATEREPO_PASSWORD=$Env:CODEARTIFACT_AUTH_TOKEN

As another example, if your repo was named "secretprojectrepo", you would use: $Env:POETRY_HTTP_BASIC_SECRETPROJECTREPO_USERNAME=echo aws

4. Configure Poetry

The repository name here should be lowercase. Notice the URL does not include the "/simple" on the end. If you copy it from the AWS Console you'll need to remove it since Poetry isn't using that protocol for uploading.

    poetry config repositories.myprivaterepo https://myprivaterepo-1234567890.d.codeartifact.us-east-1.amazonaws.com/pypi/myprivaterepo

Note: You can get the URL from the "manual setup" instructions when you View the Connection Instructions in the AWS Console. Should be the format https://aws:$[e-mail]/pypi/myprivaterepo/simple/. Just remove the credentials part, "aws:$CODEARTIFACT_AUTH_TOKEN@" from the URL since we're using environment variables for providing the authentication information to Poetry.

5. Build and Publish

    poetry build
    poetry publish -r myprivaterepo

Installing Dependencies from a Private Repository

To set up pulling from the repository, follow steps above for logging in to AWS and configuring Poetry. Next you'll need a section in your pyproject.toml that indicates to Poetry which repository to use as the source. Additional information: Poetry Documentation

6. Configure Poetry's Source Repository

    [[tool.poetry.source]]
    name = "myprivaterepo"
    url = "https://myprivaterepo-1234567890.d.codeartifact.us-east-1.amazonaws.com/pypi/myprivaterepo/simple/"
    default = true

Two important notes:

1. This is the same repository URL from the previous steps, but for pulling packages, you'll need the "/simple/" on the end in the URL.
2. The default part is optional. When default is true Poetry will not try looking at PyPi for packages. When left off, it will also search PyPi. You also have the option to set your private repository as a secondary source.

Poetry Documentation: Disabling PyPi

Buzz

Answer 3

This Python script can configure pip, poetry, and twine to work with your artifactory. Run e.g.

python ./codeartifact_login.py configure --tool poetry
poetry publish --repository your-artifactory

#!/usr/bin/env python
import os

import boto3
import fire


class CodeArtifact(object):

    @staticmethod
    def run(cmd: str):
        print(os.popen(cmd=cmd).read())

    @staticmethod
    def configure(tool: str = "twine"):
        if tool not in ["twine", "pip", "poetry"]:
            raise RuntimeError(f"Not recognise tool: {tool}")
        domain = "your-artifactory"
        repo = "your-repo"
        account_id = boto3.client('sts').get_caller_identity().get('Account')
        region = boto3.session.Session().region_name
        if tool == "poetry":
            CODEARTIFACT_REPOSITORY_URL = f"https://{domain}-{account_id}.d.codeartifact.{region}.amazonaws.com/pypi/{repo}/"
            CodeArtifact.run(cmd=f"poetry config repositories.{domain} {CODEARTIFACT_REPOSITORY_URL}")
            CODEARTIFACT_AUTH_TOKEN = boto3.client('codeartifact').get_authorization_token(
                domain=domain,
                domainOwner=account_id,
                durationSeconds=12 * 3600
            )["authorizationToken"]
            CodeArtifact.run(cmd=f"poetry config http-basic.{domain} aws {CODEARTIFACT_AUTH_TOKEN}")
        else:
            CodeArtifact.run(
                cmd=f"aws codeartifact login --tool {tool} --domain {domain} --domain-owner {account_id} --repository {repo}")


if __name__ == '__main__':
    fire.Fire(CodeArtifact)  # configure(tool="poetry")

References:

• https://github.com/google/python-fire
• https://docs.aws.amazon.com/codeartifact/latest/ug/python-configure-twine.html

Abundance

Instead of building the URL manually, it is also possible to use the get_repository_endpoint endpoint method. But you still have to add the username (aws) and login token to the URI: boto3.amazonaws.com/v1/documentation/api/latest/reference/… – Verb

Answer 4

EDIT: See the accepted answer, it works!

If someone gets here from a Google search, here's the situation according to when I'm writing this (19 Dec 2020):

No built-support for this in poetry. You can install from AWS CodeArtifact using it, but not upload unless you're OK with putting secrets in your pyproject.toml file (the renewing URL with the token). My workaround is to upload using twine (just follow AWS's guide for that) and install using poetry (need to add a poetry.toml file AND add CodeArtifact as a source in pyproject.toml).

Relevant GitHub issue..

Pappas

Thank you for the useful additional information to the accepted answer. – Newish