Credentials in pip.conf for private PyPI

Question

I have a private PyPI repository. Is there any way to store credentials in pip.conf similar to .pypirc?

What I mean. Currently in .pypirc you can have such configuration:

[distutils]
index-servers = custom

[custom]
repository: https://pypi.example.com
username: johndoe
password: changeme

From what I've found that you can put in pip.conf:

[global]
index = https://username:[email protected]/pypi
index-url = https://username:[email protected]/simple
cert = /etc/ssl/certs/ca-certificates.crt

But here I see two problems:

1. For each url you'll need each time to specify the same username and password.
2. Username and password become visible in the logs, cause they are part of the url.

Is there any way to store username and password outside of url?

Issuant

I wonder if pip can work with ssh connection and use ssh keys for authentication. SSL certificates could be used to authenticate users as well, but of course for this you would need to change how the http server handles requests. – Brainpan

for 2. you could suppress stdout logging pip install -q package-name and if you still wanted the full verbose logs somewhere you could add the --log option as well to point to a file – Valetudinary

Answer 1

Ideally, you should configure Pip's keyring support. Some backends store credentials in an encrypted/protected form. Other parts of the packaging ecosystem, like Twine, also support keyring.

Alternatively, you could store credentials for Pip to use in ~/.netrc like this:

machine pypi.example.com
    login johndoe
    password changeme

Pip will use these credentials when accessing https://pypi.example.com but won't log them. You must specify the index server separately (such as in pip.conf as in the question).

Note that ~/.netrc must be owned by the user pip executes as. It must not be readable by any other user, either. An invalid file is silently ignored. You can ensure the permissions are correct like this:

chown $USER ~/.netrc
chmod 0600 ~/.netrc

This permissions check doesn't apply before Python 3.4, but it's a good idea in any case.

Internally Pip uses requests when making HTTP requests. requests uses the standard library netrc module to read the file, so the character set is limited to an ASCII subset.

Bertelli

Thanks! That works. Strange, that pip itself doesn't have such functionality if it relies on requests. – Issuant

This is great! One important distinction between using .netrc and pip.conf is that in .netrc a password with special characters MUST NOT be URL encoded, whereas in pip.conf it is required. – Matadi

As an added comment, for those who wonder, this solution works on both linux and windows. – Colwin

Hi @Bertelli I try your solution but pip keep ignoring the configured password. I asked a new question here link. – Fairman

Unfortunately this is not ideal if your Pypi repository is not hosted on its own domain. For example Gitlab’s Pypi repos use the domain gitlab.com and so you can have different repositories with different credentials on that same domain. – Sattler

This causes conflicts with AzureDevops git lfs authentication – Lauree

pip.pypa.io/en/stable/topics/authentication This document explains the different authentication methods. – Entremets

Answer 2

How about storing the Username/Password as environment variables,

export username=username
export password=password

and referring to them in the pip.conf like so:

[global]
index = https://$username:[email protected]/pypi
index-url = https://$username:[email protected]/simple
cert = /etc/ssl/certs/ca-certificates.crt

I use Gitlab CI's secret variables for storing credentials. Check for an equivalent in your CI tool.

Bascom

For the 2nd part @MarkAWard's suggestion seems to do the trick. – Bascom

Thanks for your reply! Actually I was initially doing as you've suggested. I also use GitLab CI and I store username and password as the environment variables. The problem was, that pip install displays username and password in the output logs. But here suggestion from @Valetudinary helped. Currently only one question left, do you know if there is a way to forbid echo $username and echo $password output in GtiLab CI? – Issuant

@KetanVatsalya Inside the pip.conf file, I had to wrap the environment variables inside curly braces else the environment variables won't be resolved. so index = https://$username:[email protected]/pypi should be index = https://${username}:${password}@pypi.example.com/pypi – Rushing

@Rushing what version of pip are you using. I cant get the environment variables to resolve within pip.conf using both the curly braces and without curly braces option – Monstrosity

Was this resolved ? Even I am not able to get the variables resolved inside pip.conf . I am using version 8.0.2 – Neruda

wasn't able to get it working either. would be nice to have – Undercoat

As far as I can tell pip.conf does't expand environment variables, so you'll have to add an extra step in your CI script using e.g. envsubst gnu.org/software/gettext/manual/html_node/… – Oneida

"index-url = https://<username>:<password>@pypi.example.com/simple" do the trick – Habited

Answer 3

Given issue 4789 is still open, you can utilize the following approach in your requirements.txt:

--extra-index-url=https://${PYPI_USERNAME}:${PYPI_PASSWORD}@my.privatepypi.com
private-package==1.2.3
...

If you try to run pip install -r requirements.txt with those environment variables set, you will find that pip still asks for credentials. This is because pip does not interpolate the expression ${PYPI_USERNAME} as one would expect, but rather url encodes it. Your username and password in this example is expressed as https://%24%7BPYPI_USERNAME%7D:%24%7BPYPI_PASSWORD%[email protected]

The work around here, and I admit there should be a better approach, but we can run pip as a script:

python3 -m pip install -r requirements.txt

From man:

 -m module-name
    Searches sys.path for the named module and runs the corresponding .py file as a script.

Moreira

A couple of things I don't understand here - 1) your two commands differ in whether they use the -r flag, is that just an omission on the first one, or is it important? 2) python3 -m pip should be exactly equivalent to just pip, assuming the correct Python is invoked. I'm not sure why it would have the effect you describe, do you understand what's going on? – Eskimoaleut

Why is the behavior different when using a module versus "pip" script?? – Maynord

Answer 4

I know it is not answering the question the way it was formulated, but one could also use keyring python module for pip authentication:

https://pypi.org/project/keyring/

https://pip.pypa.io/en/stable/topics/authentication/#keyring-support%3E

Conformist

I use tox, and because tox.ini does not support multi-line install commands, keyring was too much of a pain to use. PIP_EXTRA_INDEX_URL was the way to go. – Caliban

Answer 5

Hey you can simply run pip install and pass url to private repo as argument like so (I'm using Artifactory but it doesn't matter):

Artifactory example:

pip install {lib_name} -U -i https://{user_name}:{token_to_prvate_repo}@artifactory.{company_name}.com/api/pypi/pypi/simple

Github example:

pip install git+https://${GITHUB_USER}:${GITHUB_TOKEN}@github.com/user/project.git@{version}

Bitbucket example:

pip install  git+https://${BITBUCKET_USER}:${BITBUCKET_APP_PASSWORD}@bitbucket.org/user/project.git@{version}'

Variables can be read from pipeline/keay vault or other secure solutions.

Nutritious