In my case, we wanted to get pip modules installed from requirement*.txt files, which had locked-in module's versions defined in the file and fetched from an internal Artifactory server (rather than going to online i.e. pypi.org)
Ex: requirements.txt file(s)
numpy==1.16.2
pandas==1.0.3
..
...
To fix the issue: I had to use NO_PROXY=<value>
available as an environment variable.
Let's say, if you artifactory server is: my-artifactory.company.local or my-artifactory.company.com, then all we need to ensure is that NO_PROXY
variable has that hostname's "domain" part listed in its value.
i.e. for
my-artifactory.company.com or my-artifactory.company.local, value inside
NO_PROXY variable must have: ,.company.com,.company.local,...
in it.
Sample exported variable (at command line $ prompt):
export NO_PROXY=localhost,127.0.0.1,169.254.169.254,169.254.169.123,.somecompany.com,.company.com,.company.local,pki.company.com,s3-us-gov-west-1.amazonaws.com,s3-fips-us-gov-west-1.amazonaws.com,rds.amazonaws.com,10.201.12.244,10.201.44.62,10.201.32.261
====
If you are using a Dockerfile
, then, ensure you have ARG/ENV variable are set correctly.
ARG is used during build time (can be overridden at command line using --build-arg option sent to docker build -t tag .
where it'll search current directory for a Dockerfile and create an image. ENV is used at run time (docker run
) and can be overridden as well.
Sample Dockerfile is:
FROM python:3.7
MAINTAINER [email protected]
ARG PYTHONBUFFERED=0
ARG HTTPS_PROXY=http://proxy.ext.company.com:80
ARG HTTP_PROXY=http://proxy.ext.company.com:80
ARG NO_PROXY=localhost,127.0.0.1,169.254.169.254,.company.com,.company.local,pki.company.com,s3-us-gov-west-1.amazonaws.com,s3-fips-us-gov-west-1.amazonaws.com,rds.amazonaws.com
ENV PYTHONBUFFERED=${PYTHONBUFFERED}
ENV HTTPS_PROXY=${HTTPS_PROXY}
ENV HTTP_PROXY=${HTTP_PROXY}
ENV NO_PROXY=${NO_PROXY}
# If there are 3 requirements files in source control, I'm copy all for pip install, you don't have to. Use what modules you want / file you want.
RUN mkdir -p code
COPY requirements.txt /code
COPY requirements-test.txt /code
COPY requirements-dev.txt /code
WORKDIR /code
# You can fetch from pypi.org but in my case, this was a security issue.
# RUN pip install --trusted-host pypi.org -r requirements.txt
RUN pip install --no-cache-dir --trusted-host my-artifactory.company.local -r requirements.txt -r requirements-test.txt -r requirements-dev.txt --index-url http://my-artifactory.company.local:8081/artifactory/api/pypi/pypi-local-deps/simple --disable-pip-version-check
The main line, which solved the issue in my case, was using the NO_PROXY (as listed above).
Any issues related to pip module not found, or module version not found, or any SSL errors SSLError(SSLCertVerificationError
like errors, went away after applying the above NO_PROXY at cmd line or in Dockerfile:
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1091)'))': /simple/requests/
or
ERROR: Could not find a version that satisfies the requirement requests
ERROR: No matching distribution found for requests
or
ERROR: Could not find a version that satisfies the requirement numpy==1.16.2
ERROR: No matching distribution found for numpy==1.16.2
import numpy; print(numpy.__version__)
? – Crucialnumpy
and it doesn't seem to be available in pip anymore. Is there a reason for that specific requirement? You could search for an archive of the old version, but you might have better luck finding a newer version ofquandl
that works with current versions ofnumpy
– Lammastidequandl
. A well written package shouldn't depend on a subversion like that. It's ok to expect something newer than 1.9.3 but not exactly that. That's too fragile. – Fluctuationpip
/pip3
issue? – Crucial