pip, proxy authentication and "Not supported proxy scheme"

Question

Trying to install pip on a new python installation. I am stuck with proxy errors. Looks like a bug in get-pip or urllib3??

Question is do I have to go through the pain of setting up CNTLM as described here or is there a shortcut?

get-pip.py documentation says use --proxy="[user:passwd@]proxy.server:port" option to specify proxy and relevant authentication. But seems like pip passes on the whole thing as it is to urllib3 which interprets "myusr" as the url scheme, because of the ':' I guess (?).

C:\ProgFiles\Python27>get-pip.py --proxy myusr:[email protected]:80
Downloading/unpacking pip
Cleaning up...
Exception:
Traceback (most recent call last):
  File "c:\users\sg0219~1\appdata\local\temp\tmpxwg_en\pip.zip\pip\basecommand.py", line 122, in main
    status = self.run(options, args)
  File "c:\users\sg0219~1\appdata\local\temp\tmpxwg_en\pip.zip\pip\commands\install.py", line 278, in run
    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
  File "c:\users\sg0219~1\appdata\local\temp\tmpxwg_en\pip.zip\pip\req.py", line 1177, in prepare_files
    url = finder.find_requirement(req_to_install, upgrade=self.upgrade)
  File "c:\users\sg0219~1\appdata\local\temp\tmpxwg_en\pip.zip\pip\index.py", line 194, in find_requirement
    page = self._get_page(main_index_url, req)
  File "c:\users\sg0219~1\appdata\local\temp\tmpxwg_en\pip.zip\pip\index.py", line 568, in _get_page
    session=self.session,
  File "c:\users\sg0219~1\appdata\local\temp\tmpxwg_en\pip.zip\pip\index.py", line 670, in get_page
    resp = session.get(url, headers={"Accept": "text/html"})
  File "c:\users\sg0219~1\appdata\local\temp\tmpxwg_en\pip.zip\pip\_vendor\requests\sessions.py", line 468, in get
    return self.request('GET', url, **kwargs)
  File "c:\users\sg0219~1\appdata\local\temp\tmpxwg_en\pip.zip\pip\download.py", line 237, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "c:\users\sg0219~1\appdata\local\temp\tmpxwg_en\pip.zip\pip\_vendor\requests\sessions.py", line 456, in request
    resp = self.send(prep, **send_kwargs)
  File "c:\users\sg0219~1\appdata\local\temp\tmpxwg_en\pip.zip\pip\_vendor\requests\sessions.py", line 559, in send
    r = adapter.send(request, **kwargs)
  File "c:\users\sg0219~1\appdata\local\temp\tmpxwg_en\pip.zip\pip\_vendor\requests\adapters.py", line 305, in send
    conn = self.get_connection(request.url, proxies)
  File "c:\users\sg0219~1\appdata\local\temp\tmpxwg_en\pip.zip\pip\_vendor\requests\adapters.py", line 215, in get_connection
    block=self._pool_block)
  File "c:\users\sg0219~1\appdata\local\temp\tmpxwg_en\pip.zip\pip\_vendor\requests\packages\urllib3\poolmanager.py", line 258, in proxy_fro
m_url
    return ProxyManager(proxy_url=url, **kw)
  File "c:\users\sg0219~1\appdata\local\temp\tmpxwg_en\pip.zip\pip\_vendor\requests\packages\urllib3\poolmanager.py", line 214, in __init__
    'Not supported proxy scheme %s' % self.proxy.scheme
AssertionError: Not supported proxy scheme myusr

Storing debug log for failure in C:\Users\myusr\pip\pip.log

C:\ProgFiles\Python27>

When I run the command without the usrname and password it works fine, but proxy rejects the request saying it needs authentication ("407 authenticationrequired").

C:\ProgFiles\Python27>get-pip.py --proxy 111.222.333.444:80
Downloading/unpacking pip
  Cannot fetch index base URL https://pypi.python.org/simple/
  Could not find any downloads that satisfy the requirement pip
Cleaning up...
No distributions at all found for pip
Storing debug log for failure in C:\Users\sg0219898\pip\pip.log

C:\ProgFiles\Python27>cat C:\Users\sg0219898\pip\pip.log
------------------------------------------------------------
C:\ProgFiles\Python27\get-pip.py run on 09/29/14 16:23:26
Downloading/unpacking pip
  Getting page https://pypi.python.org/simple/pip/
  Could not fetch URL https://pypi.python.org/simple/pip/: connection error: ('Cannot connect to proxy.', error('Tunnel connection failed: 407 authenticationrequired',))
  Will skip URL https://pypi.python.org/simple/pip/ when looking for download links for pip
  Getting page https://pypi.python.org/simple/
  Could not fetch URL https://pypi.python.org/simple/: connection error: ('Cannot connect to proxy.', error('Tunnel connection failed: 407 authenticationrequired',))
  Will skip URL https://pypi.python.org/simple/ when looking for download links for pip
  Cannot fetch index base URL https://pypi.python.org/simple/
  URLs to search for versions for pip:
  * https://pypi.python.org/simple/pip/
  Getting page https://pypi.python.org/simple/pip/
  Could not fetch URL https://pypi.python.org/simple/pip/: connection error: ('Cannot connect to proxy.', error('Tunnel connection failed: 407 authenticationrequired',))
  Will skip URL https://pypi.python.org/simple/pip/ when looking for download links for pip
  Could not find any downloads that satisfy the requirement pip
Cleaning up...
  Removing temporary dir c:\users\sg0219~1\appdata\local\temp\pip_build_SG0219898...
No distributions at all found for pip
Exception information:
Traceback (most recent call last):
  File "c:\users\sg0219~1\appdata\local\temp\tmp36ynxd\pip.zip\pip\basecommand.py", line 122, in main
    status = self.run(options, args)
  File "c:\users\sg0219~1\appdata\local\temp\tmp36ynxd\pip.zip\pip\commands\install.py", line 278, in run
    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
  File "c:\users\sg0219~1\appdata\local\temp\tmp36ynxd\pip.zip\pip\req.py", line 1177, in prepare_files
    url = finder.find_requirement(req_to_install, upgrade=self.upgrade)
  File "c:\users\sg0219~1\appdata\local\temp\tmp36ynxd\pip.zip\pip\index.py", line 277, in find_requirement
    raise DistributionNotFound('No distributions at all found for %s' % req)
DistributionNotFound: No distributions at all found for pip

C:\ProgFiles\Python27>

I had a brief look at urllib3\poolmanager.py and it doesn't seem to have anything to do with username/passwords.

Bracteole

Looks like it could be related to github.com/kennethreitz/requests/issues/2247 – Conclusion

It isn't. Please see the answer below – Disk

Answer 1

This is complaining about the scheme for the URL (which urlparse is understanding to be myusr), to work around that you should instead be doing:

get-pip.py --proxy http://myusr:[email protected]:80

Disk

I resolved it by removing my https_proxy, like unset https_proxy. My http_proxy is still there. – Tonality

Answer 2

This is because the script requires the environment variables http_proxy or https_proxy contains the scheme in the URL. Set the environment variables

 export http_proxy="http://<hostname>:<port>"
 export https_proxy="https://<hostname>:<port>"

before run the "python get-pip.py"

Merchant

This did not work for me, but it worked with the --proxy option. – Pennyworth

This worked for me. I was getting the same error as the OP, and I was using proxy environment variables that did not contain the http / https. Using the --proxy option did not work for me, but this may have been due to my environment variables still existing. – Kopeck

Answer 3

import requests
proxy = {
'http' : 'http://138.197.222.35:80',
'https' : 'http://1138.197.222.35:8080'
}

r = requests.get('http://httpbin.org/ip', proxies=proxy)
print (r)

Append 'http://' and 'https://' with your ips.
I had this issue when working with linux, windows worked fine with ip alone.

Relic

Answer 4

Use pip install xxx --proxy=https://xx.xx.xx.xx:xxxx

After Python 3.6, when using the proxy, the parameter value of proxies in requests.get(url=url, headers=headers, proxies=...) changed.

Before 3.6 includes, proxies={ 'https': '127.0.0.1:8080'} or proxies={'http': '127.0.0.1:8080'} is fine, but this type of dictionary is not suitable for Python 3.7 and above.

In Python3.7 and above, you must add http:// or https:// in front of ip:port, that is, proxies={'http':'http://127.0.0.1:8080'} or proxies={'https':'https://127.0.0.1:8080'}

Unequaled

Answer 5

For the problem you have mentioned, it depends on how the proxy server authentication has been set up. For example, my intranet uses Windows AD and probably the proxy server is using Windows Integrated auth. Thus when I do pip install --proxy http://<server-ip>:<port> <module-name>, it works fine. Note that I did not have to type in username & password ,likely due to integrated auth.

So, you'll need to find out the authentication being used by your proxy server.You can use Fiddler(or any other network analyzer) tool to check the WWW-Authenticate headers in the 407 response, to check what auth mechs the server supports.

Atworth

As in OP, my server expects user/password. Anyway the problem was what the accepted answer pointed out, noting to do with auth as such. – Bracteole

Answer 6

This can be a case when you use ";" in your passwords. This cannot be parsed properly that results in ProxySchemeUnknown error raised.

Permeance

Answer 7

The below things worked for me :

Step 1) Set HTTPS_PROXY and HTTP_PROXY:

export HTTPS_PROXY="http://proxy-address:8080"

export HTTP_PROXY="http://proxy-address:8080"

step 2) : Run pip with below configuration

./pip.exe install pyspark --trusted-host pypi.python.org --trusted-host files.pythonhosted.org --trusted-host pypi.org --proxy "proxy-address:8080"

Hadron

Answer 8

On Windows you can use the SET command instead of export:

• SET HTTP_PROXY="http://proxy-address:8080"
• SET HTTPS_PROXY="http://proxy-address:8080"

If you are not behind a proxy, you need to unset like this:

• SET HTTP_PROXY=
• SET HTTPS_PROXY=

Mating

Answer 9

it works to install boto3 on win7. pip install boto3 --proxy=https://user:[email protected]:8080

Collecting boto3
  Downloading boto3-1.20.33-py3-none-any.whl (131 kB)
Collecting botocore<1.24.0,>=1.23.33
  Downloading botocore-1.23.33-py3-none-any.whl (8.5 MB)
Collecting s3transfer<0.6.0,>=0.5.0
  Downloading s3transfer-0.5.0-py3-none-any.whl (79 kB)
Collecting jmespath<1.0.0,>=0.7.1
  Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting python-dateutil<3.0.0,>=2.1
  Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting urllib3<1.27,>=1.25.4
  Downloading urllib3-1.26.8-py2.py3-none-any.whl (138 kB)
Collecting six>=1.5
  Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: six, python-dateutil, jmespath, urllib3, botocore, s3transfer, boto3
Successfully installed boto3-1.20.33 botocore-1.23.33 jmespath-0.10.0 python-dateutil-2.8.2 s3transfer-0.5.0 six-1.16.0 urllib3-1.26.8

Stadler

use unset before run pip install command unset HTTPS_PROXY – Stadler

the output of echo $HTTPS_PROXY must be blank. – Stadler

This's same as the accepted answer, minus the reason why. – Bracteole