How to specify an authenticated proxy for a python http connection?

Question

What's the best way to specify a proxy with username and password for an http connection in python?

Ascus

Answer 1

Use this:

import requests

proxies = {"http":"http://username:password@proxy_ip:proxy_port"}

r = requests.get("http://www.example.com/", proxies=proxies)

print(r.content)

I think it's much simpler than using urllib. I don't understand why people love using urllib so much.

Wheelman

This is the best script probably and the best thing about using requests is you don't have to worry about whether it is Python 2 or 3. – Noddle

@Aminah Nuraini Could you please explain what do you mean by username:password here? – Snowblink

@Oshada, replace it with your proxy's username and password. You usually get it from the provider. – Wheelman

People sometimes have to use what libraries exist in servers. Can't install libraries or limited access. – Needful

How do I get these values? – Saleswoman

Answer 2

This works for me:

import urllib2

proxy = urllib2.ProxyHandler({'http': 'http://
username:password@proxyurl:proxyport'})
auth = urllib2.HTTPBasicAuthHandler()
opener = urllib2.build_opener(proxy, auth, urllib2.HTTPHandler)
urllib2.install_opener(opener)

conn = urllib2.urlopen('http://python.org')
return_str = conn.read()

Redundant

urllib2.HTTPHandler is added by default (see urllib2 doc). It seems it is redundant to add it when building opener. – Monjan

And why do you use urllib2.HTTPBasicAuthHandler() if no authentication is involved? – Monjan

Normally proxies are in an IP format. That's the same as proxyurl, right? – Christ

Yes. I guess if you've got an IP Address for your proxy you can put this IP address as proxyurl in there. – Redundant

I used the same methods but a different order to input the info and it did not work, thanks ! proxy = urllib2.ProxyHandler({'http': '172.24.8.69:8080'}) #proxy = urllib2.ProxyHandler({}) proxy_auth_handler = urllib2.HTTPBasicAuthHandler() proxy_auth_handler.add_password(None, 'ixx.groupe', 'poutrathor', 'PoutrathorPassword') opener = urllib2.build_opener(proxy, proxy_auth_handler) opener = urllib2.build_opener(proxy) urllib2.install_opener(opener) response = urllib2.urlopen('google.com') html = response.read() print html – Cromwell

After reading a related link, I have decided to comment out #auth = urllib2.HTTPBasicAuthHandler() and change opener to opener = urllib2.build_opener(proxy, urllib2.HTTPHandler), then it worked for me – Londonderry

Answer 3

Setting an environment var named http_proxy like this: http://username:password@proxy_url:port

Seville

But not really answering the answer. – Backspace

@Backspace Answering the Question, not the Answer. And it's not answering neither. – Gavel

Answer 4

The best way of going through a proxy that requires authentication is using urllib2 to build a custom url opener, then using that to make all the requests you want to go through the proxy. Note in particular, you probably don't want to embed the proxy password in the url or the python source code (unless it's just a quick hack).

import urllib2

def get_proxy_opener(proxyurl, proxyuser, proxypass, proxyscheme="http"):
    password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
    password_mgr.add_password(None, proxyurl, proxyuser, proxypass)

    proxy_handler = urllib2.ProxyHandler({proxyscheme: proxyurl})
    proxy_auth_handler = urllib2.ProxyBasicAuthHandler(password_mgr)

    return urllib2.build_opener(proxy_handler, proxy_auth_handler)

if __name__ == "__main__":
    import sys
    if len(sys.argv) > 4:
        url_opener = get_proxy_opener(*sys.argv[1:4])
        for url in sys.argv[4:]:
            print url_opener.open(url).headers
    else:
        print "Usage:", sys.argv[0], "proxy user pass fetchurls..."

In a more complex program, you can seperate these components out as appropriate (for instance, only using one password manager for the lifetime of the application). The python documentation has more examples on how to do complex things with urllib2 that you might also find useful.

Empson

Answer 5

Or if you want to install it, so that it is always used with urllib2.urlopen (so you don't need to keep a reference to the opener around):

import urllib2
url = 'www.proxyurl.com'
username = 'user'
password = 'pass'
password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
# None, with the "WithDefaultRealm" password manager means
# that the user/pass will be used for any realm (where
# there isn't a more specific match).
password_mgr.add_password(None, url, username, password)
auth_handler = urllib2.HTTPBasicAuthHandler(password_mgr)
opener = urllib2.build_opener(auth_handler)
urllib2.install_opener(opener)
print urllib2.urlopen("http://www.example.com/folder/page.html").read()

Deming

Answer 6

Here is the method use urllib

import urllib.request

# set up authentication info
authinfo = urllib.request.HTTPBasicAuthHandler()
proxy_support = urllib.request.ProxyHandler({"http" : "http://ahad-haam:3128"})

# build a new opener that adds authentication and caching FTP handlers
opener = urllib.request.build_opener(proxy_support, authinfo,
                                     urllib.request.CacheFTPHandler)

# install it
urllib.request.install_opener(opener)

f = urllib.request.urlopen('http://www.python.org/')
"""

Wellspoken

And where is the proxy authentication in your example? Your ProxyHandler doesn't contain proxy auth details. – Episodic

please refer to this link epydoc.sourceforge.net/stdlib/urllib2-module.html – Wellspoken