Proxies with Python 'Requests' module

Question

Just a short, simple one about the excellent Requests module for Python.

I can't seem to find in the documentation what the variable 'proxies' should contain. When I send it a dict with a standard "IP:PORT" value it rejected it asking for 2 values. So, I guess (because this doesn't seem to be covered in the docs) that the first value is the ip and the second the port?

The docs mention this only:

proxies – (optional) Dictionary mapping protocol to the URL of the proxy.

So I tried this... what should I be doing?

proxy = { ip: port}

and should I convert these to some type before putting them in the dict?

r = requests.get(url,headers=headers,proxies=proxy)

Carbonate

Answer 1

The proxies' dict syntax is {"protocol": "scheme://ip:port", ...}. With it you can specify different (or the same) proxie(s) for requests using http, https, and ftp protocols:

http_proxy  = "http://10.10.1.10:3128"
https_proxy = "https://10.10.1.11:1080"
ftp_proxy   = "ftp://10.10.1.10:3128"

proxies = { 
              "http"  : http_proxy, 
              "https" : https_proxy, 
              "ftp"   : ftp_proxy
            }

r = requests.get(url, headers=headers, proxies=proxies)

Deduced from the requests documentation:

Parameters:
method – method for the new Request object.
url – URL for the new Request object.
...
proxies – (optional) Dictionary mapping protocol to the URL of the proxy.
...

---

On linux you can also do this via the HTTP_PROXY, HTTPS_PROXY, and FTP_PROXY environment variables:

export HTTP_PROXY=10.10.1.10:3128
export HTTPS_PROXY=10.10.1.11:1080
export FTP_PROXY=10.10.1.10:3128

On Windows:

set http_proxy=10.10.1.10:3128
set https_proxy=10.10.1.11:1080
set ftp_proxy=10.10.1.10:3128

Tuesday

@cigar I knew because urllib2 uses the exact same format for their proxies dict, and when I saw docs.python-requests.org/en/latest/api/#module-requests say "proxies – (optional) Dictionary mapping protocol to the URL of the proxy.", I knew right away. – Tuesday

ahhh i see, never used proxies with urllib2 because of the advice to get rid of it obtained from here, replaced 2 pages of code with 8 lines :/ re:shoulder :))) great stay here, you have already saved me hours in total! if you ever need any help with music gimme a shout, that i can give advice on, otherwise cant think of way to repay other than massive thanks or cups of tea! – Carbonate

It seems requests and moreover urllib3 can't do a CONNECT when using a proxy :( – Nailbiting

@Nailbiting I have not yet used urllib3 so I'll have to look into that. Thanks for the heads up. – Tuesday

request is a wrapper on urllib3 which is bundle into this module. github.com/kennethreitz/requests/tree/develop/requests/packages/… – Nailbiting

@Tuesday the syntax changed with requests 2.0.0. You'll need to add a schema to the url: docs.python-requests.org/en/latest/user/advanced/#proxies It'd nice if you could add this to your answer here – Seamanship

@Jay: I added the URL schema. – Forevermore

This will not work for socks5 proxy: 'http' : "socks5://myproxy:9191", – Ladon

Those are bad examples of the linux environment variables HTTP_PROXY and HTTPS_PROXY. The protocol should always be included (not just host:port), and either proxy can itself be "https" or "http". HTTPS_PROXY=myhttpsproxy:8080 is valid, it just means proxy "https" requests using myhttpsproxy:8080 instead of the value of HTTP_PROXY. If you don't define HTTPS_PROXY linux apps typically use CONNECT over the HTTP_PROXY. – Kellum

What if you want multiple proxies per protocol. Currently you just have one for each. – Dorathydorca

@Seamanship link is dead. New link is 2.python-requests.org/en/master/user/advanced/#proxies – Altman

Not only protocol, but host is possible as well: proxies = {'http://10.20.1.128': 'http://10.10.1.10:5323'} – Hadron

@Tuesday I tried out the exact same code with my IP and port number but it still blocks the website that I use to scrape data from(craiglist.com). Any idea about this? – Lontson

@Tuesday I replaced the 'http_proxy ' with my IP address and the port as 8888 because that's where my localhost was running. Should the port values like '3128,1080' be the same for all devices? What about the IP addresses then? if the website gets hits from the same IPs every time, it will surely block! – Lontson

this doesn't work in windows 10 from command line, I continue to get errors when python tries to connect to a youtube source – Begird

Answer 2

You can refer to the proxy documentation here.

If you need to use a proxy, you can configure individual requests with the proxies argument to any request method:

import requests

proxies = {
  "http": "http://10.10.1.10:3128",
  "https": "https://10.10.1.10:1080",
}

requests.get("http://example.org", proxies=proxies)

To use HTTP Basic Auth with your proxy, use the http://user:[email protected]/ syntax:

proxies = {
    "http": "http://user:[email protected]:3128/"
}

Statecraft

Answer 3

I have found that urllib has some really good code to pick up the system's proxy settings and they happen to be in the correct form to use directly. You can use this like:

import urllib

...
r = requests.get('http://example.org', proxies=urllib.request.getproxies())

It works really well and urllib knows about getting Mac OS X and Windows settings as well.

Everyway

Does it work without a proxy? Some of our users has no proxy and some has. – Underwent

@jonasl Yes, it does work even when there's no system proxy defined. In that case, it's just an empty dict. – Selfcontained

Does it include no_proxy and does requests respect no_proxy? Nevermind, it seems there are solutions: github.com/kennethreitz/requests/issues/879 – Elmiraelmo

getting err: module 'urllib' has no attribute 'getproxies' – Baalman

Greenish: urllib.request.getproxies() – Prescott

@Baalman try urllib2.getproxies() – Chesna

@Zahra: use import urllib.request and afterwards urllib.request.getproxies(). Source: #37042652 – Jennefer

Answer 4

The accepted answer was a good start for me, but I kept getting the following error:

AssertionError: Not supported proxy scheme None

Fix to this was to specify the http:// in the proxy url thus:

http_proxy  = "http://194.62.145.248:8080"
https_proxy  = "https://194.62.145.248:8080"
ftp_proxy   = "10.10.1.10:3128"

proxyDict = {
              "http"  : http_proxy,
              "https" : https_proxy,
              "ftp"   : ftp_proxy
            }

I'd be interested as to why the original works for some people but not me.

Edit: I see the main answer is now updated to reflect this :)

Dunstan

changed with 2.0.0: Proxy URLs now must have an explicit scheme. A MissingSchema exception will be raised if they don't. – Seamanship

Answer 5

If you'd like to persisist cookies and session data, you'd best do it like this:

import requests

proxies = {
    'http': 'http://user:[email protected]:3128',
    'https': 'https://user:[email protected]:3128',
}

# Create the session and set the proxies.
s = requests.Session()
s.proxies = proxies

# Make the HTTP request through the session.
r = s.get('http://www.showmemyip.com/')

Hadley

Do we have to send the "Proxy-Connection: Keep-alive" header manually in the python requests ? – Frankly

Answer 6

8 years late. But I like:

import os
import requests

os.environ['HTTP_PROXY'] = os.environ['http_proxy'] = 'http://http-connect-proxy:3128/'
os.environ['HTTPS_PROXY'] = os.environ['https_proxy'] = 'http://http-connect-proxy:3128/'
os.environ['NO_PROXY'] = os.environ['no_proxy'] = '127.0.0.1,localhost,.local'

r = requests.get('https://example.com')  # , verify=False

Hays

I like this last resort solution that no one else mentioned here. It just saved my day as there was no other way of passing proxy settings to a 3rd party library I'm using. – Communicate

Answer 7

The documentation gives a very clear example of the proxies usage

import requests

proxies = {
  'http': 'http://10.10.1.10:3128',
  'https': 'http://10.10.1.10:1080',
}

requests.get('http://example.org', proxies=proxies)

What isn't documented, however, is the fact that you can even configure proxies for individual urls even if the schema is the same! This comes in handy when you want to use different proxies for different websites you wish to scrape.

proxies = {
  'http://example.org': 'http://10.10.1.10:3128',
  'http://something.test': 'http://10.10.1.10:1080',
}

requests.get('http://something.test/some/url', proxies=proxies)

Additionally, requests.get essentially uses the requests.Session under the hood, so if you need more control, use it directly

import requests

proxies = {
  'http': 'http://10.10.1.10:3128',
  'https': 'http://10.10.1.10:1080',
}
session = requests.Session()
session.proxies.update(proxies)

session.get('http://example.org')

I use it to set a fallback (a default proxy) that handles all traffic that doesn't match the schemas/urls specified in the dictionary

import requests

proxies = {
  'http': 'http://10.10.1.10:3128',
  'https': 'http://10.10.1.10:1080',
}
session = requests.Session()
session.proxies.setdefault('http', 'http://127.0.0.1:9009')
session.proxies.update(proxies)

session.get('http://example.org')

Renault

Answer 8

i just made a proxy graber and also can connect with same grabed proxy without any input here is :

#Import Modules

from termcolor import colored
from selenium import webdriver
import requests
import os
import sys
import time

#Proxy Grab

options = webdriver.ChromeOptions()
options.add_argument('headless')
driver = webdriver.Chrome(chrome_options=options)
driver.get("https://www.sslproxies.org/")
tbody = driver.find_element_by_tag_name("tbody")
cell = tbody.find_elements_by_tag_name("tr")
for column in cell:

        column = column.text.split(" ")
        print(colored(column[0]+":"+column[1],'yellow'))
driver.quit()
print("")

os.system('clear')
os.system('cls')

#Proxy Connection

print(colored('Getting Proxies from graber...','green'))
time.sleep(2)
os.system('clear')
os.system('cls')
proxy = {"http": "http://"+ column[0]+":"+column[1]}
url = 'https://mobile.facebook.com/login'
r = requests.get(url,  proxies=proxy)
print("")
print(colored('Connecting using proxy' ,'green'))
print("")
sts = r.status_code

Potman

Answer 9

here is my basic class in python for the requests module with some proxy configs and stopwatch !

import requests
import time
class BaseCheck():
    def __init__(self, url):
        self.http_proxy  = "http://user:pw@proxy:8080"
        self.https_proxy = "http://user:pw@proxy:8080"
        self.ftp_proxy   = "http://user:pw@proxy:8080"
        self.proxyDict = {
                      "http"  : self.http_proxy,
                      "https" : self.https_proxy,
                      "ftp"   : self.ftp_proxy
                    }
        self.url = url
        def makearr(tsteps):
            global stemps
            global steps
            stemps = {}
            for step in tsteps:
                stemps[step] = { 'start': 0, 'end': 0 }
            steps = tsteps
        makearr(['init','check'])
        def starttime(typ = ""):
            for stemp in stemps:
                if typ == "":
                    stemps[stemp]['start'] = time.time()
                else:
                    stemps[stemp][typ] = time.time()
        starttime()
    def __str__(self):
        return str(self.url)
    def getrequests(self):
        g=requests.get(self.url,proxies=self.proxyDict)
        print g.status_code
        print g.content
        print self.url
        stemps['init']['end'] = time.time()
        #print stemps['init']['end'] - stemps['init']['start']
        x= stemps['init']['end'] - stemps['init']['start']
        print x


test=BaseCheck(url='http://google.com')
test.getrequests()

Reviviscence

Answer 10

Already tested, the following code works. Need to use HTTPProxyAuth.

import requests
from requests.auth import HTTPProxyAuth


USE_PROXY = True
proxy_user = "aaa"
proxy_password = "bbb"
http_proxy = "http://your_proxy_server:8080"
https_proxy = "http://your_proxy_server:8080"
proxies = {
    "http": http_proxy,
    "https": https_proxy
}

def test(name):
    print(f'Hi, {name}')  # Press Ctrl+F8 to toggle the breakpoint.
    # Create the session and set the proxies.
    session = requests.Session()
    if USE_PROXY:
        session.trust_env = False
        session.proxies = proxies
        session.auth = HTTPProxyAuth(proxy_user, proxy_password)

    r = session.get('https://www.stackoverflow.com')
    print(r.status_code)

if __name__ == '__main__':
    test('aaa')

Niven

Answer 11

It’s a bit late but here is a wrapper class that simplifies scraping proxies and then making an http POST or GET:

ProxyRequests

https://github.com/rootVIII/proxy_requests

Nihility

Answer 12

I share some code how to fetch proxies from the site "https://free-proxy-list.net" and store data to a file compatible with tools like "Elite Proxy Switcher"(format IP:PORT):

##PROXY_UPDATER - get free proxies from https://free-proxy-list.net/

from lxml.html import fromstring
import requests
from itertools import cycle
import traceback
import re

######################FIND PROXIES#########################################
def get_proxies():
    url = 'https://free-proxy-list.net/'
    response = requests.get(url)
    parser = fromstring(response.text)
    proxies = set()
    for i in parser.xpath('//tbody/tr')[:299]:   #299 proxies max
        proxy = ":".join([i.xpath('.//td[1]/text()') 
        [0],i.xpath('.//td[2]/text()')[0]])
        proxies.add(proxy)
    return proxies



######################write to file in format   IP:PORT######################
try:
    proxies = get_proxies()
    f=open('proxy_list.txt','w')
    for proxy in proxies:
        f.write(proxy+'\n')
    f.close()
    print ("DONE")
except:
    print ("MAJOR ERROR")

Glossy

Do they allow unlimited scraping? – Surculose

This has nothing to do with OP's question – Arrowworm

This is mint and totally 100% related to why I landed on this page. Thank you for sharing! – Aftertaste