Python Selenium Webdriver - Changing proxy settings on the fly

Question

I'm currently successfully using the code below to use a proxy with the Selenium webdriver. Unfortunately, I can't seem to make it change the proxy settings without restarting the whole browser. I had hoped that simply updating the proxy settings, just like I did to set the proxy to start with, would change the proxy, but it doesn't seem to work. Any help on this subject would be greatly appreciated.

profile = webdriver.FirefoxProfile()
profile.set_preference("network.proxy.type", 1)
profile.set_preference("network.proxy.http", proxyAddress)
profile.set_preference("network.proxy.http_port", proxyPort)
profile.update_preferences()
driver = webdriver.Firefox(firefox_profile=profile)

Ferine

create a local proxy which chains to your proxies. When necessary ask your local proxy to change the "exit". – Millennium

That seems overly complicated. I hope there's an easier way of doing this. – Ferine

it actually sounds like 10 lines of code, there's a python library for anything out there.. – Millennium

Did you by any chance find a solution for this? Or a python lib that solves the problem as @Millennium describes? Just asking, before implementing it on my own... – Hortensehortensia

@Hortensehortensia Nope. I never found a library that did this, and I dropped the project, so I never got around to solving the problem myself. – Ferine

Thanks for your quick response, I'll look into alternatives, maybe proxychains has some way of inter process communication... – Hortensehortensia

Answer 1

This is a slightly old question. But it is actually possible to change the proxies dynamically thru a "hacky way" I am going to use Selenium JS with Firefox but you can follow thru in the language you want.

Step 1: Visiting "about:config"

driver.get("about:config");

Step 2 : Run script that changes proxy

var setupScript=`var prefs = Components.classes["@mozilla.org/preferences-service;1"]
.getService(Components.interfaces.nsIPrefBranch);

prefs.setIntPref("network.proxy.type", 1);
prefs.setCharPref("network.proxy.http", "${proxyUsed.host}");
prefs.setIntPref("network.proxy.http_port", "${proxyUsed.port}");
prefs.setCharPref("network.proxy.ssl", "${proxyUsed.host}");
prefs.setIntPref("network.proxy.ssl_port", "${proxyUsed.port}");
prefs.setCharPref("network.proxy.ftp", "${proxyUsed.host}");
prefs.setIntPref("network.proxy.ftp_port", "${proxyUsed.port}");
                  `;    

//running script below  
driver.executeScript(setupScript);

//sleep for 1 sec
driver.sleep(1000);

Where use ${abcd} is where you put your variables, in the above example I am using ES6 which handles concatenation as shown, you can use other concatenation methods of your choice , depending on your language.(The SetupScript is a string containing the script to be runned enclosed by ``)

Step 3: : Visit your site

driver.get("https://whatismyip.com");

Explanation:the above code takes advantage of Firefox's API to change the preferences using JavaScript code.

Rotogravure

I admittedly haven't tested the solution, and the project was abandoned long ago, but I'll accept the answer anyway. Thanks! – Ferine

I have tested this solution and it works great as such. – Burgeon

Can you provide a python version? Thank – Hum

Can you provide a chrome version?? please. – Shelled

Sorry for a stupid question but where do you run this script? And how do you run it? Can you use variables in it or do you need to create one script for each proxy? I want to use it with Java, if that makes a difference. – Aquila

@Shelled sadly id don't know how to do it in chrome. – Rotogravure

@Aquila You can use java code to nagivate to about:config then use driver.execute script to run the code above. You can use a function that returns a string of the script for each proxy. – Rotogravure

Answer 2

To set a proxy on the fly with Firefox:

def set_proxy(driver, http_addr='', http_port=0, ssl_addr='', ssl_port=0, socks_addr='', socks_port=0):

    driver.execute("SET_CONTEXT", {"context": "chrome"})

    try:
        driver.execute_script("""
          Services.prefs.setIntPref('network.proxy.type', 1);
          Services.prefs.setCharPref("network.proxy.http", arguments[0]);
          Services.prefs.setIntPref("network.proxy.http_port", arguments[1]);
          Services.prefs.setCharPref("network.proxy.ssl", arguments[2]);
          Services.prefs.setIntPref("network.proxy.ssl_port", arguments[3]);
          Services.prefs.setCharPref('network.proxy.socks', arguments[4]);
          Services.prefs.setIntPref('network.proxy.socks_port', arguments[5]);
          """, http_addr, http_port, ssl_addr, ssl_port, socks_addr, socks_port)

    finally:
        driver.execute("SET_CONTEXT", {"context": "content"})

Usage:

 driver = webdriver.Firefox()

 set_proxy(driver, http_addr="212.35.56.21", http_port=8080)

 driver.get("http://....")

 set_proxy(driver, http_addr="212.35.56.22", http_port=8888)

 driver.get("http://....")

Flavourful

I'm trying to use your solution but i keep getting KeyError: 'SET_CONTEXT' and i can't figure what i'm doing wrong. Can't even find the documentation on these functions. Any help is much appreciated. Thanks! – Tabret

A simpler method for getting into chrome context temporarily is with driver.context(driver.CONTEXT_CHROME): <...> – Racemic

Answer 3

To change proxy on the fly with Chrome (work on selenium 3.141.0, key point is driver.start_session(cap)):

   proxy = get_new_proxy()     # x.x.x.x:y
   
   c = {
       "proxyType": "MANUAL",
       "httpProxy": proxy,
       "sslProxy": proxy
   }
   
   cap = webdriver.DesiredCapabilities.CHROME.copy()
   cap['proxy'] = c
   driver.start_session(cap)
   try:
       b.get('https://whatismyip.com')
   except Exception as e:
       print(e)

p.s. selenium.webdriver.common.proxy.Proxy.add_to_capabilities() may also be used when specifying proxy (so you do not need to use the c dict above.)

Privileged

Answer 4

Another option without changing the proxy in selenium would be connecting to a rotating proxy-ip. Most proxy providers offer this functionality where a new ip is assigned for each request you send to the same static proxy-ip.

You need to send an actual request to the rotating ip though, so just opening a new page with webdriver.get() won't work, but you can just send a normal get-request via python's request-module and selenium's traffic now will also be routed to the new ip.

Magpie