Using and Randomizing Proxies

Question

I'm just wondering how you would go about setting a specific proxy for each request?!

The following block quote is the only thing the documentation says about this. Also, the documentation only provides an example for Java...

Firefox version 48 and newer - GeckoDriver
Firefox maintains its proxy configuration in a profile. You can preset the proxy in a profile and use that Firefox Profile or you can set it on profile that is created on the fly as is shown in the following example. With GeckoDriver the proxy has to be passed through the required capabilities.

Any advice would be appreciated!

Thoer

some Java code to start with FirefoxOptions options = new FirefoxOptions(); FirefoxProfile profile = new FirefoxProfile(); profile.setPreference("network.proxy.type", 1); profile.setPreference("network.proxy.http", "localhost"); profile.setPreference("network.proxy.http_port", 6565); profile.setPreference("network.proxy.ssl", "localhost"); profile.setPreference("network.proxy.ssl_port", 6565); options.setProfile(profile); setWebDriver(new FirefoxDriver(options)); – Disfeature

above code only set the proxy for once. Not sure how to change for every request. may be you keep resetting above code with different ports b/w each requets. – Disfeature

@NaveenKumarRB hm... as the tag suggets, looking for Python code – Thoer

setting through FirefoxProfile worked for me. please try out the following code: profile = FirefoxProfile() myProxy = "localhost:3456" proxy = Proxy({ 'proxyType': ProxyType.MANUAL, 'httpProxy': myProxy, 'ftpProxy': myProxy, 'sslProxy': myProxy, 'noProxy': '' }) profile.set_proxy(proxy) driver = webdriver.Firefox(firefox_profile=profile) driver.get("https://www.google.com") driver.quit(). – Disfeature

imports for above code: from selenium import webdriver from selenium.webdriver import Firefox from selenium.webdriver.common.proxy import Proxy, ProxyType from selenium.webdriver.firefox.firefox_profile import FirefoxProfile – Disfeature

@NaveenKumarRB hm... that's the syntax for firefox 47.0.1 and below again... – Thoer

It worked for me for firefox 48 & above also. I tried on firefox 61 – Disfeature

@NaveenKumarRB i'll try it nonetheless. i'm not sure why they wouldn't stick that in the documentation if that's how it's supposed to be implemented for Firefox 47 and above – Thoer

any luck with the above code? – Disfeature

@NaveenKumarRB have yet to implement it. i was just trying to get my ducks in a row for when the time comes! if u check back in a month or something i should hopefully have it implemented by then! i'm stuck on some other bug right now – Thoer

Answer 1

I solved this by setting up proxies on the about:config page on Firefox. Here is the code that you need in order to do it:

devices = {
    "mobile" : "Mozilla/5.0 (Android 4.4; Tablet; rv:41.0) Gecko/41.0 Firefox/41.0",
    "desktop" : "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36 Edge/12.246"
}
scripts = 'var prefs = Components.classes["@mozilla.org/preferences-service;1"].getService(Components.interfaces.nsIPrefBranch); prefs.setIntPref("network.proxy.type", 1); prefs.setCharPref("network.proxy.socks", "' + proxy + '"); prefs.setIntPref("network.proxy.socks_port", port); prefs.setBoolPref("dom.webnotifications.enabled", false); prefs.setCharPref("general.useragent.override", "' + devices[device] + '");'

browser.execute_script(scripts)

If you dont want to override the UA then you dont need to use devices list and just remove the last js rule set in the script.

Nisus

interesting. will have to try it out next time im using proxies and run into this issue – Thoer

Answer 2

usually if I am using proxies with selenium I prefer something a little simple to read and understand

class Properties:
    def __init__(self):
        self.options = Options()
        self.options.headless = True
        self.options.add_argument("ignore-certificate-errors")
        self.options.add_argument("--proxy-server=http://xxx.xxx.xx.54:xx28") #sets a proxy
        self.driver = webdriver.Chrome(options=self.options)
    

What I tend to do is get several different proxies from 'sites that offer free proxies' and test them out, the ones that don't throw an error I store them in a list or file and iterate over them while initializing the selenium class. Which ever one works is the one selenium runs on, if it gets blocked there are several more to pick from. I pull proxies by scraping such sites and storing them in a file on my computer so I do not have to go back to the site the next time I want a working proxy.

Impede

Answer 3

I've set proxies using PhantomJS before, but not using Firefox as the driver. Nonetheless, following this SO post's lead (repasting here for ease of use):

from selenium.webdriver.common.proxy import Proxy, ProxyType
myProxy = "xx.xx.xx.xx:xxxx"

proxy = Proxy({
    'proxyType': ProxyType.MANUAL,
    'httpProxy': myProxy,
    'ftpProxy': myProxy,
    'sslProxy': myProxy,
    'noProxy': '' # set this value as desired
    })
driver = webdriver.Firefox(proxy=proxy)
driver.get("http://www.google.com")

I would attempt to loop through a list of specified proxies and just modify (or re-create) the proxy variable on each request. If you want to randomize it, just call random.choice on the proxy list.

Malleus

hm... your the 2nd person to reference that code. that code is precisely what selenium's very documentation says is obsolete. "Firefox up to version 47.0.1". The current version of Firefox is 61... :/ – Thoer