How to set proxy authentication (user & password) using Python + Selenium

Question

I am using Firefox WebDriver in Python 2.7 with Selenium. My python program starts Firefox browser and visits different websites when I run the program. But, I need to set the proxy with authentication, so that when program visits any website, it will visit through the proxy server.

There are some similar questions on SO. But, there is no specific solution for Selenium Firefox WebDriver of Python.

• Python Selenium Webdriver - Proxy Authentication
• Running selenium behind a proxy server

Kasiekask

u may go through this link: #17989321 – Corkboard

tried that one. But that doesn't set username and password for authentication. – Kasiekask

Firefox maintains its proxy configuration in a profile. You can preset the proxy in a profile and use that Firefox Profile. so i think u have to change ur existing firefox profile. U will found a lots of resource how to change FF profile for proxy. – Corkboard

can you please give me some reference that how can I change firefox profile from python program and save it and reuse it..? – Kasiekask

so far i know, selenium can't do it by itself. have to search more for the latest result. but i think u can handle it through python – Corkboard

Answer 1

Dup of: How to set proxy AUTHENTICATION username:password using Python/Selenium

Selenium-wire: https://github.com/wkeeling/selenium-wire

Install selenium-wire

pip install selenium-wire

Import it

from seleniumwire import webdriver

Auth to proxy

options = {
'proxy': {
    'http': 'http://username:password@host:port',
    'https': 'https://username:password@host:port',
    'no_proxy': 'localhost,127.0.0.1,dev_server:8080'
    }
}
driver = webdriver.Firefox(seleniumwire_options=options)

Warning
Take a look to the selenium-wire cache folder. I had a problem because it take all my disk space. You have to remove it sometimes in your script when you want.

Meridethmeridian

Just an addition to your solution. This will only support Python 3+.. – Trichinosis

Answer 2

In addition to running Firefox with a profile which has the credentials saved. You can do it loading an extension that writes in the loginTextbox and password1Textbox of chrome://global/content/commonDialog.xul (the alert window).

There are already some extensions that will do the job. For instance: Close Proxy Authentication

https://addons.mozilla.org/firefox/downloads/latest/close-proxy-authentication/addon-427702-latest.xpi

from selenium import webdriver
from base64 import b64encode

proxy = {'host': HOST, 'port': PORT, 'usr': USER, 'pwd': PASSWD}

fp = webdriver.FirefoxProfile()

fp.add_extension('closeproxy.xpi')
fp.set_preference('network.proxy.type', 1)
fp.set_preference('network.proxy.http', proxy['host'])
fp.set_preference('network.proxy.http_port', int(proxy['port']))
# ... ssl, socks, ftp ...
fp.set_preference('network.proxy.no_proxies_on', 'localhost, 127.0.0.1')

credentials = '{usr}:{pwd}'.format(**proxy)
credentials = b64encode(credentials.encode('ascii')).decode('utf-8')
fp.set_preference('extensions.closeproxyauth.authtoken', credentials)

driver = webdriver.Firefox(fp)

Liberty

this plugin is no longer working in new firefoxes, any other idea what we could do? – Landlocked

New WebExtensions, unlike good old XUL add-ons, usually store their settings in chrome.storage not in browser's preferences (so you can't find them in about:config). I'd install some new WebExtension which does the job, set their proxy preferences, inspect its storage (chrome.storage.local.get((res)=>{console.log(res)})) and then load it dynamically in the geckodriver and set its preferences through execute_script. – Plagio

Link is not working - I guess you meant that one -> addons.thunderbird.net/en-us/firefox/addon/… (Seems to be outdated) – Focal

Answer 3

You can write own firefox extension for proxy, and launch from selenium. You need write 2 files and pack it.

background.js

var proxy_host = "YOUR_PROXY_HOST";
var proxy_port = YOUR_PROXY_PORT;

var config = {
    mode: "fixed_servers",
    rules: {
      singleProxy: {
        scheme: "http",
        host: proxy_host,
        port: proxy_port
      },
      bypassList: []
    }
 };


function proxyRequest(request_data) {
    return {
        type: "http",
        host: proxy_host, 
        port: proxy_port
    };
}

browser.proxy.settings.set({value: config, scope: "regular"}, function() {;});

function callbackFn(details) {
return {
    authCredentials: {
        username: "YOUR_USERNAME",
        password: "YOUR_PASSWORD"
    }
};
}

browser.webRequest.onAuthRequired.addListener(
        callbackFn,
        {urls: ["<all_urls>"]},
        ['blocking']
);

browser.proxy.onRequest.addListener(proxyRequest, {urls: ["<all_urls>"]});

manifest.json

{
  "name": "My Firefox Proxy",
  "version": "1.0.0b",
  "manifest_version": 2,
  "permissions": [
    "browsingData",
    "proxy",
    "storage",
    "tabs",
    "webRequest",
    "webRequestBlocking",
    "downloads",
    "notifications",
    "<all_urls>"
  ],
  "background": {
    "scripts": ["background.js"]
  },
  "browser_specific_settings": {
    "gecko": {
      "id": "[email protected]"
    }
  }
}

Next you need packed this files to zip archive in DEFLATED mode with .xpi at end like my_proxy_extension.xpi.

You have two choices:

1. Sign your extension Firefox extension .xpi file structure: description, contents, creation, and installation

OR

2. Run unsigned. For this step:

   1. Open firefox flags at about:config and set options xpinstall.signatures.required to false

      OR

   2. Update firefox profile in:

      Windows: C:\Program Files\Mozilla Firefox\defaults\pref\channel-prefs.js

      Linux: /etc/firefox/syspref.js

   Add next line to end of file:

    pref("xpinstall.signatures.required",false);
   

After this steps run selenium and install this extension:

from selenium import webdriver

driver = webdriver.Firefox()
driver.install_addon("path/to/my_proxy_extension.xpi")

driver.get("https://yoursite.com")


 

Reaction

If anyone has an issue getting this working replace browser with chrome in background.js. This code uses callbacks which is how chrome does it. Firefox uses promises. If you use chrome keyword it will work in compatibility mode. – Borisborja

works for me after replacing browser with chrome. Thanks @Borisborja – Topple

Answer 4

There is an example for Firefox + Python but without the authentication here. Then you can find other available parameters here in source code. So it looks like you need the following:

socksUsername
socksPassword

For example:

from selenium import webdriver
from selenium.webdriver.common.proxy import *

myProxy = "host:8080"

proxy = Proxy({
    'proxyType': ProxyType.MANUAL,
    'httpProxy': myProxy, # set this value as desired
    'ftpProxy': myProxy,  # set this value as desired
    'sslProxy': myProxy,  # set this value as desired
    'noProxy': ''         # set this value as desired
    'socksUsername': = ''
    'socksPassword': = ''
    })

driver = webdriver.Firefox(proxy=proxy)

or with preferences:

driverPref = webdriver.FirefoxProfile()
driverPref.set_preference("network.proxy.type", 1)
.
.
.
driverPref.set_preference('network.proxy.socks', proxyHost)
driverPref.set_preference('network.proxy.socks_port', proxyPort)
driverPref.update_preferences()

driver = webdriver.Firefox(firefox_profile=driverPref)

EDIT:

I looked at it again and it seems that it is impossible to set authentication details in FF, even manually. The only way is just to remember the details that you have already entered which done by 2 parameters:

signon.autologin.proxy=true
network.websocket.enabled=false

that can be configured with the set_preference() method. You can also manually view all FF options by browsing to about:config.

Livraison

None of these set the username and password for proxy authentication... – Kasiekask

@RafayetUllah Yeah, you're probably right. Have a look at the edit. – Livraison

Thanks Eugene. I got the solution. Saved the firefox profile with proxy manually entered and running firefox with specific firefox profile. It works fine for cross platform... – Kasiekask

@RafayetUllah That's a great idea. Please post it as an answer here and accept it so other users will be able to see it too. – Livraison

Thanks a lot for the suggestion. I will be adding the answer here. – Kasiekask

how can i set http proxy username/password dynamically? – Brickle

@RafayetUllah I come across this problem recently. can you share your solution? – Telford

@Telford me too, did you ever find a solution? – Landlocked

@TintinabulatorZea Not yet. What I can do is only to get around this problem by exhibiting proxy in a profile. I still cannot figure out how to set proxy/password. anyone got any idea? – Telford

@EugeneS but how to do with GeckoDriver? – Equipollent

Answer 5

In an addition to the answer with extension.

You can also use form filling to dynamically change credentials on your proxy. Just load the extension page, fill the form automatically and click save!

Piderit

Answer 6

We can switch to authentication alert box and enter username password manually.

profile = webdriver.FirefoxProfile()
profile.set_preference("network.proxy.type", 1)
profile.set_preference('network.proxy.ssl_port', int(ProxyPort))
profile.set_preference('network.proxy.ssl', ProxyHost)
profile.set_preference("network.proxy.http", ProxyHost)
profile.set_preference("network.proxy.http_port", int(ProxyPort))

webdriver = webdriver.Firefox(firefox_profile=profile, executable_path=GECKO_DRIVER_PATH)

webdriver.get("https://whatismyipaddress.com/")
try:
    alert = webdriver.switch_to_alert()
    print("switched to alert window")
    alert.send_keys(proxy_username + Keys.TAB + proxy_password)
    alert.accept()
    webdriver.switch_to.default_content()
except Exception:
    print("Error in alert switch")
    pass

Equally