How to preload cookies before first request with Python3, Selenium Chrome WebDriver?

Question

Is it possible to add cookies using add_cookie() for a domain, say stackoverflow.com in Selenium Chrome WebDriver before doing an actual request using get() to a page on domain stackoverflow.com ?

When trying:

driver.webdriver.add_cookie({'name' : 'testcookie', 'value' : 'testvalue', 'domain' : 'stackoverflow.com'})
driver.webdriver.get('https://stackoverflow.com/')

I get "You may only set cookies for the current domain".

I want to say I saw and tried some solutions of avoiding the issue instead of solving it, such as visiting a 404 page on the domain beforehand to create the "domain slot" in Selenium before adding cookies to it, but while these solution allow adding the cookies they still require making one extra request and making contact with the site while not having any cookies set.

This is an issue when dealing with CAPTCHA systems and some very specific WAFs which frown upon seeing, in succession, a request with no cookies then another request with cookies that should only be had say after going through the login process.

Zebrass

Answer 1

Starting Chrome 64 we now have access to Chrome DevTools Protocol v1.3 which allows setting cookies to any domain through the method Network.setCookie, thus eliminating the need for an extra get call to set up the domain beforehand.

e.g. Python3

import os.path
import pickle
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

def save_cookies():
    print("Saving cookies in " + selenium_cookie_file)
    pickle.dump(driver.get_cookies() , open(selenium_cookie_file,"wb"))

def load_cookies():
    if os.path.exists(selenium_cookie_file) and os.path.isfile(selenium_cookie_file):
        print("Loading cookies from " + selenium_cookie_file)
        cookies = pickle.load(open(selenium_cookie_file, "rb"))

        # Enables network tracking so we may use Network.setCookie method
        driver.execute_cdp_cmd('Network.enable', {})

        # Iterate through pickle dict and add all the cookies
        for cookie in cookies:
            # Fix issue Chrome exports 'expiry' key but expects 'expire' on import
            if 'expiry' in cookie:
                cookie['expires'] = cookie['expiry']
                del cookie['expiry']

            # Replace domain 'apple.com' with 'microsoft.com' cookies
            cookie['domain'] = cookie['domain'].replace('apple.com', 'microsoft.com')

            # Set the actual cookie
            driver.execute_cdp_cmd('Network.setCookie', cookie)

        # Disable network tracking
        driver.execute_cdp_cmd('Network.disable', {})
        return 1

    print("Cookie file " + selenium_cookie_file + " does not exist.")
    return 0

def pretty_print(pdict):
    for p in pdict:
        print(str(p))
    print('',end = "\n\n")


# Minimal settings
selenium_cookie_file = '/home/selenium/test.txt'

browser_options = Options()
browser_options.add_argument("--headless")

# Open a driver, get a page, save cookies
driver = webdriver.Chrome(chrome_options=browser_options)
driver.get('https://apple.com/')
save_cookies()
pretty_print(driver.get_cookies())


# Rewrite driver with a new one, load and set cookies before any requests
driver = webdriver.Chrome(chrome_options=browser_options)
load_cookies()
driver.get('https://microsoft.com')
pretty_print(driver.get_cookies())

You will see above that we get cookies from domain 'apple.com' with one request and have them loaded and set for the second request to domain 'microsoft.com' before actually making the request to 'microsoft.com'.

The code is tested and works as long as you set the variable selenium_cookie_file to a valid writable file path.

Zebrass

Just wanted to say thanks! I've been searching for ways how to preload my cookies before the driver call and this solution have worked! Unfortunately, the other recommendations such as navigate to a 404 or image didn't work for me as the entire site is MFA protected and the cookies are triggered once the site loads which I now managed to achieve with this solution. PS: besides the selenium_cookie_file variable have to be a valid path also the replace domain section has to be edited as well for everyone's own needs :-) – Planet

Nice answer! Thank you! – Alcaide

This doesn't work on Firefox, is there something similar? – Fitful

@CatHariss, this solution is Chrome specific - Q/A title -, sorry nothing similar that I know of for Firefox. – Zebrass

I have opened a separate topic for this here: #74667372 – Fitful

Thanks for sharing this @AlexProtopopescu! Really helpful! I'm just wondering, where did you learn about these CDP-commands? Like "Network.enable", "Network.setCookie" and so on? Is there any documentation somewhere? – Lewert

Never mind =D I found this greate website called Google that helped me find this: chromedevtools.github.io/devtools-protocol/tot/Network – Lewert