How do I load session and cookies from Selenium browser to requests library in Python?

Question

How can I load session and cookies from Selenium browser? The following code:

import requests

cookies = [{u'domain': u'academics.vit.ac.in',
            u'name': u'ASPSESSIONIDAEQDTQRB',
            u'value': u'ADGIJGJDDGLFIIOCEZJHJCGC',
            u'expiry': None, u'path': u'/',
            u'secure': True}]
response = requests.get(url2, cookies=cookies)

gives me the following exception:

Traceback (most recent call last):
  File "F:\PYTHON\python_scripts\cookies\cookies3.py", line 23, in <module>
    response = requests.get(url2, cookies=cookies)
  File "C:\Python27\lib\site-packages\requests\api.py", line 68, in get
    return request('get', url, **kwargs)<br/>
  File "C:\Python27\lib\site-packages\requests\sessions.py", line 450, in request
    prep = self.prepare_request(req)
    cookies = cookiejar_from_dict(cookies)
  File "C:\Python27\lib\site-packages\requests\cookies.py", line 439, in cookiejar_from_dict
    cookiejar.set_cookie(create_cookie(name, cookie_dict[name]))
TypeError: list indices must be integers, not dict

Waine

Yeah its possible, but why would you want to do that? – Whisler

@Whisler am using selenium to overcome captcha based login. then onwards i need to parse more than 1000 urls for xml info. But if i use selenium i will have to load the page using browser, but i need to do it in background only – Waine

Answer 1

First you have to get the cookies from your driver instance:

cookies = driver.get_cookies()

This returns cookie dictionaries for your session.

Next, set those cookies in requests:

s = requests.Session()
for cookie in cookies:
    s.cookies.set(cookie['name'], cookie['value'])

Whisler

getting error in this line s.cookies.set(**cookie) TypeError: create_cookie() got unexpected keyword arguments: [u'expiry'] the cookie when printed is like this [{u'domain': u'acemics.it.ac.in', u'secure': True, u'value': u'PBHHJGJDHMSEEHGCDJLICARE', u'expiry': None, u'path': u'/', u'name': u'ASPSESSIONIDAEQDTQRB'}] – Waine

error in line a = requests.get(url2, cookies=cookies) TypeError: list indices must be integers, not dict – Waine

That1Guy, most probably the downvotes are due to the fact you don't care about extra parameters like http only, path etc. Though, it's just an educated guess, you know. – Tzong

@MichaelCGood I exported cookies from selenium driver first. It was then onboarded to requests session object before making the next request. – Waine

Can you help me with reverse of this,that is from request to selenium, I am trying to, but selenium always seems to start a fresh session :( – Sleekit

I am not able to get this to work, e.g. if I take a website with consent cookies and I open it in selenium then pull the cookies and inject them into requests then they do not work and have not impact (I tried with Mashable as it requires consent from EU users) – Easley

@nonein Are you certain the cookies are being set? I'd set up some print statements so you can track which cookies are received and which are set. – Whisler

Answer 2

You can use 3rd party package like selenium-requests or requestium. They provide function to share cookies between requests and selenium.

Smriti

Answer 3

I made this solution, it's solved unexpected keyword arguments expiry

def set_cookies(cookies, s):
    for cookie in cookies:
        if 'httpOnly' in cookie:
            httpO = cookie.pop('httpOnly')
            cookie['rest'] = {'httpOnly': httpO}
        if 'expiry' in cookie:
            cookie['expires'] = cookie.pop('expiry')
        s.cookies.set(**cookie)
    return s

Slaver

Answer 4

I think this doesn't directly answer the question but if your goal is to make requests pretending to be the same session that selenium is, you can do this:

resp = browser.execute_script(f'''
    var xmlHttp = new XMLHttpRequest();
    xmlHttp.open( "GET", "{yourUrl}", false );
    xmlHttp.send( null );
    return xmlHttp.responseText;
'''
)

Archiepiscopal