Python authenticate and launch private page using webbrowser, urllib and CookieJar

Question

I want to login with cookiejar and and launch not the login page but a page that can only be seen after authenticated. I know mechanize does that but besides not working for me now, I rather do this without it. Now I have,

import urllib, urllib2, cookielib, webbrowser
from cookielib import CookieJar

username = 'my_username'
password = 'my_password'
url = 'my_login_page'

cj = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
login_data = urllib.urlencode({'my_username' : username, 'my_password' : password})
opener.open(url, login_data)
page_to_launch = 'my_authenticated_url'
webbrowser.open(page_to_launch, new=1, autoraise=1)

I am either able to login and dump the authenticated page to stdout, or launch the login page without recognizing the cookie, but I am not able to launch the page I want to after logging in. Help appreciated.

Simonsimona

Answer 1

You could use the selenium module to do this. It starts a browser (chrome, Firefox, IE, etc) with an extension loaded into it that allows you to control the browser.

Here's how you load cookies into it:

from selenium import webdriver
driver = webdriver.Firefox() # open the browser

# Go to the correct domain
driver.get("http://www.example.com")

# Now set the cookie. Here's one for the entire domain
# the cookie name here is 'key' and it's value is 'value'
driver.add_cookie({'name':'key', 'value':'value', 'path':'/'})
# additional keys that can be passed in are:
# 'domain' -> String,
# 'secure' -> Boolean,
# 'expiry' -> Milliseconds since the Epoch it should expire.

# finally we visit the hidden page
driver.get('http://www.example.com/secret_page.html')

Sheya

That's actually a good idea. Thank you. I was thinking to try Selenium. I will give it a try and let you know if it worked. – Simonsimona

Answer 2

Your cookies aren't making it to the browser.

webbrowser has no facilities for accepting the cookies stored in your CookieJar instance. It's simply a generic interface for launching a browser with a URL. You will either have to implement a CookieJar that can store cookies in your browser (which is almost certainly no small task) or use an alternative library that solves this problem for you.

Urinal

Hey Zigg, do you have any library suggestions I can use? Any working example? Any further documentation link? What if I decide then to use mechanize? – Simonsimona

Sorry, no. I'm fairly well-versed in what urllib2 and cookielib can do, but not familiar with the alternatives. – Urinal

You might find Selenium Webdriver interesting. It allows you to pass it cookies. Then open a page in with most major browsers (chrome, FF, IE, etc) – Sheya

@Sheya You should write that up as an answer if it can indeed solve the questioner's problem. – Urinal