Python how to preserve HTTP cookies

Question

I used this piece to

cj = cookielib.LWPCookieJar()
cookie_support = urllib2.HTTPCookieProcessor(cj)
opener = urllib2.build_opener(cookie_support, urllib2.HTTPHandler)
urllib2.install_opener(opener)

// ..... log in with username and password. 
// urllib2.urlopen() to get the stuff I need. 

Now, how do I preserve the cookie and set the expiration dates to forever, so next time I don't have to log in with username and password again. I can directly use urllib2.urlopen() ?

By "next time" I mean after the program ends, when I start a new program, I can just reload the cookie from the disk and use it

Thanks a lot

Baribaric

Answer 1

I would highly recommend using the Requests HTTP library. It will handle all this for you.

http://docs.python-requests.org/en/latest/

import requests
sess = requests.session()
sess.post("http://somesite.com/someform.php", data={"username": "me", "password": "pass"})
#Everything after that POST will retain the login session
print sess.get("http://somesite.com/otherpage.php").text

edit: To save the session to disk, there are a lot of ways. You could do the following:

from requests.utils import cookiejar_from_dict as jar
cookies = jar(sess.cookies)

Then read the following documentation. You could convert it to a FileCookieJar and save the cookies to a text file, then load them at the start of the program.

http://docs.python.org/2/library/cookielib.html#cookiejar-and-filecookiejar-objects

Alternatively you could pickle the dict and save that data to a file, and load it with pickle.load(file).

http://docs.python.org/2/library/pickle.html

edit 2: To handle expiration, you can iterate over the CookieJar as follows. cj is assumed to be a CookieJar obtained in some fashion.

for cookie in cj:
    if cookie.is_expired():
        #re-attain session

To check if any of the cookies are expired, it may be more convenient to do if any(c.is_expired() for c in cj).

Roseannroseanna

Hi Anorov, maybe I wasn't clear about my question. I was asking, after the program ends, when I start a new program, I can just reload the cookie from the disk and use it, without having to re-login. – Baribaric

Requests is very awesome. Great suggestion :) – Giule

Thanks Anorov. One last question and I will accept your answer. How do I set the expiration time of my CookieJar object? I read the link you pointed but, but I had trouble understanding the difference between CookieJar objects and Cookie objects – Baribaric

A CookieJar acts as a wrapper collection of Cookie objects, with some added methods. Cookie is a separate class and can be found here: docs.python.org/2/library/cookie.html – Roseannroseanna

Why exactly do you want to set your own expiration time? If you want to see if the original cookies the server sent have expired, you can use is_expired() on them. You should only need to do this when checking cookies loaded from a file. – Roseannroseanna

@Roseannroseanna I wanted to set the expiration time to be longer so I don't have to re-login as frequently? – Baribaric

@CodeNoob Well, that's not possible. The server sets the expiration time. When that time is up, the session is no longer valid. You can't do anything to change that. So you can just check if each cookie is expired, and if any of them are, then re-login. – Roseannroseanna

@Roseannroseanna For me this approach returns the entire HTML of the page, is that expected? How can I determine what the cookie is specifically? – Emmery