How to "keep-alive" with cookielib and httplib in python?

Question

In python, I'm using httplib because it "keep-alive" the http connection (as oppose to urllib(2)). Now, I want to use cookielib with httplib but they seem to hate each other!! (no way to interface them together).

Does anyone know of a solution to that problem?

Khosrow

is using httplib a requirement? cookielib works great with urllib2 and you can add your own keep-alive headers. – Socialite

To keep everything consistent, you may want to change all the references to capitalize() by lower() and has_header and get_headers key converted to lower as well – Longsome

Answer 1

HTTP handler for urllib2 that supports keep-alive

Solidago

Answer 2

You should consider using the Requests library instead at the earliest chance you have to refactor your code. In the mean time;

HACK ALERT! :)

I'd go other suggested way, but I've done a hack (done for different reasons though), which does create an interface between httplib and cookielib.

What I did was creating a fake HTTPRequest with minimal required set of methods, so that CookieJar would recognize it and process cookies as needed. I've used that fake request object, setting all the data needed for cookielib.

Here is the code of the class:

class HTTPRequest( object ):
"""
Data container for HTTP request (used for cookie processing).
"""

    def __init__( self, host, url, headers={}, secure=False ):
        self._host = host
        self._url = url
        self._secure = secure
        self._headers = {}
        for key, value in headers.items():
            self.add_header(key, value)

    def has_header( self, name ):
        return name in self._headers

    def add_header( self, key, val ):
        self._headers[key.capitalize()] = val

    def add_unredirected_header(self, key, val):
        self._headers[key.capitalize()] = val

    def is_unverifiable( self ):
        return True

    def get_type( self ):
        return 'https' if self._secure else 'http'

    def get_full_url( self ):
        port_str = ""
        port = str(self._host[1])
        if self._secure:
            if port != 443:
                port_str = ":"+port
        else:
            if port != 80:
                port_str = ":"+port
        return self.get_type() + '://' + self._host[0] + port_str + self._url

    def get_header( self, header_name, default=None ):
        return self._headers.get( header_name, default )

    def get_host( self ):
        return self._host[0]

    get_origin_req_host = get_host

    def get_headers( self ):
        return self._headers

Please note, the class has support for HTTPS protocol only (all I needed at the moment).

The code, which used this class was (please note another hack to make response compatible with cookielib):

cookies = CookieJar()

headers = {
    # headers that you wish to set
}

# construct fake request
fake_request = HTTPRequest( host, request_url, headers )

# add cookies to fake request
cookies.add_cookie_header(fake_request)

# issue an httplib.HTTPConnection based request using cookies and headers from the fake request
http_connection.request(type, request_url, body, fake_request.get_headers())

response = http_connection.getresponse()

if response.status == httplib.OK:
    # HACK: pretend we're urllib2 response
    response.info = lambda : response.msg

    # read and store cookies from response
    cookies.extract_cookies(response, fake_request)

    # process response...

Plush

This hack just saved me a couple hours of reimplementing more or less the same thing myself. Thank you. – Upraise

Welcome, Zack! That's the main reason for sharing. :) – Plush