How to handle timeouts with httplib (python 2.6)?

Question

I'm using httplib to access an api over https and need to build in exception handling in the event that the api is down.

Here's an example connection:

connection = httplib.HTTPSConnection('non-existent-api.com', timeout=1)
connection.request('POST', '/request.api', xml, headers={'Content-Type': 'text/xml'})
response = connection.getresponse()

This should timeout, so I was expecting an exception to be raised, and response.read() just returns an empty string.

How can I know if there was a timeout? Even better, what's the best way to gracefully handle the problem of a 3rd-party api being down?

Emmery

Don't know the answer to your question, but I would encourage you to try MECHANIZE: a higher level API that wraps around httplib and provides an easier way to use HTTP protocol in python. – Mack

Answer 1

urllib and httplib don't expose timeout. You have to include socket and set the timeout there:

import socket
socket.settimeout(10) # or whatever timeout you want

Architect

socket.timeout takes a non-negative floating point in seconds: docs.python.org/library/socket.html#socket.socket.settimeout – Plumy

Ahh yeah. I was just shooting from memory. – Architect

this doesn't answer the question. and yes, httplib does have a timeout since 2.6: docs.python.org/library/httplib.html#httplib.HTTPConnection – Swahili

Answer 2

Even better, what's the best way to gracefully handle the problem of a 3rd-party api being down?

what's mean API is down , API return http 404 , 500 ...

or you mean when the API can't be reachable ?

first of all i don't think you can know if a web service in general is down before trying to access it so i will recommend for first one you can do like this:

import httplib

conn = httplib.HTTPConnection('www.google.com')  # I used here HTTP not HTTPS for simplify
conn.request('HEAD', '/')  # Just send a HTTP HEAD request 
res = conn.getresponse()

if res.status == 200:
   print "ok"
else:
   print "problem : the query returned %s because %s" % (res.status, res.reason)  

and for checking if the API is not reachable i think you will be better doing a try catch:

import httplib
import socket

try:
   # I don't think you need the timeout unless you want to also calculate the response time ...
   conn = httplib.HTTPSConnection('www.google.com') 
   conn.connect()
except (httplib.HTTPException, socket.error) as ex:
   print "Error: %s" % ex

You can mix the two ways if you want something more general ,Hope this will help

Extortionate

Thanks. The second approach is what I was looking for. However, it doesn't raise an exception when trying to connect to a non-existent domain. – Emmery

are you sure because it should raise socket.gaierror which is a subclass of socket.error ??? – Extortionate

and all the errors (i think) are catched using except (httplib.HTTPResponse, socket.error) right ? – Extortionate

I replaced www.google.com with some non-existent domain name and ran it in a python shell. No exception gets raised. Hmmm... – Emmery

try it without the try except and you will see that it raise error :) – Extortionate

@chipotle_warrior: it's weird because after creating a connection to some no existing url like this: conn = httplib.HTTPSConnection('blahblah') and when you try to connect conn.connect() then it should raise the exception gaierror because blahblah it's non-existing domain the gaierror is raised by socket.getaddrinfo() that it's called to get the ip of this url. This is the exception that should be raise : gaierror: [Errno -2] Name or service not known , hope you found out what's the problem :) – Extortionate

Makes sense. Doesn't seem to be the case though. I'll look around for another method. – Emmery

Answer 3

This is what I found to be working correctly with httplib2. Posting it as it might still help someone :

    import httplib2, socket

    def check_url(url):
        h = httplib2.Http(timeout=0.1) #100 ms timeout
        try:
            resp = h.request(url, 'HEAD')
        except (httplib2.HttpLib2Error, socket.error) as ex:
            print "Request timed out for ", url
            return False
        return int(resp[0]['status']) < 400

Waxen