python urllib2 timing

Question

I'd like to collect statistics related to how long each phase of a web request takes. httplib offers:

def run(self):
    conn = httplib.HTTPConnection('www.example.com')
    start = time.time()
    conn.request('GET', '/')
    request_time = time.time()
    resp = conn.getresponse()
    response_time = time.time()
    conn.close()
    transfer_time = time.time()

    self.custom_timers['request sent'] = request_time - start
    self.custom_timers['response received'] = response_time - start
    self.custom_timers['content transferred'] = transfer_time - start

    assert (resp.status == 200), 'Bad Response: HTTP %s' % resp.status

Are these statistics available from a more high-level interface like urllib2? Is there high level library offering such statistics?

Panter

As far as I know urllib2 doesn't provide that functionality. I'd just add an argument for the URL to your function. – Twelvetone

Answer 1

As mentioned in a related question a good way to do this now is to use the requests library. You can use it to measure the request latency, though I'm not sure if you can measure the content transfer timing. You could potentially do that by comparing a HEAD request to a GET request.

Clinkscales

I deleted my near duplicate response. The "elapsed" option on requests is the easiest way to measure this. If you're interested in measuring the speed of your function itself, that may be solved another way. – Omnirange

Answer 2

time.time is not the most reliable and precise. You can use the timeIt module in python for your profiling purpose. http://docs.python.org/library/timeit.html Here is a code snippet that uses timeit

    statmnt = 'print "Replace print with the snippet you want to profile"'
    setup = 'print "Replace this line with some snippet specific imports"' 
    n = 1 #Number of times you want the timeit module to execute the statmnt
    t = timeit.Timer(statmnt, setup)
    qTime = t.timeit(n)

In your case you will hae to give create three timeit objects, for request, response and content. Do refer the documentation for more info on the module timeit

Sulfonal

According to the documentation, timeit itself uses time.time or time.clock. I think it makes sure to use the right time function for the given platform, but if you do that by yourself it could be even more precise to use time.time or time.clock directly. – Twelvetone

Question isn't about the accuracy of time.time(), but about what can be measured using httplib. – Panter