httplib is not getting all the redirect codes

Question

I am trying to get the final url of a page that seems to redirect more than once. Try this sample URL in your browser and compare it to the final URL at the bottom of my code snippet:

Link that redirects more than once

And here is the test code I was running, notice the final URL that gets a code of 200 isn't the same as the one in your browser. What are my options?

Python 2.7.1+ (r271:86832, Apr 11 2011, 18:13:53)
[GCC 4.5.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import httplib
>>> from urlparse import urlparse
>>> url = 'http://www.usmc.mil/units/hqmc/'
>>> host = urlparse(url)[1]
>>> req = ''.join(urlparse(url)[2:5])
>>> conn = httplib.HTTPConnection(host)
>>> conn.request('HEAD', req)
>>> resp = conn.getresponse()
>>> print resp.status
    301
>>> print resp.msg.dict['location']
    http://www.marines.mil/units/hqmc/

>>> url = 'http://www.marines.mil/units/hqmc/'
>>> host = urlparse(url)[1]
>>> req = ''.join(urlparse(url)[2:5])
>>> conn = httplib.HTTPConnection(host)
>>> conn.request('HEAD', req)
>>> resp = conn.getresponse()
>>> print resp.status
    302
>>> print resp.msg.dict['location']
    http://www.marines.mil/units/hqmc/default.aspx

>>> url = 'http://www.marines.mil/units/hqmc/default.aspx'
>>> host = urlparse(url)[1]
>>> req = ''.join(urlparse(url)[2:5])
>>> conn = httplib.HTTPConnection(host)
>>> conn.request('HEAD', req)
>>> resp = conn.getresponse()
>>> print resp.status
    200
>>> print resp.msg.dict['location']
    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    KeyError: 'location'
>>> print url
    http://www.marines.mil/units/hqmc/default.aspx //THIS URL DOES NOT RETURN A 200 IN ANY BROWSER I HAVE TRIED 

Mesencephalon

Answer 1

you can try to set your User-Agent header to the User-Agent of your browser.

ps: urllib2 automatically redirects

EDIT:

In [2]: import urllib2
In [3]: resp = urllib2.urlopen('http://www.usmc.mil/units/hqmc/')
In [4]: resp.geturl()
Out[4]: 'http://www.marines.mil/units/hqmc/default.aspx

Unknowable

I know, but urllib2 does not tell me what the final proper url is. That's what I am trying to find. – Mesencephalon

Open that final URL you got http://www.marines.mil/units/hqmc/default.aspx in your browser. It redirects to http://www.marines.mil/unit/hqmc/Pages/default.aspx. I checked the server response code in chrome and firefox. Those browsers detect a 302 redirect when python says 200 – Mesencephalon

maybe there is a redirect by javascript or html? – Unknowable

jep, <meta http-equiv="refresh" content="0;url=/unit/hqmc" /> – Unknowable

Nice catch. Just saw that too. I will find a good link on capturing meta refresh redirects, or even javascript ones. Thanks! – Mesencephalon

Answer 2

You can use HttpLib2 to get the actual location of an URL:

import httplib2

def getContentLocation(link):
    h = httplib2.Http(".cache_httplib")
    h.follow_all_redirects = True
    resp = h.request(link, "GET")[0]
    contentLocation = resp['content-location']
    return contentLocation

if __name__ == '__main__':
    link = 'http://podcast.at/podcast_url344476.html'
    print getContentLocation(link)

Execution looks like this:

$ python2.7 getContentLocation.py
http://keyinvest.podcaster.de/8uhr30.rss

Note this example also uses caching (which is not supported neither by urllib nor by httplib). So this will run repeatedly significantly faster. This might be interesting for crawling/scraping. If you do not want caching, replace h = httplib2.Http(".cache_httplib") with h = httplib2.Http().

Bustamante