Why am I getting “HTTP Error 405: Method Not Allowed” when requesting a URL using urllib2?

Question

I am using urllib2 and urllib libraries in python

suppose i had the following code

import urllib2
import urllib

url = 'http://ah.example.com'
half_url = u'/servlet/av/jd?ai=782&ji=2624743&sn=I'

req = urllib2.Request(url, half_url.encode('utf-8'))
response = urllib2.urlopen(req)
print response

when i run the above code i am getting the following error

Traceback (most recent call last):
  File "example.py", line 39, in <module>
    response = urllib2.urlopen(req)
  File "/usr/lib64/python2.7/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib64/python2.7/urllib2.py", line 398, in open
    response = meth(req, response)
  File "/usr/lib64/python2.7/urllib2.py", line 511, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib64/python2.7/urllib2.py", line 436, in error
    return self._call_chain(*args)
  File "/usr/lib64/python2.7/urllib2.py", line 370, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.7/urllib2.py", line 519, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 405: Method Not Allowed

can anyone let me know whats happening here and why its not working

Thanks in advance............

Fabric

Answer 1

The server you are calling is telling you that the POST method is not allowed for the URL you are trying to call.

By passing in the path portion of your URL as the Request object data parameter you are making this a POST instead of a GET.

I suspect you wanted to send a GET request instead:

req = urllib2.Request(url + half_url.encode('utf-8'))

Downtrodden

I suspect that utf-8 encoding the URL without urlencoding it may be a problem, though. – Fretwork

@Wooble: Not with the one in the OP. – Downtrodden

Well, true, although in that example .encode('utf-8') is a no-op since it's already all ASCII and will stay the same. – Fretwork

@Wooble: Yup, quite aware of that. The problem in this question is the POST vs. GET issue though, and I wanted to focus on that. – Downtrodden

Thanks guys for u r support, i got the output by concatenating.I am working with beautiful soup actually for scraping.(Don't mind this is another question may be)Is it possible to use xpath in beutiful soup or just findAll tag itself, because by using findAll i am not getting right data form the tags – Fabric

That is indeed another question; use lxml, it supports xpath 1.0. – Downtrodden

ok thanks for suggestion,but that means we cant use xpath with beautiful soap ? – Fabric

BeautifulSoup doesn't support XPath, no. You can use lxml's BeautifulSoup support to get the best of both worlds. – Downtrodden

Answer 2

I got it because Python 3.7 from ppa:deadsnakes does not support SSL in the Ubuntu Trusty package. The "solution" was to downgrade to Python 3.6.

Paola