Python 3 urllib.request.urlopen

Question

How can I avoid exceptions from urllib.request.urlopen if response.status_code is not 200? Now it raise URLError or HTTPError based on request status.

Is there any other way to make request with python3 basic libs?

How can I get response headers if status_code != 200 ?

Headpin

I dont understand, if there is an error opening the url what do you want to receive? – Yours

Can you give us some code to explain what you're doing? – Saddlebow

@Yours the question is legit, the fact that a webpage returned HTTP 400 mean the the HTTP has error, but it's not a programmatic error, and in some cases you would to read everything (response code, payload, headers) without an exception being thrown. – Squire

Answer 1

Use try except, the below code:

from urllib.request import Request, urlopen
from urllib.error import URLError, HTTPError
req = Request("http://www.111cn.net /")
try:
    response = urlopen(req)
except HTTPError as e:
    # do something
    print('Error code: ', e.code)
except URLError as e:
    # do something
    print('Reason: ', e.reason)
else:
    # do something
    print('good!')

Simile

So how can I get headers in this case? – Headpin

Answer 2

The docs state that the exception type, HTTPError, can also be treated as a HTTPResponse. Thus, you can get the response body from an error response as follows:

import urllib.request
import urllib.error

def open_url(request):
  try:
    return urllib.request.urlopen(request)
  except urllib.error.HTTPError as e:
    # "e" can be treated as a http.client.HTTPResponse object
    return e

and then use as follows:

result = open_url('http://www.stackoverflow.com/404-file-not-found')
print(result.status)           # prints 404
print(result.read())           # prints page contents
print(result.headers.items())  # lists headers

Felker

Answer 3

I found a solution from py3 docs

>>> import http.client
>>> conn = http.client.HTTPConnection("www.python.org")
>>> # Example of an invalid request
>>> conn.request("GET", "/parrot.spam")
>>> r2 = conn.getresponse()
>>> print(r2.status, r2.reason)
404 Not Found
>>> data2 = r2.read()
>>> conn.close()

https://docs.python.org/3/library/http.client.html#examples

Headpin