How to read the response body on Python urllib when the status is an error like 400 which raises an exception?

Question

I'm trying to make a request to the GitHub API with Python 3 urllib to create a release, but I made some mistake and it fails with an exception:

Traceback (most recent call last):
  File "./a.py", line 27, in <module>
    'Authorization': 'token ' + token,
  File "/usr/lib/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.6/urllib/request.py", line 532, in open
    response = meth(req, response)
  File "/usr/lib/python3.6/urllib/request.py", line 642, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python3.6/urllib/request.py", line 570, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.6/urllib/request.py", line 650, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 422: Unprocessable Entity

GitHub however is nice, and explains why it failed on the response body as shown at: 400 vs 422 response to POST of data

So, how do I read the response body? Is there a way to prevent the exception from being raised?

I've tried to catch the exception and explore it in ipdb, which gives an object of type urllib.error.HTTPError but I couldn't find that body data there, only headers.

The script:

#!/usr/bin/env python3

import json
import os
import sys

from urllib.parse import urlencode
from urllib.request import Request, urlopen

repo = sys.argv[1]
tag = sys.argv[2]
upload_file = sys.argv[3]

token = os.environ['GITHUB_TOKEN']
url_template = 'https://{}.github.com/repos/' + repo + '/releases'

# Create.
_json = json.loads(urlopen(Request(
    url_template.format('api'),
    json.dumps({
        'tag_namezxcvxzcv': tag,
        'name': tag,
        'prerelease': True,
    }).encode(),
    headers={
        'Accept': 'application/vnd.github.v3+json',
        'Authorization': 'token ' + token,
    },
)).read().decode())
# This is not the tag, but rather some database integer identifier.
release_id = _json['id']

usage: Can someone give a python requests example of uploading a release asset in github?

Cutcliffe

Answer 1

The HTTPError has a read() method that allows you to read the response body. So in your case, you should be able to do something such as:

try:
    body = urlopen(Request(
        url_template.format('api'),
        json.dumps({
            'tag_namezxcvxzcv': tag,
            'name': tag,
            'prerelease': True,
        }).encode(),
        headers={
            'Accept': 'application/vnd.github.v3+json',
            'Authorization': 'token ' + token,
        },
    )).read().decode()
except urllib.error.HTTPError as e:
    body = e.read().decode()  # Read the body of the error response

_json = json.loads(body)

The docs explain in more detail how the HTTPError instance can be used as a response, and some of its other attributes.

Shipload

Perfect, thank you! Next time my debugging will be much easier. I wish the exception would print that by default. – Cutcliffe