How to mock raise urllib errors

Question

After reading this in the python docs, I am catching the HTTPError and URLError exceptions in get_response_from_external_api that the make_request_and_get_response (via urllib's urlopen call) can raise:

foo.main.py

from urllib.request import urlopen
import contextlib
from urllib.error import HTTPError, URLError

def make_request_and_get_response(q):
    with contextlib.closing(urlopen(q)) as response:
        return response.read()

def get_response_from_external_api(q):
    try:
        resp = make_request_and_get_response(q)
        return resp
    except URLError as e:
        print('Got a URLError: ', e)
    except HTTPError as e:
        print('Got a HTTPError: ', e)

if __name__ == "__main__":
    query = 'test'
    result = get_response_from_external_api(query)
    print(result)

While testing the get_response_from_external_api method, I am trying to mock raising the HTTPError and URLError exceptions:

foo.test_main.py

from foo.main import get_response_from_external_api

import pytest
from unittest.mock import patch, Mock
from urllib.error import HTTPError, URLError

def test_get_response_from_external_api_with_httperror(capsys):
    with patch('foo.main.make_request_and_get_response') as mocked_method:
        with pytest.raises(HTTPError) as exc:
            mocked_method.side_effect = HTTPError()  # TypeError
            resp = get_response_from_external_api(mocked_method)

            out, err = capsys.readouterr()
            assert resp is None
            assert 'HTTPError' in out
            assert str(exc) == HTTPError

def test_get_response_from_external_api_with_urlerror(capsys):
    with patch('foo.main.make_request_and_get_response') as mocked_method:
        with pytest.raises(URLError) as exc:
            mocked_method.side_effect = URLError()  # TypeError
            resp = get_response_from_external_api(mocked_method)

            out, err = capsys.readouterr()
            assert resp is None
            assert 'URLError' in out
            assert str(exc) == URLError

But I get a TypeError: __init__() missing 5 required positional arguments: 'url', 'code', 'msg', 'hdrs', and 'fp'. I am new to python mocks syntax and looking for examples.

I have read this answer but I cannot see how this can be applied in my case where the return value of the urllib.urlopen (via get_response_from_external_api) is outside of the scope of the except-block. Not sure if I should instead mock the whole urllib.urlopen.read instead as seen here?

Recluse

Answer 1

There's no need to mock parts of urlopen - by mocking your function to raise an exception you are ensuring that urlopen will not get called.

Since you are creating these exceptions to check that your error-handling code is working, they don't need to be complete - they need only contain the minimum information required to satisfy your tests.

HTTPError expects five arguments:

• a url
• an HTTP status code
• an error message
• the request headers
• a file-like object (the body of the response)

For mocking purposes these could all be None, but it may be helpful to construct an object that looks like a real error. If something is going to read the "file-like object" you can pass io.BytesIO instance containing an example response, but this doesn't seem necessary, based on the code in the question.

>>> h = HTTPError('http://example.com', 500, 'Internal Error', {}, None)
>>> h
<HTTPError 500: 'Internal Error'>

URLError expects a single argument, which can be a string or an exception instance; for mocking purposes, a string is sufficient.

>>> u = URLError('Unknown host')
>>> u

URLError('Unknown host')

Here is the code from the question, amended to take the above into account. And there is no need to pass the mocked function to itself - just pass an arbitrary string. I removed the with pytest.raises blocks because the exception is captured in your code's try/except blocks: you are testing that your code handles the exception itself, not that the exception percolates up to the test function.

from foo.main import get_response_from_external_api

import pytest
from unittest.mock import patch, Mock
from urllib.error import HTTPError, URLError

def test_get_response_from_external_api_with_httperror(capsys):
    with patch('foo.main.make_request_and_get_response') as mocked_method:
        mocked_method.side_effect = HTTPError('http://example.com', 500, 'Internal Error', {}, None)
        resp = get_response_from_external_api('any string')
        assert resp is None
        out, err = capsys.readouterr()
        assert 'HTTPError' in out


def test_get_response_from_external_api_with_urlerror(capsys):
    with patch('foo.main.make_request_and_get_response') as mocked_method:
        mocked_method.side_effect = URLError('Unknown host')
        resp = get_response_from_external_api('any string')
        assert resp is None
        out, err = capsys.readouterr()
        assert 'URLError' in out

Finally, you need to reverse the order of your try except blocks - HTTPError is a subclass of URLError, so you need to test for it first, otherwise it will be handled by the except URLError block.

Tillery