How do I mock an open used in a with statement (using the Mock framework in Python)?

Question

How do I test the following code with unittest.mock:

def testme(filepath):
    with open(filepath) as f:
        return f.read()

Frannie

Answer 1

The way to do this has changed in mock 0.7.0 which finally supports mocking the python protocol methods (magic methods), particularly using the MagicMock:

http://www.voidspace.org.uk/python/mock/magicmock.html

An example of mocking open as a context manager (from the examples page in the mock documentation):

>>> open_name = '%s.open' % __name__
>>> with patch(open_name, create=True) as mock_open:
...     mock_open.return_value = MagicMock(spec=file)
...
...     with open('/some/path', 'w') as f:
...         f.write('something')
...
<mock.Mock object at 0x...>
>>> file_handle = mock_open.return_value.__enter__.return_value
>>> file_handle.write.assert_called_with('something')

Abigailabigale

Wow! This looks much simpler than the context-manager example currently at voidspace.org.uk/python/mock/magicmock.html which explicitly sets __enter__ and __exit__ to mock objects as well — is the latter approach out of date, or still useful? – Carrell

The "latter approach" is showing how to do it without using a MagicMock (i.e. it is just an example of how Mock supports magic methods). If you use a MagicMock (as above) then enter and exit are preconfigured for you. – Abigailabigale

you could point to your blog post where you explain in more details why/how that works – Borrow

In Python 3, 'file' is not defined (used in the MagicMock spec), so I'm using io.IOBase instead. – Spring

Note: in Python3 the builtin file is gone! – Jeopardous

All I had to read was "...use MagicMock" and that solved it. I was using create_autospec() not realizing the file object had to be a MagicMock. – Windbag

What's the value of file? – Pergrim

It seems you can avoid the explicit MagicMock doing with patch('builtins.open', spec=io.IOBase) as mock: – Unconventional

I'm confused. Why is mock_open still available in the outer scope? – Warden

Answer 2

Python 3

Patch builtins.open and use mock_open, which is part of the mock framework. patch used as a context manager returns the object used to replace the patched one:

from unittest.mock import patch, mock_open
with patch("builtins.open", mock_open(read_data="data")) as mock_file:
    assert open("path/to/open").read() == "data"
mock_file.assert_called_with("path/to/open")

If you want to use patch as a decorator, using mock_open()'s result as the new= argument to patch can be a little bit weird. Instead, use patch's new_callable= argument and remember that every extra argument that patch doesn't use will be passed to the new_callable function, as described in the patch documentation:

patch() takes arbitrary keyword arguments. These will be passed to the Mock (or new_callable) on construction.

@patch("builtins.open", new_callable=mock_open, read_data="data")
def test_patch(mock_file):
    assert open("path/to/open").read() == "data"
    mock_file.assert_called_with("path/to/open")

Remember that in this case patch will pass the mocked object as an argument to your test function.

Python 2

You need to patch __builtin__.open instead of builtins.open and mock is not part of unittest, you need to pip install and import it separately:

from mock import patch, mock_open
with patch("__builtin__.open", mock_open(read_data="data")) as mock_file:
    assert open("path/to/open").read() == "data"
mock_file.assert_called_with("path/to/open")

Pyemia

Grazie! My problem was a bit more complex (I had to channel the return_value of mock_open into another mock object and assert the second mock's return_value), but it worked by adding mock_open as new_callable. – Auxiliary

How should I code my test to work on both py2 and py3? I don't like the idea of checking sys.version_info during the test – Gaming

@ArthurZopellaro take a look to six module to have a consistent mock module. But I don't know if it map also builtins in a common module. – Lacrosse

How do you find the correct name to patch? I.e. how do yo find the first argument to @patch ('builtins.open' in this case) for an arbitrary function? – Phthisic

@Phthisic You have to know what function you're invoking. open() is part of Python's builtin functions (see docs.python.org/3/library/builtins.html). A different example: if you open a file using pathlib.Path.open(), you need to patch accordingly: @patch('pathlib.Path.open', new_callable=mock_open). – Shoot

Please note you may need to refine your assertions. I get AssertionError. Expected: open('file_path') Actual: open('file_path', 'r', -1, None, None) – Ripley

Answer 3

With the latest versions of mock, you can use the really useful mock_open helper:

mock_open(mock=None, read_data=None)

A helper function to create a mock to replace the use of open. It works for open called directly or used as a context manager.

The mock argument is the mock object to configure. If None (the default) then a MagicMock will be created for you, with the API limited to methods or attributes available on standard file handles.

read_data is a string for the read method of the file handle to return. This is an empty string by default.

>>> from mock import mock_open, patch
>>> m = mock_open()
>>> with patch('{}.open'.format(__name__), m, create=True):
...    with open('foo', 'w') as h:
...        h.write('some stuff')

>>> m.assert_called_once_with('foo', 'w')
>>> handle = m()
>>> handle.write.assert_called_once_with('some stuff')

Strawn

how do you check if there are multiple .write calls? – Bahr

@naxa One way is to pass each expected parameter to handle.write.assert_any_call(). You can also use handle.write.call_args_list to get each call if the order is important. – Brunner

m.return_value.write.assert_called_once_with('some stuff') is better imo. Avoids registering a call. – Antonantone

Manually asserting about Mock.call_args_list is safer than calling any of the Mock.assert_xxx, methods. If you mis-spell any of the latter, being attributes of Mock, they will always silently pass. – Spring

Answer 4

To use mock_open for a simple file read() (the original mock_open snippet already given on this page is geared more for write):

my_text = "some text to return when read() is called on the file object"
mocked_open_function = mock.mock_open(read_data=my_text)

with mock.patch("__builtin__.open", mocked_open_function):
    with open("any_string") as f:
        print f.read()

Note as per docs for mock_open, this is specifically for read(), so won't work with common patterns like for line in f, for example.

Uses python 2.6.6 / mock 1.0.1

Lonee

Looks good, but I can't get it to work with for line in opened_file: type of code. I tried experimenting with iterable StringIO that implements __iter__ and using that instead of my_text, but no luck. – Dorfman

@EvgeniiPuchkaryov This works specifically for read() so won't work in your for line in opened_file case; I've edited the post to clarify – Lonee

@EvgeniiPuchkaryov for line in f: support can be achieved by mocking the return value of open() as a StringIO object instead. – Anneliese

To clarify, the system under test (SUT) in this example is: with open("any_string") as f: print f.read() – Directrix

For Python 3+, just change __builtin__ to builtins: #9048245. For the record, builtins is the module containing all top-level functions, see documentation:docs.python.org/3/library/builtins.html (edit: see updated answer a bit below for more details) – Atelectasis

Answer 5

If you don't need any file further, you can decorate the test method:

@patch('builtins.open', mock_open(read_data="data"))
def test_testme():
    result = testeme()
    assert result == "data"

Baeda

Answer 6

The top answer is useful but I expanded on it a bit.

If you want to set the value of your file object (the f in as f) based on the arguments passed to open() here's one way to do it:

def save_arg_return_data(*args, **kwargs):
    mm = MagicMock(spec=file)
    mm.__enter__.return_value = do_something_with_data(*args, **kwargs)
    return mm
m = MagicMock()
m.side_effect = save_arg_return_array_of_data

# if your open() call is in the file mymodule.animals 
# use mymodule.animals as name_of_called_file
open_name = '%s.open' % name_of_called_file

with patch(open_name, m, create=True):
    #do testing here

Basically, open() will return an object and with will call __enter__() on that object.

To mock properly, we must mock open() to return a mock object. That mock object should then mock the __enter__() call on it (MagicMock will do this for us) to return the mock data/file object we want (hence mm.__enter__.return_value). Doing this with 2 mocks the way above allows us to capture the arguments passed to open() and pass them to our do_something_with_data method.

I passed an entire mock file as a string to open() and my do_something_with_data looked like this:

def do_something_with_data(*args, **kwargs):
    return args[0].split("\n")

This transforms the string into a list so you can do the following as you would with a normal file:

for line in file:
    #do action

Critic

If the code being tested handles the file in a different way, for example by calling its function "readline", you can return any mock object you want in the function "do_something_with_data" with the desired attributes. – Sluice

Is there a way to avoid touching __enter__? It definitely looks more like a hack than a recommended way. – Auxiliary

enter is how conext managers like open() are written. Mocks will often be a bit hacky in that you need to access "private" stuff to mock, but the enter here isnt ingerintly hacky imo – Critic

Answer 7

I might be a bit late to the game, but this worked for me when calling open in another module without having to create a new file.

test.py

import unittest
from mock import Mock, patch, mock_open
from MyObj import MyObj

class TestObj(unittest.TestCase):
    open_ = mock_open()
    with patch.object(__builtin__, "open", open_):
        ref = MyObj()
        ref.save("myfile.txt")
    assert open_.call_args_list == [call("myfile.txt", "wb")]

MyObj.py

class MyObj(object):
    def save(self, filename):
        with open(filename, "wb") as f:
            f.write("sample text")

By patching the open function inside the __builtin__ module to my mock_open(), I can mock writing to a file without creating one.

Note: If you are using a module that uses cython, or your program depends on cython in any way, you will need to import cython's __builtin__ module by including import __builtin__ at the top of your file. You will not be able to mock the universal __builtin__ if you are using cython.

Scouring

A variation of this approached worked for me, as the majority of the code under test was in other modules as shown here. I did need to make sure to add import __builtin__ to my test module. This article helped clarify why this technique works as well as it does: ichimonji10.name/blog/6 – Naos

Answer 8

To patch the built-in open() function with unittest:

This worked for a patch to read a json config.

class ObjectUnderTest:
    def __init__(self, filename: str):
        with open(filename, 'r') as f:
            dict_content = json.load(f)

The mocked object is the io.TextIOWrapper object returned by the open() function

@patch("<src.where.object.is.used>.open",
        return_value=io.TextIOWrapper(io.BufferedReader(io.BytesIO(b'{"test_key": "test_value"}'))))
    def test_object_function_under_test(self, mocker):

Astrophysics

This is what worked for me in Python 3.7+ Thank you! – Elwin

Answer 9

I'm using pytest in my case, and the good news is that in Python 3 the unittest library can also be imported and used without issue.

Here is my approach. First, I create a conftest.py file with reusable pytest fixture(s):

from functools import cache
from unittest.mock import MagicMock, mock_open

import pytest
from pytest_mock import MockerFixture


class FileMock(MagicMock):

    def __init__(self, mocker: MagicMock = None, **kwargs):
        super().__init__(**kwargs)

        if mocker:
            self.__dict__ = mocker.__dict__
            # configure mock object to replace the use of open(...)
            # note: this is useful in scenarios where data is written out
            _ = mock_open(mock=self)

    @property
    def read_data(self):
        return self.side_effect

    @read_data.setter
    def read_data(self, mock_data: str):
        """set mock data to be returned when `open(...).read()` is called."""
        self.side_effect = mock_open(read_data=mock_data)

    @property
    @cache
    def write_calls(self):
        """a list of calls made to `open().write(...)`"""
        handle = self.return_value
        write: MagicMock = handle.write
        return write.call_args_list

    @property
    def write_lines(self) -> str:
        """a list of written lines (as a string)"""
        return ''.join([c[0][0] for c in self.write_calls])


@pytest.fixture
def mock_file_open(mocker: MockerFixture) -> FileMock:
    return FileMock(mocker.patch('builtins.open'))

Where I decided to make the read_data as a property, in order to be more pythonic. It can be assigned in a test function with whatever data that open() needs to return.

In my test file, named something like test_it_works.py, I have a following test case to confirm intended functionality:

from unittest.mock import call


def test_mock_file_open_and_read(mock_file_open):
    mock_file_open.read_data = 'hello\nworld!'

    with open('/my/file/here', 'r') as in_file:
        assert in_file.readlines() == ['hello\n', 'world!']

    mock_file_open.assert_called_with('/my/file/here', 'r')


def test_mock_file_open_and_write(mock_file_open):
    with open('/out/file/here', 'w') as f:
        f.write('hello\n')
        f.write('world!\n')
        f.write('--> testing 123 :-)')

    mock_file_open.assert_called_with('/out/file/here', 'w')

    assert call('world!\n') in mock_file_open.write_calls

    assert mock_file_open.write_lines == """\
hello
world!
--> testing 123 :-)
""".rstrip()

Check out the gist here.

Isolt

Answer 10

Sourced from a github snippet to patch read and write functionality in python.

The source link is over here

import configparser
import pytest

simpleconfig = """[section]\nkey = value\n\n"""

def test_monkeypatch_open_read(mockopen):
    filename = 'somefile.txt'
    mockopen.write(filename, simpleconfig)
 
    parser = configparser.ConfigParser()
    parser.read(filename)
    assert parser.sections() == ['section']
 
def test_monkeypatch_open_write(mockopen):
    parser = configparser.ConfigParser()
    parser.add_section('section')
    parser.set('section', 'key', 'value')
 
    filename = 'somefile.txt'
    parser.write(open(filename, 'wb'))
    assert mockopen.read(filename) == simpleconfig

Reidreidar

Answer 11

SIMPLE @patch with assert

If you're wanting to use @patch. The open() is called inside the handler and is read.

    @patch("builtins.open", new_callable=mock_open, read_data="data")
    def test_lambda_handler(self, mock_open_file):
        
        lambda_handler(event, {})

Dudek