Getting the results of a unittest programmatically

Question

I wrote some 'unittest' code, and it's great. I can see it on the CLI when I run it manually.

Now I want to hook it up to run automatically as part of a merge to master hook on my repository. I have everything set up with dummy code except for the part of grabbing the results of the unittest programmatically.

When I call unittest.main() to run them all, it throws a SystemExit. I've tried catching it and rerouting the standard output, but I wasn't able to get it to work, and it also feels like I'm doing it wrong. Is there an easier way to get the results of the unittests, like in a Python list of line strings, or even a more complicated result object?

Really for my purposes, I'm only interested in 100% pass or fail, and then showing that visual in the repository on a pull request to master, with a link to the full unittest result details.

I'm also not married to 'unittest' if some other Python unit test framework can be called and pass off results easily.

Affra

Answer 1

You can pass exit=False to unittest.main, and capture the return value. To run it from another script, or the interactive interpreter, you can specify a target module as well.

That gives us an instance of the TestCase or TestSuite class that was executed. The internal unittest machinery will make a TestResult object available in the result attribute of that object.

That object will have a TestResult.wasSuccessful method that gives the result you're looking for.

Assuming you have some file tests.py:

from unittest import main

test = main(module='tests', exit=False)
print(test.result.wasSuccessful())

Hayley

Answer 2

You can get result by using unittest.TextTestRunner():

import unittest
import os


loader = unittest.TestLoader()

directory_name = 'tests_package' # your directory which contains all your test files
start_dir = os.path.join(os.path.dirname(__file__), directory_name)

suite = loader.discover(start_dir=start_dir,
                        pattern='*_test.py',
                        top_level_dir='./'
                        )

runner = unittest.TextTestRunner()

test_result = runner.run(suite)
print(f"Ran {test_result.testsRun} tests")
print(f"Failures: {len(test_result.failures)}")
print(f"Errors: {len(test_result.errors)}")

I created a little class that simplifies using test_result:

 class ResultRunTests:

    def __init__(self, test_result: unittest.TestResult):
        self.test_result = test_result

    @property
    def is_success(self):
        return not bool(self.test_result.failures)

    @property
    def status(self):
        return 'OK' if self.is_success else 'FAILED'

    @property
    def verbose_result(self):
        text_chunks: typing.List = [self.__str__()]

        if len(self.test_result.failures) > 0 or len(self.test_result.errors) > 0:
            text_chunks.append("Test details:")
            for failed_test, traceback in self.test_result.failures:
                text_chunks.append(f"Failure in {failed_test}:\n"
                                   f"{traceback}"
                                   )

            for errored_test, traceback in self.test_result.errors:
                text_chunks.append(f"Error in {errored_test}:\n"
                                   f"{traceback}"
                                   )

        return '\n'.join(text_chunks)

    def __str__(self):
        return f"Status: {self.status}\n" \
               f"Ran: {self.test_result.testsRun}\n" \
               f"Failures: {len(self.test_result.failures)}\n" \
               f"Errors: {len(self.test_result.errors)}\n"

Topping