Collecting and Reporting pytest Results

Question

I am doing some Selenium testing via pytest. The next step is to start doing some reporting. I'd like to write something that allows me to run the tests, collect the results and send out an email. So far the closest thing that I've found to this is writing out the test result to the result log and then use a plugin to check the exist status and send an email from there. This works but is slightly cumbersome and I'm hoping that there is a more elegant way to do it. While the overall pytest documentation is great, the plugin documentation is rather poor - I can't even find pytest_sessionfinish anywhere, even though it appears to work.

import pytest

class MyPlugin:
    def pytest_sessionfinish(self, exitstatus):
        if exitstatus == 0:
            #Send success email
            pass
        else: 
            #Read output.txt
            #Add output.txt to email body
            #Send email
            pass

pytest.main("--resultlog=output.txt", plugins=[MyPlugin()])

---

Q: What is the best way to run and collect results from pytest?

---

Bianka

Answer 1

Install pytest-html and then run test with --html=pytest_report.html option.

Kept

And while you're at it, use the "--self-contained-html" option to generate a single HTML file that you can load into a browser to see the results. – Mazdaism

Answer 2

One easy way to generate a result report is using the pytest option --junitxml when running tests. pytest will generate a test report in JUnit format.

Since JUnit is widely used, it's easy to find tools to parse the report and generate some good looking output, like HTML reports. As far as I know, there are some plugins on Jenkins that work fine for parsing JUnit reports and provide nice reports.

Take a look at 'Creating JUnitXML format files' section in the pytest documentation for more information.

More than that, pytest provides a way to expand the JUnit XML report when you have access to the pytest object request or config:

if hasattr(request.config, "_xml"):
    request.config._xml.add_custom_property(name, value)

If in test cases, pytest provides a fixture to do that:

def test_function(record_xml_property):
    record_xml_property("key", "value")
    assert 0

This will add a custom property to the JUnit XML report.

Camus

How/ where did you access request object? Did you do it in pytest_runtest_logreport? – Titanomachy

The link to pytest documentation does not contain the mentioned section anymore. Replace it with pytest.org/en/latest/how-to/…? – Trilbie

Answer 3

If you want to implement reporting, there are many easy off-the-shelf solutions for Pytest. Below are some of them:

• Excel Reporting - https://pypi.python.org/pypi/pytest-excel
• HTML Reporting - https://pypi.python.org/pypi/pytest-html
• Allure Reporting (I prefer this) - https://allurereport.org/docs/pytest/

Curtiscurtiss

for Allure link has changed: docs.qameta.io/allure-report – Scrapbook

Answer 4

Another way to create reports is to first collect the results as a dictoinary or dataframe, and then do whatever you like with it (write a csv, etc).

If you wish to go down that path, you can use pytest-harvest for this. Simply install it, and you can directly use the pre-defined fixtures:

import pytest
import time

@pytest.mark.parametrize('p', ['world', 'self'], ids=str)
def test_foo(p):
    """
    A dummy test, parametrized so that it is executed twice
    """
    print('\n   hello, ' + p + ' !')
    time.sleep(len(p) / 10)

def test_synthesis(module_results_df):
    """
    Shows that the `module_results_df` fixture already contains what you need
    """
    # drop the 'pytest_obj' column
    module_results_df.drop('pytest_obj', axis=1, inplace=True)

    print("\n   `module_results_df` dataframe:\n")
    print(module_results_df)

Yields

>>> pytest -s -v

============================= test session starts =============================
...
collecting ... collected 3 items
test_basic.py::test_foo[world] 
   hello, world !
PASSED
test_basic.py::test_foo[self] 
   hello, self !
PASSED
test_basic.py::test_synthesis 
   `module_results_df` dataframe:

                 status  duration_ms      p
test_id                                    
test_foo[world]  passed   500.028610  world
test_foo[self]   passed   400.022745   self
PASSED

========================== 3 passed in 0.05 seconds ===========================

You can also start from the 'dict' fixture, which contains more details about setup/teardown times, and convert it to a dataframe using the provided helper methods. See documentation for details.

Finally, if you wish to also use parameters, fixtures, steps... you might wish to look at this datascience benchmark example

I'm the author by the way ;)

Marengo

I am interested in using this library. I'm having some trouble understanding how to use it. I've installed but I'm unable to use, for example module_results_dct without importing things. The examples indicate this should unnecessary. I am doing something wrong. Sorry for messaging this way but I tried in vain to privately message you. – Rockies

@AndrewFalanga 's issue is fixed by the way: github.com/smarie/python-pytest-harvest/issues/37 – Marengo

Answer 5

I have found pytest-json-report to be super useful for generating pytest reports.

It creates a json with all of your data which you can then push up to your data warehouse. It's going to require a bit more set up than pytest-html for example, but I've found the flexibility is worth it.

I wrote a medium article on how to build a pipline using pytest-json-report, snowflake & an engineering friendly BI tool called Visivo.io.

Article: https://medium.com/@jared_86317/visualizing-pytest-results-with-the-modern-data-stack-29daeab99b66

Once you have the dashboard pipeline running it's super easy to set up email alerts, additional reporting ect.

Riordan