How can I have Jenkins fail a build only when the number of test failures changes?

Question

We've customized a product which includes their own phpunit test suite. In Jenkins, I have two jobs setup: the first runs our own test suite that covers our customizations, and the second job runs the existing core unit tests.

The core unit tests were not designed to be run on a customized version, so failures are expected. Out of the ~5000 tests, 81 fail. What I'd like to setup in Jenkins, is have the build marked as a failure only if the number of failed tests changes from the previous build.

I've looked at the Performance plugin but the documentation seems sparse and I'm trying to find something that matches our use case.

Any suggestions?

Bureaucratic

Answer 1

You should have a look at the plugin https://wiki.jenkins-ci.org/display/JENKINS/xUnit+Plugin

It handle a threasolding mechanism (I specified this requirement for the xunit plugin when my team developed it )

expect this helps.. But you want to associates the failure to a change .... Hum maybe more complex .. have to ask .. if such thing should be developped.

Octahedral

This worked like a charm! First run failed (96 out of 5126 test failed), but the second run was green as I set the New Failed threshold to 1, thanks! – Bureaucratic

Is it possible to achieve the same for TestCafe tests runs on Jenkins? – Metrics

If you have or design a plugin for xunit .. to handle TestCafe output .. have a look to other plugins .. there is a lot or eventually ask community how to set it up or find another tool already implemeted that have output in the same format as TestCafe. Or develop a reformating engine to simulate another kind of Xunit Plugin input .. :D – Octahedral