Buildbot parsing Python Unit test results

Question

I have a test suite that outputs test results in the Python Unit Test format: http://docs.python.org/library/unittest.html

Is there an existing Buildbot module/plugin that can parse this form?

Example:

DigitalReadWrite_02                                         ... ok
DigitalReadWrite_03                                         ... ok
DigitalReadWrite_04                                         ... ok
PWMoutput_02  (PWM=128 50% LOW 49% HIGH)                    ... ok
PWMoutput_03  (PWM=128 50% LOW 49% HIGH)                    ... ok
PWMoutput_04  (PWM=128 50% LOW 49% HIGH)                    ... ok
--------------------------
Ran 6 tests in 1.652s

OK

I've written a custom parser, but it's only got the basic cases. Is it worth the effort to make it comprehensive for all flavors of Python Unit test format.

Wenonawenonah

It is a job for a test runner such as nose or py.test to provide a parseable output such as XUnit XML format. – Beavers

Answer 1

No, it makes no sense to develop a parser. You can obtain the equivalent information from classes in runner.py module.

Consider extending both classes TextTestRunner and TextTestResult with your custom logic (python 2.7). The output you have listed is produced by TextTestResult.

Alternatively you can extend only TextTestResult and alter the class attribute TextTestRunner.resultclass setting it to your new extension class name.

The data you can extract from TextTestResult and put into some list of dictionaries is greater or equivalent to the data your parser is able to extract.

The unittest framework allows such tricks due to its flexible design. Hope this was helpful.

[EDIT]

I would find publishing your results so far (e.g. as an Open Source code on github) potentially very useful for people that find your question!

Having said that I doubt it would be easy to improve the actual parser beyond basic regexp parsing.

If you still would like to pursuit the text parsing approach - you might need to enumerate and describe "all flavors of Python Unit test format" that you want to cover/support. If you are lucky to put such a description in form of a context-free grammar, then maybe you would be able to develop a parser for it, that would cover "those" cases as a form of a language.

Please take my word of caution: if text parsing is not covered by simple regexp'ing and there is a chance that you are trying to parse some irregular (context-sensitive) language - most likely you would find it extremely difficult to accomplish.

Elyn

Would still be nice to know for people who don't have access to the runner – Ens

Any choice between regexps and object-oriented programming is only a matter of convenience. BTW if you say that you don't have access or control over Runner class, i.e. over organizing your testing environment, then parsing text can be good for text reports. But I would be very much alerted if this is indeed so.. if you don't have proper control over your testing workflow == it might not bring you much anyway. – Elyn

I agree with you practically speaking. But in case someone is interested in doing this as an exercise or for fun, they might want to an answer to the original question. – Ens

I don't think you can do it better than regexp parsing.. Using something like ANTLR requires that you are parsing a formal language with context-free grammar. Seems like a total overkill. In my experience the only thing you can learn by processing text (parsing things out with regexps) w/o being able to define before-head what can be or can not be your input - it is a laborious waste of time. If your are lucky - you get some result. Then this result encourages you to spent even more time - you end up having really complicated hacks of code.. No, I would not advise to do this as an exercise. – Elyn