Order of tests in python unittest

Question

I was looking at similar questions and I couldn't find an answer to my problem.

I wrote Tests in a python class that derives from unittest.TestCase

class TestEffortFormula(unittest.TestCase)

I need to give an order to the tests (please, do not tell me that I shouldn't rely on test's order, I just do).

Before I needed to give order to the tests the command I used to run the tests was:

unittest.main(testRunner=TeamcityTestRunner())

Then I wanted to make the order dissappear, so I tried the following:

loader = unittest.TestLoader()
loader.sortTestMethodsUsing(None)
loader.loadTestsFromTestCase(TestEffortFormula)
suite = loader.suiteClass()

but from here I don't know how to run the tests, specially with testRunner=TeamcityTestRunner() as I did before.

Appreciate your help

Saavedra

unittest makes no guarantee about the order of execution. And if your tests rely on the order, then you're not doing unit testing properly! Initial and final state should be handled by setup and teardown, not be other test methods – Eladiaelaeoptene

I read some places around stackoverflow that there is a way of doing that, I just couldn't understand it under my conditions. Also, I asked not to tell me why I shouldn't... – Saavedra

Do you need unittest, or can you use another framework? AFAIR proboscis allows test dependency. – Gobbet

Does this answer your question? Python unittest.TestCase execution order – Azurite

Answer 1

Option 1.

One solution to this (as a workaround) was given here - which suggests writing the tests in numbered methods step1, step2, etc., then collecting and storing them via dir(self) and yielding them to one test_ method which trys each.

Not ideal but does what you expect. Each test sequence has to be a single TestClass (or adapt the method given there to have more than one sequence generating method).

Option 2.

Another solution, also in the linked question, is you name your tests alphabetically+numerically sorted so that they will execute in that order.

But in both cases, write monolithic tests, each in their own Test Class.

P.S. I agree with all the comments that say unit testing shouldn't be done this way; but there are situations where unit test frameworks (like unittest and pytest) get used to do integration tests which need modular independent steps to be useful. Also, if QA can't influence Dev to write modular code, these kinds of things have to be done.

Incommode

Answer 2

I've searched a long time to solve this problem myself.
One of the answers in this question does exactly what you need.

Applied to your code:

ln = lambda f: getattr(TestEffortFormula, f).im_func.func_code.co_firstlineno
lncmp = lambda _, a, b: cmp(ln(a), ln(b))
unittest.TestLoader.sortTestMethodsUsing = lncmp

suite = unittest.TestLoader().loadTestsFromTestCase(TestEffortFormula)
unittest.TextTestRunner(failfast=True).run(suite)

Unfortunately, setting unittest.TestLoader.sortTestMethodsUsing=None does not work, although it is documented that this should avoid sorting the tests alphabetically.

Weiland

Because unittest calls dir() to get a list of (possible) test methods and they are already sorted... :-( – Espresso