How/why do unit tests affect each other in a TestCase and how to prevent this behaviour?

Question

I am writing tests for a webapp that is made in Python3.6/Django 2.0 and I have the following situation:

class TestFoo(TestCase):
    def test_a(self):
        obj = Foo.objects.create(a = "bar")
        expectation = {"a" : "bar"}
        self.assertEquals(obj.method_x(), expectation)

    def test_b(self):
        obj = Foo.objects.create(a = "baz")
        expectation = {"a" : "baz"}
        self.assertEquals(obj.method_x(), expectation)

    def test_c(self):
        obj = Foo.objects.create(a = "bar")
        obj2 = Foo.objects.create(a = "baz")
        obj.b.add(obj2)
        expectation = {"a" : "bar", "b" : {"a": "baz"}}
        self.assertEquals(obj.method_x(), expectation)

To my understanding every test is run in isolation, yet when I run test_c alongside test a or b, all tests fail. Basically this:

• test_a + test_b + test_c = ALL FAIL
• test_a + test_b = ALL PASS
• test_c = ALL PASS
• test_a + test_c = ALL FAIL
• test_b + test_c = ALL FAIL

I have tried:

1. Deleting all Foo objects in teardown (in case this didn't happen), this had no effect
2. Using patch.object, but this was not the behavior I desired because I want to test the method works correctly
3. Sticking test_c in a separate class, this had no effect
4. Running the tests in a certain order (a, then b, then c and first c, then a/b, this resulted in different points of failure; if I first run c, it passes, then a/b fail. If I run a/b first, then c fails

I am unsure what causes this behavior, but would like to resolve it; I know all tests by themselves should pass. I have been reading about mock/patch methods but I am pretty sure that this is not what I need because I need to verify that the method(s) of my object return valid data rather than ensure that they get called or anything like that.

So basically my question is twofold:

1. Why is this happening?
2. How do I prevent it?

Edit 1: The assertion error traceback is pretty weird too; obviously the values do not equal each other, but what's more is that the values are somehow mixed up. Somehow test_a.method_x() == test_c.method_x(), but test_a.a =/= test_c.a

method_x is something like:

def method_x(self):
    if not self.b:
        return {"a": self.a}
    else:
        return {"a": self.a, "b":{"a":self.b.a}}

The model looks something like:

class Foo(models.Model):
    A_TYPES = (
        ("bar", "Bar"),
        ("baz", "Baz")
    )
    a = models.CharFields(max_length20, choices=A_TYPES)
    b = models.ManyToManyField("self")
    c = models.IntegerField(null=True)
    d = models.BooleanField(default=False)

Cursive

This probably needs more detail on the models and the tests. There are a couple things that could cause this behavior (most likely: something setting a default that's stored in code rather than the DB, and so the first test to call it "wins" by setting the default), but without more detail it's impossible to tell. – Helle

I am not sure this is the case. There are only 2 relevant properties to the method ('a' and 'b'), all other properties default to null/default value, the only property that is set is 'a' in these tests and in the case of test c, a manytomany relation is added to itself on property 'b'. But even that is not relevant, because if I make a test_d that does not have a manytomany relation to itself, the test fails. I shall add the model details just in case though – Cursive

Which TestCase are you extending? Presumably it is Django's version and not unittest.TestCase? – Motheaten

That is correct, however from what I've read django Testcase extends the python TestCase, so I imagine they work similarly – Cursive

Can you reproduce the behavior using only the simplified code you've posted here? – Helle

Answer 1

I had the problem of tests interfering with each other today. Problem turned out to be a caching issue, as in:

https://mcmap.net/q/1919120/-django-cache-isolation-when-running-tests-in-parallel

from django.core.cache import cache
from django.test import TestCase

class TheTestCase(TestCase):

    def tearDown(self):
        super().tearDown()
        cache.clear()

Or you could disable the cache during tests. That may slow other tests so you'll have to decide.

Hoban