django.db.utils.InterfaceError: connection already closed failures when updating to Django 3.0

Question

I am updating a medium-sized project to Django 3.0 and I am encountering several errors in my tests after doing nothing more than bumping the Django version from 2.3.

The whole test suite has been running correctly for years and I couldn't find any relevant change in the changelog that may point to the cause of this issue. Apparently a single test fail is triggering every remaining test in the same TestCase class to fail with the following exception:

Traceback (most recent call last):
  File "/Users/federicobond/code/forks/core/env/lib/python3.7/site-packages/django/db/backends/base/base.py", line 238, in _cursor
    return self._prepare_cursor(self.create_cursor(name))
  File "/Users/federicobond/code/forks/core/env/lib/python3.7/site-packages/django/utils/asyncio.py", line 24, in inner
    return func(*args, **kwargs)
  File "/Users/federicobond/code/forks/core/env/lib/python3.7/site-packages/django/db/backends/postgresql/base.py", line 231, in create_cursor
    cursor = self.connection.cursor()
psycopg2.InterfaceError: connection already closed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/federicobond/code/forks/core/apps/participants/tests/test_views.py", line 40, in setUp
    self.client.force_login(self.user)
  File "/Users/federicobond/code/forks/core/env/lib/python3.7/site-packages/django/test/client.py", line 602, in force_login
    self._login(user, backend)
  File "/Users/federicobond/code/forks/core/env/lib/python3.7/site-packages/django/test/client.py", line 611, in _login
    if self.session:
  File "/Users/federicobond/code/forks/core/env/lib/python3.7/site-packages/django/test/client.py", line 461, in session
    session.save()
  File "/Users/federicobond/code/forks/core/env/lib/python3.7/site-packages/django/contrib/sessions/backends/db.py", line 81, in save
    return self.create()
  File "/Users/federicobond/code/forks/core/env/lib/python3.7/site-packages/django/contrib/sessions/backends/db.py", line 51, in create
    self._session_key = self._get_new_session_key()
  File "/Users/federicobond/code/forks/core/env/lib/python3.7/site-packages/django/contrib/sessions/backends/base.py", line 162, in _get_new_session_key
    if not self.exists(session_key):
  File "/Users/federicobond/code/forks/core/env/lib/python3.7/site-packages/django/contrib/sessions/backends/db.py", line 47, in exists
    return self.model.objects.filter(session_key=session_key).exists()
  File "/Users/federicobond/code/forks/core/env/lib/python3.7/site-packages/django/db/models/query.py", line 777, in exists
    return self.query.has_results(using=self.db)
  File "/Users/federicobond/code/forks/core/env/lib/python3.7/site-packages/django/db/models/sql/query.py", line 534, in has_results
    return compiler.has_results()
  File "/Users/federicobond/code/forks/core/env/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1107, in has_results
    return bool(self.execute_sql(SINGLE))
  File "/Users/federicobond/code/forks/core/env/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1135, in execute_sql
    cursor = self.connection.cursor()
  File "/Users/federicobond/code/forks/core/env/lib/python3.7/site-packages/django/utils/asyncio.py", line 24, in inner
    return func(*args, **kwargs)
  File "/Users/federicobond/code/forks/core/env/lib/python3.7/site-packages/django/db/backends/base/base.py", line 260, in cursor
    return self._cursor()
  File "/Users/federicobond/code/forks/core/env/lib/python3.7/site-packages/django/db/backends/base/base.py", line 238, in _cursor
    return self._prepare_cursor(self.create_cursor(name))
  File "/Users/federicobond/code/forks/core/env/lib/python3.7/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/Users/federicobond/code/forks/core/env/lib/python3.7/site-packages/django/db/backends/base/base.py", line 238, in _cursor
    return self._prepare_cursor(self.create_cursor(name))
  File "/Users/federicobond/code/forks/core/env/lib/python3.7/site-packages/django/utils/asyncio.py", line 24, in inner
    return func(*args, **kwargs)
  File "/Users/federicobond/code/forks/core/env/lib/python3.7/site-packages/django/db/backends/postgresql/base.py", line 231, in create_cursor
    cursor = self.connection.cursor()
django.db.utils.InterfaceError: connection already closed

I am out of ideas as to what could be going on here.

Braziel

Please show a full stack trace and a relevant code part – Kacerek

I updated the issue with a longer stack trace. Unfortunately, I cannot pinpoint it to a specific code part. – Braziel

Install django 2.2 and try running it with deprecation warnings (python -Wall manage.py test) – Kacerek

Did you ever solve this? I'm having the same issue. – Cutup

No, every test that errors triggers connection already closed for all the remaining tests in the TestCase class. – Braziel

Note: with the latest Django patch release and after updating some other dependencies to their latest versions I can no longer reproduce this. – Braziel

A good way to debug this kind of issue is to capture the database traffic using something like tcpdump and inspect it with Wireshark; it seems Postgres is closing the connection so the queries might reveal why. – Preponderant

I did notice that if you use TransactionTestCase for every test instead of TestCase it somehow keeps this from happening. I don't understand how or why. – Sheathing

If you are overwriting tearDownClass or setUpClass then make sure you are calling super – Gahan

Answer 1

I ran into this as well. It appears to be a bug in pytest-django. Here's the relevant issue. There's an open PR to resolve it. If it's a big enough inconvenience you can use the branch in that PR or pin your dependencies to an earlier version.

Safekeeping

Although I'm not using pytest-django, I was hit by this. This answer helped me figure out that I had forgotten to call the inherited method inside tearDownClass(). – Kippie

Answer 2

We were hitting the same issue, and upgrading from Django 3.0.2 to Django-3.0.4 resolved it. There are several DB related fixes in those two versions, but I don't know which one solved our problem.

Enculturation

Answer 3

Just a forward note, it's near impossible to provide more information than already is in the stacktrace. However, you can investigate:

• Check when the connection get closed and by which test (run them individually via script for example).
• For tests that fail, check the code for deprecated parts of Django (searching for things that were removed/deprecated between 2.3 and 3.0).
• Run a linter to see if someone changed a private variable inside of Django framework as workaround.
• Check the transactions of postgres.

Then once you have which part of the code has the error narrow it down by creating smaller failing tests.

Albertinaalbertine

Answer 4

I had the same issue using pytest.

Downgrading from 5.4.1 to 5.3.5 fixed it.

Insensate

Answer 5

All these issues occur because of the incompatibility of the other packages with django 3.0 when i ran into this error i updated my requirement.txt file manually and then intalled all the requirements using pip in the same env.

Antonina

Answer 6

In my case I just forgot to run python manage.py makemigrations.

Keys