DRF + CoreAPIClient + psycopg2 throws exception, Interface error: connection already closed

P

2

13

I'm writing some integration tests for a web API I build using Django Rest Framework and Django 1.11. My tests are passing when I use DRF's APIClient to send requests to my API endpoints, but when I use the CoreAPIClient the request throws an exception and returns an internal server error. However if I use Postman, or even the coreapi-cli commands it works without a problem.

my test code:

this works:

from rest_framework.test import APITestCase, APIClient

self.client = APIClient()
account_params = {
    'email': '[email protected]',
    'username': 'testuser1',
    'password': 'test1password'
}
account_response = self.client.post(reverse('account-list'), data=account_params)

but using the core api client does not work:

from rest_framework.test import APITestCase, CoreAPIClient

self.client = CoreAPIClient()
self.schema = self.client.get('http://localhost/api/schema')
params = {
    'email': '[email protected]',
    'username': 'testuser3',
    'password': 'test3password'
}
self.test_user = self.client.action(self.schema, ['accounts', 'create'], params=params)

here is the exception I get:

Error
Traceback (most recent call last):
  File "/home/pitt/dev/cps/cps_mono/surveys/tests.py", line 429, in setUp
    self.test_user = self.client.action(self.schema, ['accounts', 'create'], params=params)
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/coreapi/client.py", line 178, in action
    return transport.transition(link, self.decoders, params=params, link_ancestors=link_ancestors)
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/coreapi/transports/http.py", line 386, in transition
    raise exceptions.ErrorMessage(result)
coreapi.exceptions.ErrorMessage: <Error: 500 Internal Server Error>
    message: "<h1>Server Error (500)</h1>"

I switched on the DEBUG_PROPAGATE_EXCEPTIONS flag and the exception throw is now in more detail:

Error
Traceback (most recent call last):
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/django/db/backends/base/base.py", line 231, in _cursor
    return self._prepare_cursor(self.create_cursor(name))
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/django/db/backends/postgresql/base.py", line 220, in create_cursor
    cursor = self.connection.cursor()
psycopg2.InterfaceError: connection already closed

The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/home/pitt/dev/cps/cps_mono/surveys/tests.py", line 429, in setUp
    self.test_user = self.client.action(self.schema, ['accounts', 'create'], params=params)
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/coreapi/client.py", line 178, in action
    return transport.transition(link, self.decoders, params=params, link_ancestors=link_ancestors)
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/coreapi/transports/http.py", line 379, in transition
    response = session.send(request)
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/rest_framework/test.py", line 98, in send
    wsgi_response = self.app(environ, start_response)
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/django/core/handlers/wsgi.py", line 157, in __call__
    response = self.get_response(request)
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/django/core/handlers/base.py", line 124, in get_response
    response = self._middleware_chain(request)
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/django/core/handlers/exception.py", line 43, in inner
    response = response_for_exception(request, exc)
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/django/core/handlers/exception.py", line 93, in response_for_exception
    response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info())
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/django/utils/deprecation.py", line 140, in __call__
    response = self.get_response(request)
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/django/core/handlers/exception.py", line 43, in inner
    response = response_for_exception(request, exc)
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/django/core/handlers/exception.py", line 93, in response_for_exception
    response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info())
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/django/utils/deprecation.py", line 140, in __call__
    response = self.get_response(request)
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/django/core/handlers/exception.py", line 43, in inner
    response = response_for_exception(request, exc)
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/django/core/handlers/exception.py", line 93, in response_for_exception
    response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info())
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/django/utils/deprecation.py", line 140, in __call__
    response = self.get_response(request)
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/django/core/handlers/exception.py", line 43, in inner
    response = response_for_exception(request, exc)
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/django/core/handlers/exception.py", line 93, in response_for_exception
    response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info())
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/django/utils/deprecation.py", line 140, in __call__
    response = self.get_response(request)
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/django/core/handlers/exception.py", line 43, in inner
    response = response_for_exception(request, exc)
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/django/core/handlers/exception.py", line 93, in response_for_exception
    response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info())
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/django/utils/deprecation.py", line 140, in __call__
    response = self.get_response(request)
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/django/core/handlers/exception.py", line 43, in inner
    response = response_for_exception(request, exc)
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/django/core/handlers/exception.py", line 93, in response_for_exception
    response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info())
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/django/utils/deprecation.py", line 140, in __call__
    response = self.get_response(request)
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/django/core/handlers/exception.py", line 43, in inner
    response = response_for_exception(request, exc)
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/django/core/handlers/exception.py", line 93, in response_for_exception
    response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info())
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/django/utils/deprecation.py", line 140, in __call__
    response = self.get_response(request)
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/django/core/handlers/exception.py", line 43, in inner
    response = response_for_exception(request, exc)
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/django/core/handlers/exception.py", line 93, in response_for_exception
    response = handle_uncaught_exception(request, get_resolver(get_urlconf()), sys.exc_info())
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/django/core/handlers/exception.py", line 41, in inner
    response = get_response(request)
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/django/core/handlers/base.py", line 187, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/rest_framework/viewsets.py", line 103, in view
    return self.dispatch(request, *args, **kwargs)
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/rest_framework/views.py", line 483, in dispatch
    response = self.handle_exception(exc)
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/rest_framework/views.py", line 443, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/rest_framework/views.py", line 480, in dispatch
    response = handler(request, *args, **kwargs)
  File "/home/pitt/dev/cps/cps_mono/accounts/views.py", line 38, in create
    if serializer.is_valid(raise_exception=True):
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/rest_framework/serializers.py", line 236, in is_valid
    self._validated_data = self.run_validation(self.initial_data)
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/rest_framework/serializers.py", line 434, in run_validation
    value = self.to_internal_value(data)
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/rest_framework/serializers.py", line 488, in to_internal_value
    validated_value = field.run_validation(primitive_value)
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/rest_framework/fields.py", line 776, in run_validation
    return super(CharField, self).run_validation(data)
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/rest_framework/fields.py", line 524, in run_validation
    self.run_validators(value)
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/rest_framework/fields.py", line 538, in run_validators
    validator(value)
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/rest_framework/validators.py", line 81, in __call__
    if qs_exists(queryset):
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/rest_framework/validators.py", line 24, in qs_exists
    return queryset.exists()
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/django/db/models/query.py", line 670, in exists
    return self.query.has_results(using=self.db)
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/django/db/models/sql/query.py", line 517, in has_results
    return compiler.has_results()
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/django/db/models/sql/compiler.py", line 858, in has_results
    return bool(self.execute_sql(SINGLE))
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/django/db/models/sql/compiler.py", line 887, in execute_sql
    cursor = self.connection.cursor()
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/django/db/backends/base/base.py", line 254, in cursor
    return self._cursor()
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/django/db/backends/base/base.py", line 231, in _cursor
    return self._prepare_cursor(self.create_cursor(name))
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/django/utils/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/django/db/backends/base/base.py", line 231, in _cursor
    return self._prepare_cursor(self.create_cursor(name))
  File "/home/pitt/.virtualenvs/cps_mono/lib/python3.5/site-packages/django/db/backends/postgresql/base.py", line 220, in create_cursor
    cursor = self.connection.cursor()
django.db.utils.InterfaceError: connection already closed

I stepped through my code to find the source of the exception and it seems to be on the Django server side, when I call .is_valid() on my serializer to validate the incoming data. However this only happens when I use the CoreAPIClient that Django REST Framework provides.

Any ideas what the problem could be?

update:

I am running the tests locally though a virtualenvironment and pycharm. My database is running as a remote db in a docker container. My local IP of the local machine and docker host on the docker network is 172.18.0.1 and I added it to the config file. Here is what I have in my pg_hba.conf file

# TYPE  DATABASE        USER            ADDRESS                 METHOD

# "local" is for Unix domain socket connections only
local   all             all                                     trust
# IPv4 local connections:
host    all             all             127.0.0.1/32            trust
host    all             all             172.18.0.1/32           trust
# IPv6 local connections:
host    all             all             ::1/128                 trust
# Allow replication connections from localhost, by a user with the
# replication privilege.
#local   replication     postgres                                trust
#host    replication     postgres        127.0.0.1/32            trust
#host    replication     postgres        ::1/128                 trust

host all all all md5

Update 2:

After having had a look at the database logs, I can confirm that at least part of the SetUp of the CoreAPIClient test is done. The test database was created and the migrations were fully executed before the connection is closed again (as expected). So the initial connection between Django and the database containers is working.

I also tried running the test suite from inside the API container and it throws the same error. So the connection between the docker host and the Django container is not the issue.

Stepping through the code I also saw that the CoreAPIRequest did land on the backend and the endpoint's code is being executed. However it looks like Django is unable to open a new database connection to run any queries. Now why could that be?

Pernick answered 13/4, 2018 at 15:7 Comment(5)

Did you check that local connections are allowed in your pg_hba.conf file? postgresql.org/docs/9.3/static/auth-pg-hba-conf.html – Consequence 13/4, 2018 at 16:32

@JacquesGaudin yes local connections are allowed. I am using docker to run a postgresql container btw. I added IP address of my local machine / docker host to the pg_hba.conf file but it did not fix the issue unfortunately. As I mentioned in the question, the api endpoint and db connection work normally, just when I use the CoreAPIClient it throws this exception. – Pernick 13/4, 2018 at 17:48

It sounds like you can access the network layer of your app but not the interface layer. That's why I thought you didn't have a line starting with local in pg_hba.conf. Do you run the test code from your local machine with a remote database? – Consequence 13/4, 2018 at 18:38

@JacquesGaudin I've updated the question with the contents of my pg_hba.conf file. It is already accepting all local traffic. There must be something in the CoreAPIClient code that causes the database connection to drop. – Pernick 13/4, 2018 at 21:6

What is that 'account-list' view doing in post? I have run into this issue and it is when the view resolves while a separate task is being handled asynchronously. When the post method of the view returns the response, it closes the db connection. I always close the connection before kicking off asynchronous tasks that require it. – Unscratched 6/4, 2022 at 0:53

M

1

I would suggest using rest_framework.test.APITransactionTestCase instead of rest_framework.test.APITestCase

Mooncalf answered 29/12, 2021 at 20:21 Comment(0)

C

0

This seems to be a code issue, are you using django-cors-headers and new open-api based solution? core API is kind of deprecated now.

The 500 Internal Server Error is a "server-side" error, meaning the problem is not with your PC or Internet connection but instead is a problem with the web site's server. ... So definitely the problem is in servers you were using.

Chapa answered 13/4, 2021 at 2:10 Comment(0)

Recommended topics

Hot tags