Trouble with psycopg2 in virtualenv python3 for use with Django

Question

I have a Django project that I would like to use with a PostgreSQL database but I am having an issue with psycopg2. When I try to run the dev server or perform a syncdb I get this error. My project is in a virtualenv running Python 3.4 with Django 1.6.

Traceback (most recent call last):
  File "/home/gabriel/DevSpace/Django_Projects/Kevmo/lib/python3.4/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 22, in <module>
    import psycopg2.extensions
ImportError: No module named 'psycopg2.extensions'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/home/gabriel/DevSpace/Django_Projects/Kevmo/lib/python3.4/site-packages/django/core/management/__init__.py", line 399, in execute_from_command_line
    utility.execute()
  File "/home/gabriel/DevSpace/Django_Projects/Kevmo/lib/python3.4/site-packages/django/core/management/__init__.py", line 392, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/gabriel/DevSpace/Django_Projects/Kevmo/lib/python3.4/site-packages/django/core/management/base.py", line 242, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/home/gabriel/DevSpace/Django_Projects/Kevmo/lib/python3.4/site-packages/django/core/management/base.py", line 280, in execute
    translation.activate('en-us')
  File "/home/gabriel/DevSpace/Django_Projects/Kevmo/lib/python3.4/site-packages/django/utils/translation/__init__.py", line 130, in activate
    return _trans.activate(language)
  File "/home/gabriel/DevSpace/Django_Projects/Kevmo/lib/python3.4/site-packages/django/utils/translation/trans_real.py", line 188, in activate
    _active.value = translation(language)
  File "/home/gabriel/DevSpace/Django_Projects/Kevmo/lib/python3.4/site-packages/django/utils/translation/trans_real.py", line 177, in translation
    default_translation = _fetch(settings.LANGUAGE_CODE)
  File "/home/gabriel/DevSpace/Django_Projects/Kevmo/lib/python3.4/site-packages/django/utils/translation/trans_real.py", line 159, in _fetch
    app = import_module(appname)
  File "/home/gabriel/DevSpace/Django_Projects/Kevmo/lib/python3.4/importlib/__init__.py", line 109, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 2231, in _gcd_import
  File "<frozen importlib._bootstrap>", line 2214, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2203, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1200, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1129, in _exec
  File "<frozen importlib._bootstrap>", line 1448, in exec_module
  File "<frozen importlib._bootstrap>", line 321, in _call_with_frames_removed
  File "/home/gabriel/DevSpace/Django_Projects/Kevmo/lib/python3.4/site-packages/django/contrib/admin/__init__.py", line 6, in <module>
    from django.contrib.admin.sites import AdminSite, site
  File "/home/gabriel/DevSpace/Django_Projects/Kevmo/lib/python3.4/site-packages/django/contrib/admin/sites.py", line 4, in <module>
    from django.contrib.admin.forms import AdminAuthenticationForm
  File "/home/gabriel/DevSpace/Django_Projects/Kevmo/lib/python3.4/site-packages/django/contrib/admin/forms.py", line 6, in <module>
    from django.contrib.auth.forms import AuthenticationForm
  File "/home/gabriel/DevSpace/Django_Projects/Kevmo/lib/python3.4/site-packages/django/contrib/auth/forms.py", line 17, in <module>
    from django.contrib.auth.models import User
  File "/home/gabriel/DevSpace/Django_Projects/Kevmo/lib/python3.4/site-packages/django/contrib/auth/models.py", line 48, in <module>
    class Permission(models.Model):
  File "/home/gabriel/DevSpace/Django_Projects/Kevmo/lib/python3.4/site-packages/django/db/models/base.py", line 96, in __new__
    new_class.add_to_class('_meta', Options(meta, **kwargs))
  File "/home/gabriel/DevSpace/Django_Projects/Kevmo/lib/python3.4/site-packages/django/db/models/base.py", line 264, in add_to_class
    value.contribute_to_class(cls, name)
  File "/home/gabriel/DevSpace/Django_Projects/Kevmo/lib/python3.4/site-packages/django/db/models/options.py", line 124, in contribute_to_class
    self.db_table = truncate_name(self.db_table, connection.ops.max_name_length())
  File "/home/gabriel/DevSpace/Django_Projects/Kevmo/lib/python3.4/site-packages/django/db/__init__.py", line 34, in __getattr__
    return getattr(connections[DEFAULT_DB_ALIAS], item)
  File "/home/gabriel/DevSpace/Django_Projects/Kevmo/lib/python3.4/site-packages/django/db/utils.py", line 198, in __getitem__
    backend = load_backend(db['ENGINE'])
  File "/home/gabriel/DevSpace/Django_Projects/Kevmo/lib/python3.4/site-packages/django/db/utils.py", line 113, in load_backend
    return import_module('%s.base' % backend_name)
  File "/home/gabriel/DevSpace/Django_Projects/Kevmo/lib/python3.4/importlib/__init__.py", line 109, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/home/gabriel/DevSpace/Django_Projects/Kevmo/lib/python3.4/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 25, in <module>
    raise ImproperlyConfigured("Error loading psycopg2 module: %s" % e)
django.core.exceptions.ImproperlyConfigured: Error loading psycopg2 module: No module named 'psycopg2.extensions'

Which I have taken as I don't have the module installed in the virtualenv which was confirmed with a pip freeze, so I tried to install it using pip but the installer failed and psycopg2, according to pip freeze still wasn't installed. The weird thing was that I could import psycopg2 using the python interpreter but Django just can't see it for some reason and it is not listed in pip. This is where I run into an issue, after running pip install psycopg2 I get this error message.

Creating build/temp.linux-i686-3.4/psycopg

i686-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -fPIC -DPSYCOPG_DEFAULT_PYDATETIME=1 -DPSYCOPG_VERSION="2.5.3 (dt dec pq3 ext)" -DPSYCOPG_EXTENSIONS=1 -DPSYCOPG_NEW_BOOLEAN=1 -DHAVE_PQFREEMEM=1 -DPG_VERSION_HEX=0x090304 -DPSYCOPG_EXTENSIONS=1 -DPSYCOPG_NEW_BOOLEAN=1 -DHAVE_PQFREEMEM=1 -I/usr/include/python3.4m -I/home/gabriel/DevSpace/Django_Projects/Kevmo/include/python3.4m -I. -I/usr/include/postgresql -I/usr/include/postgresql/9.3/server -c psycopg/psycopgmodule.c -o build/temp.linux-i686-3.4/psycopg/psycopgmodule.o -Wdeclaration-after-statement

In file included from psycopg/psycopgmodule.c:27:0:

./psycopg/psycopg.h:30:20: fatal error: Python.h: No such file or directory

 #include <Python.h>

                    ^

compilation terminated.

error: command 'i686-linux-gnu-gcc' failed with exit status 1

----------------------------------------
Cleaning up...
Command /home/gabriel/DevSpace/Django_Projects/Kevmo/bin/python3.4 -c "import setuptools, tokenize;__file__='/home/gabriel/DevSpace/Django_Projects/Kevmo/build/psycopg2/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-36y2vwa9-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/gabriel/DevSpace/Django_Projects/Kevmo/include/site/python3.4 failed with error code 1 in /home/gabriel/DevSpace/Django_Projects/Kevmo/build/psycopg2
Storing debug log for failure in /tmp/tmpc3blqa5r

I have looked around on SO and some other places and found some people that seem to be having the same problem as me, such as this guy, How to install psycopg2 with "pip" on Python?, and everyone seems to say that my issue will be fixed by performing a apt-get install libpq-dev python-dev which contain the necessary C components to perform the install. Unfortunately I have these packages already, as well as PostgreSQL and the postgreSQL dev tools so I am not sure what I should do to fix this error. Does anybody know of another way for this error to be caused that isn't missing packages?

Dessalines

Try apt-get build-dep psycopg2 – Jordaens

apt-get install python-dev is for Python2, try to install Python3-dev – Suicide

The python3-dev did it. Thank you! Do you have any idea why I was able to import psycopg2 without having it installed though? – Dessalines

Answer 1

To resolve this issue I installed Python3-dev as Leonardo suggested. I was trying to use the python 2 dev tools which obviously didn't work.

Dessalines

Answer 2

I fixed this issue using the package "aiopg" which internally installs psycopg2.This package is specially designed for async operations in postgres, but will solve your purpose aswell

pip install aiopg

Lafond

Answer 3

I too was facing this problem while trying to run a Django project with PostgreSQL database running in a virtual environment on Ubuntu 18.04 which was configured under the Windows WSL subsystem.

As suggested in other answers, I too was installing everything for Python2 unknowingly, later I just installed all the required tools for Python3 and it worked.

I ran the following commands:

sudo apt install postgresql postgresql-contrib  

sudo apt install libpq-dev

sudo apt install python3-dev

sudo apt install python3-pip

sudo apt install python3-psycopg2

pip3 install psycopg2

pip3 install psycopg2-binary

Plus, also check if the Ubuntu and Python and Psycopg versions are compatible together.

Total