Using SQLite3 with Django 2.2 and Python 3.6.7 on Centos7

Question

I am moving my Django code from 2.1.7 directly to the new Django 2.2. The only problem I encountered in my Centos7 development environment was that my local development database (sqlite3) version was incompatible using my Python 3.6.7.

The error I was getting from "manage.py runserver" was:

django.core.exceptions.ImproperlyConfigured: SQLite 3.8.3 or later

I am unable to use another version of Python because this is the maximum supported by AWS elasticbeanstalk. The Python 3.6.7 seems to come with sqlite module of version:

>>> import sqlite3
>>> sqlite3.version
'2.6.0'
>>> sqlite3.sqlite_version
'3.7.17'
>>> 

I use a seperate development account on my local Centos7 workstation and issue pipenv shell to begin my code development and IDE.

The only workaround I've found is to manually download SQLite3 autoconf version 3.27.2 and manually compile into that development account home folder using the following commands:

wget https://www.sqlite.org/2019/sqlite-autoconf-3270200.tar.gz
gzip -d sqlite-autoconf-3270200.tar.gz
tar -xvf sqlite-autoconf-3270200.tar
cd sqlite-autoconf-3270200/
./configure --prefix=/home/devuser/opt/
make
make install

Following that, I have modified my .bashrc to reflect the following:

export LD_LIBRARY_PATH="${HOME}/opt/lib"

This seems to do the trick when I log back into my devuser account. My app seems to run correctly using my local development database.

Python 3.6.7 (default, Dec  5 2018, 15:02:05)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-36)] on linux
>>>import sqlite3
>>> sqlite3.version
'2.6.0'
>>> sqlite3.sqlite_version
'3.27.2'

My local development database is SQLite, but my settings.py does not load any SQLite3 database backend when it senses it's in production on AWS (uses Mysql production database as backend when environment variable flag PRODUCTION is checked).

Is my understanding of the problem correct and is my approach and implementation acceptable?

I felt that recompiling python was a huge waste of time, and to be honest it may have been faster to stand up a local mysql version and stop wasting time with sqlite... but it's so nice to just copy or dump a file, migrate, and loaddata for a fresh start.

Rainie

It's best practice to keep same configuration for development and production environments. If you are using mysql in the production, you'd better use it in development. I doubt there is any convenience gain in data migration between different database engines. – Ultramundane

While I agree in general that it is best to keep production and development environments tightly linked, there are scenarios where this may not be possible. I'll give some examples: 1) When testing sensitive data sets. 2) When developing a particular view or functionality that can be done with "disposable" data. – Rainie

What do you mean by testing? Unit testing? – Ultramundane

Function or feature testing where you don't need the entire copy of a database to prove the results of your code changes will be successful when unleashed in production. A subset, or sample, exported from production database into a testing database. This can reduce exposure risk, and make batch process interations quicker to run. Didn't like the last results? Delete the sqlite3 file and replace it with a snapshot taken before the experiment. So much easier. That's why we're here, right? sqlite3 for development. YMMV. – Rainie

I'm not sure if I get you correct. Isn't this a case for mock data? – Ultramundane

So in the end you have settings with testing.sqlite setup, right? – Ultramundane

For testing purposes you can have an initialise command which imports required initial data into db, using FactoryBoy for convenience. – Ultramundane

Does this answer your question? ImproperlyConfigured('SQLite 3.8.3 or later is required (found %s).' % Database.sqlite_version) – Vo

Answer 1

I am using Centos7 with python36.

[shmakovpn@localhost ~]$ ldd /usr/lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so
linux-vdso.so.1 =>  (0x00007ffcafdf6000)
libsqlite3.so.0 => /lib64/libsqlite3.so.0 (0x00007f0adf439000)
libpython3.6m.so.1.0 => /lib64/libpython3.6m.so.1.0 (0x00007f0adef10000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f0adecf4000)
libc.so.6 => /lib64/libc.so.6 (0x00007f0ade927000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f0ade723000)
libutil.so.1 => /lib64/libutil.so.1 (0x00007f0ade520000)
libm.so.6 => /lib64/libm.so.6 (0x00007f0ade21e000)
/lib64/ld-linux-x86-64.so.2 (0x00007f0adf903000)

This means that python36 uses /lib64/libsqlite3.so.0 In python console it looks like this

>>> import sqlite3
>>> sqlite3.version
'2.6.0'
>>> sqlite3.sqlite_version
'3.7.17'

Needs to replace the library, so we must have a new version of it. There are one of way how to do this.

wget http://www6.atomicorp.com/channels/atomic/centos/7/x86_64/RPMS/atomic-sqlite-sqlite-3.8.5-3.el7.art.x86_64.rpm
sudo yum localinstall atomic-sqlite-sqlite-3.8.5-3.el7.art.x86_64.rpm
sudo mv /lib64/libsqlite3.so.0.8.6{,-3.17}
sudo cp /opt/atomic/atomic-sqlite/root/usr/lib64/libsqlite3.so.0.8.6 /lib64

Go to python console once again

>>> import sqlite3
>>> sqlite3.sqlite_version
'3.8.5'

Now it looks much better. Let's try to create Django project and apply migrations

django-admin startproject sqlite3test
cd sqlite3test/
python manage.py migrate
    Operations to perform:
        Apply all migrations: admin, auth, contenttypes, sessions
    Running migrations:
        Applying contenttypes.0001_initial... OK
        ... and so on
ls -al | grep db
    -rw-r--r--.  1 shmakovpn shmakovpn 40960 апр 19 01:02 db.sqlite3

Sqlite3 database successfully created!

Palaeography