WARNING: IPython History requires SQLite, your history will not be saved

Question

Hi I'm using Ubuntu release 12.10 (quantal) 32-bit with Linux Kernel 3.5.0-21-generic. I'm trying to get IPython's History to work. I've set it up using pythonbrew and a virtual environment. In there I use pip to install IPython. Currently, when I start up IPython in a terminal I get:

WARNING: IPython History requires SQLite, your history will not be saved
Python 2.7.3 (default, Nov  8 2012, 18:25:10) 
Type "copyright", "credits" or "license" for more information.

IPython 0.13.1 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

Searching on the warning in the first line, I found this issue report, so I went back and installed the following:

sudo apt-get install libsqlite0 libsqlite0-dev libsqlite3-0 libsqlite3-dev

and then removed and reinstalled pysqlite using pip

pip uninstall pysqlite
pip install pysqlite

After that I thought I would check the installation by importing the module:

Python 2.7.3 (default, Nov  8 2012, 18:25:10) 
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlite3
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/me/.pythonbrew/pythons/Python-2.7.3/lib/python2.7/sqlite3/__init__.py", line 24, in <module>
    from dbapi2 import *
  File "/home/me/.pythonbrew/pythons/Python-2.7.3/lib/python2.7/sqlite3/dbapi2.py", line 27, in <module>
    from _sqlite3 import *
ImportError: No module named _sqlite3

So now it seems the file _sqlite3.so can't be found. That's when I found this SO question. Either it doesn't exist or it's not in my PYTHONPATH environment variable. Searching for the file, I get:

$ locate _sqlite3.so
/home/me/Desktop/.dropbox-dist/_sqlite3.so
/home/me/epd/lib/python2.7/lib-dynload/_sqlite3.so
/usr/lib/python2.7/lib-dynload/_sqlite3.so

So the file is there, but when I looked in my python path:

import sys
for p in sys.path:
    print p

none of the above paths that contain _sqlite3.so were contained in my PYTHONPATH. For giggles, I added the path /usr/lib/python2.7/lib-dynload to my PYTHONPATH in a terminal and then tried to import sqlite3 again:

Python 2.7.3 (default, Nov  8 2012, 18:25:10) 
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.append("/usr/lib/python2.7/lib-dynload")
>>> import sqlite3
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/me/.pythonbrew/pythons/Python-2.7.3/lib/python2.7/sqlite3/__init__.py", line 24, in <module>
    from dbapi2 import *
  File "/home/me/.pythonbrew/pythons/Python-2.7.3/lib/python2.7/sqlite3/dbapi2.py", line 27, in <module>
    from _sqlite3 import *
ImportError: /usr/lib/python2.7/lib-dynload/_sqlite3.so: undefined symbol: PyUnicodeUCS4_DecodeUTF8

Uh oh. Now I'm completely stuck. Can anyone help me out? I've also read in a few places that I may have to rebuild Python. I have no idea how to do this in pythonbrew. Can anyone point me in the right direction?

Hayse

Do you have a good reason to use Pythonbrew? You could avoid this sort of hassle by using the copy of Python that's already there. You can still use virtualenvs with it. – Belga

Hi Thomas K. Yes, I use pythonbrew so that I can keep a few different python versions on my computer to test out code that I write. My main use is to check compatibility between Python 2.X and Python 3.X. – Hayse

Although, come to think of it, I started using pythonbrew at work because I needed a newer version of Python (2.7 I believe) so that I could use some new multiprocessing functions. At work, I have an older Ubuntu distro with Python 2.6, and everything I read warned against trying to upgrade the system version of Python. Upgrading Ubuntu was not an option unfortunately. – Hayse

That makes sense. But all recent versions of Ubuntu should have 2.7, and you can easily have Python 2 and 3 side by side (the latter is just python3). – Belga

I agree. I've actually run into a second problem now. It seems matplotlib won't draw plots on this pythonbrew version of python because pyGTK isn't installed, but pyGTK won't install under pip in Ubuntu. Looks like I'm back to square one. I'm gradually coming to appreciate your original comment. Perhaps I can talk IT into letting me update my Ubuntu distro or maybe I should pickup something prepackaged like EPD. Thoughts? – Hayse

Yeah, I wouldn't expect pip to handle something like PyGTK. If you can get a newer version of Ubuntu (the latest LTS would do), that's probably best. Otherwise, go with one of the Python distros - there's a list on the Scipy Stack install page. – Belga

Answer 1

Thanks to minrk for pointing me in the right direction. All I had to do was rebuild python. I've outlined the steps below for those that are using pythonbrew. Notice that I already installed the libsqlite3-dev package up in the question section.

First, with the proper version of python and virtual environment loaded up run the command:

$ pip freeze -l > requirements.txt

This gives us a text file list of all of the pip packages that have been installed in the virtual environment for this particular python release in pythonbrew. Then we remove the version of python from pythonbrew and reinstall it (this is the "rebuild python" step):

$ pythonbrew uninstall 2.7.3
$ pythonbrew install 2.7.3

After that, we switch over to the newly installed python version 2.7.3 and create a new virtual environment (which I've called "sci"):

$ pythonbrew switch 2.7.3
$ pythonbrew venv create sci
$ pythonbrew venv use sci

Ideally you should be able to run the command:

$ pip install -r requirements.txt

and according to this pip should reinstall all the modules that you had in the virtual environment before we clobbered that version of python (2.7.3). It didn't work for me for whatever reason so I manually installed all of the modules using pip individuality.

$ ipython --pylab

Python 2.7.3 (default, Jan  5 2013, 18:48:27) 
Type "copyright", "credits" or "license" for more information.

IPython 0.13.1 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

and IPython history works!

Hayse

warning: page github.com/utahta/pythonbrew says pythonbrew is no longer under development – Forwarder

Answer 2

I've also read in a few places that I may have to rebuild Python.

This is correct. SQLite is part of the standard library, and is built when you compile Python. There are a few 'optional' parts of the standard library, which Python will simply skip (with minimal warning, unfortunately) if the dependencies are missing at build time, and sqlite is one of these. You should be able to just install libsqlite3-dev, then rebuild Python and you should be set. Keep an eye on the build messages, as they do report which modules they are skipping due to missing dependencies.

Kremlin

+1 for pointing me in the right direction. I documented the process of rebuilding Python for pythonbrew users in one of the answers. – Hayse

Answer 3

What worked for me (using osx + homebrew + brewed python):

# Reinstall Python 2.7 with sqlite
brew remove python
brew install readline sqlite gdbm --universal
brew install python --universal --framework

# Reinstall iPython with correct bindings
pip uninstall ipython    
pip install ipython

And you should be good to go.

Ectoparasite

Answer 4

You should rebuild your python with sqlite support

sudo apt-get install libsqlite3-dev
wget https://www.python.org/ftp/python/2.7.15/Python-2.7.15.tgz
tar -xvf Python-2.7.15.tgz
cd Python-2.7.15
./configure
make
sudo make install

Recreate your virtual environment and you should be good to go

rmvirtualenv venv
mkvirtualenv -p python2 venv
workon venv
pip install -r requirements.txt
# or
pip install ipython

Tannenberg

I don't think reinstalling the venv is necessary as the python bin is just symlinked. – Owner

Answer 5

This warning appears on macOS when python is installed with pyenv. By default it installs python without sqlite. These commands reinstall python with sqlite support:

pyenv uninstall 3.7
CFLAGS="-I$(xcrun --show-sdk-path)/usr/include" pyenv install 3.7

Shanika