Force Python to forego native sqlite3 and use the (installed) latest sqlite3 version
Asked Answered
T

4

39

The error message I am trying to get rid of is:

AttributeError: 'sqlite3.Connection' object has no attribute 'enable_load_extension'

I have 'easy_install'-ed the latest sqlite3 version and python somehow know it is there since sqlite3.version_info produces 3.6.13. In this version the Connection should have the 'enable_load_extension' attribute.

What I think is going on is that python still uses the native sqlite3 module which I think is 2.4.1 since sqlite3.version (i.s.o. sqlite3.version_info) produces 2.4.1.

The question is how do I force python to use the new sqlite3 module for all sqlite3 calls?

Trichology answered 9/10, 2009 at 18:53 Comment(0)
D
73

sqlite3 support in Python can be a bit confusing. The sqlite database adapter started out as a separate project, pysqlite2, but for Python 2.5 a version of it was incorporated into the Python standard library under the name sqlite3. The original adapter continues to be developed as that separate project while periodically the version in Python itself is updated to match it. If you are trying to use a newer version of the adapter, it is usually installed as pysqlite2 so as not to conflict with the version included in the standard library. And, depending how it was built, it may link to a different version of the underlying sqlite3 database library. So make sure you are importing it properly:

>>> import sqlite3
>>> sqlite3.version_info
(2, 4, 1)
>>> sqlite3.sqlite_version_info
(3, 6, 11)

>>> from pysqlite2 import dbapi2 as sqlite3
>>> sqlite3.version_info
(2, 5, 5)
>>> sqlite3.sqlite_version_info
(3, 6, 18)

version_info is the version of the sqlite3 (pysqlite2 or built-in sqlite3) database adapter. sqlite_version_info is the version of the underlying sqlite3 database library.

Using from ... import ... as sqlite3 is suggested so that the rest of your code does not need to change if you move from one version to the other.

Note, enable_load_extension first appeared in pysqlite2 2.5.0.

EDIT: enable_load_extension is disabled by default when you build the adapter. To enable it, you can build pysqlite2 manually. The following recipe assumes a unix-y system and the lastest version of pysqlite2, which as of this writing is 2.5.5.

First, if you installed the adapter originally via easy_install, uninstall it by first running:

$ sudo /path/to/easy_install -m pysqlite # or whatever package name you first used

There will be some output from that including lines like:

Removing pysqlite 2.5.5 from easy-install.pth file

Using /path/to/site-packages/pysqlite-2.5.5-py2.x-something.egg

Remove the egg using the file name listed (the name will vary depending on your platform and version and it may refer to a file or a directory):

$ sudo rm -r /path/to/site-packages/pysqlite-2.5.5-py2.x-something.egg

Now download and extract the pysqlite-2.5.5 source tarball:

$ mkdir /tmp/build
$ cd /tmp/build
$ curl http://oss.itsystementwicklung.de/download/pysqlite/2.5/2.5.5/pysqlite-2.5.5.tar.gz | tar xz
$ cd pysqlite-2.5.5

Then edit the setup.cfg file to comment out the SQLITE_OMIT_LOAD_EXTENSION directive:

$ ed setup.cfg <<EOF
> /SQLITE_OMIT_LOAD_EXTENSION/s/define=/#define=/
> w
> q
> EOF

Since the version of sqlite3 is so old (3.4.0), you should also build with the latest sqlite3 library. This is made easy in the pysqlite2 setup.py script:

$ /path/to/python2.x setup.py build_static

This will automatically download the latest sqlite3 amalgamation source and build the adapter along with an up-to-date statically-linked version of sqlite3. This step may take a long while to finish.

UPDATE (2015/07/21): According to the latest pysqlite 2.6.3 commit you have to download sqlite source code by yourself and put them in pysqlite root folder.

Now, install the adapter:

$ sudo /path/to/python2.x setup.py install

and run the tests:

$ cd     # somewhere out of the build directory
$ /path/to/python2.x
>>> from pysqlite2 import test
>>> test.test()

and, if they pass, you should be all set.

As a bonus, if the reason you want load extension support is to use sqlite3's full text search extension, FTS3, you should find that it was included as part of the static library and no further work is necessary:

>>> from pysqlite2 import dbapi2 as sqlite3
>>> con = sqlite3.connect(":memory:")
>>> con.execute("create virtual table recipe using fts3(name, ingredients)")
<pysqlite2.dbapi2.Cursor object at 0xca5e0>
Dacy answered 9/10, 2009 at 21:36 Comment(16)
Dear Ned: This surely has helped me a lot - thanks! Now my python script does seem to load the most recent version of the adapter after using the 'from ... import ... as sqlite3' construction, as I get 2.5.5 for version_info and 3.4.0 for sqlite_version. Yet, one line further I try to use 'enable_load_extension' and I now get: AttributeError: 'pysqlite2.dbapi2.Connection' object has no attribute 'enable_load_extension' By all the information you have given me this seems to be surprising, no? Thanks a lot for your help thus far - really appreciated! --AlbertTrichology
Darn! I've expanded the answer to include how to build the adapter with load extension support enabled and with an up-to-date version of the sqlite3 library.Dacy
Ned, thanks for your quick reply. I have followed your instructions literally. My system now does have the latest sqlite3 version. However, if I type 'from pysqlite2 import test', I get the following error: from pysqlite2._sqlite import * ImportError: dlopen(/Library/Python/2.5/site-packages/pysqlite2/_sqlite.so, 2): Symbol not found: _sqlite3_enable_load_extension Referenced from: /Library/Python/2.5/site-packages/pysqlite2/_sqlite.so Expected in: dynamic lookup :(, sorry to bother you again with this. I feel I am very close now... Thanks a lot (!) for all your help thus far, AlbertTrichology
It looks like you missed one important detail: the build command is now /usr/bin/python2.5 setup.py build_static . The symptoms you see are what you would get if you typed "build" instead of "build_static". To rebuild: cd /tmp/build ; rm -r build ; /usr/bin/python2.5 setup.py build_static ; sudo ...Dacy
I have tried again based on your advice, but now I get the following error after running 'from pysqlite2 import test': 'ImportError: No module named _sqlite' I do get a lot of warnings along the way when running the 'build_static' command. Ah, I wish I did not have to bother you again with this. AlbertTrichology
Sorry, there was one other thing: you need to cd out of the build directory to be able to import pysqlite2 successfully.Dacy
I have done that (and tried it again just now). Still same issue. I feel that somehow it has something to do with using the standard python that ships with mac os x (which I always update). I plan to do a complete fresh install of the latest python version from source. Do you think this might be the issue? Best, Albert P.S.: Which timezone are you - you seem to be always online :-)? I am in New York.Trichology
From the messages you've reported, I assumed you were using OS X 10.5 and I've stepped through the recipe using the Apple-supplied python 2.5 on 10.5. There's no reason it shouldn't work for you. Installing from source is seldom necessary; if you need a newer version, use an OS X installer from python.org. Just be careful you to know which python you are using: /usr/bin/python2.5 for the Apple one, generally /usr/local/bin/python2.x for a python.org one. ...Dacy
That said: one more idea. Perhaps you have a file permission issue. Try this command: sudo chmod 755 /Library/Python/2.5/site-packages/pysqlite2/_sqlite.so It should not give any error message. Then try the test again.Dacy
IT WORKS - IT WORKS - IT WORKS !!! I am really happy with this Ted, thanks a lot! I did not install the latest python. And, I did not even execute the chmod command you suggested. Somehow the permissioning was done overnight. Happy surprise this morning. Again, thanks and this closes the issues. I am sure others will benefit from reading all your helpful advice here. Best, AlbertTrichology
The day I reach my threshold reputation points I'll vote for you; I understood from trying just now that I do not have any such points yet as a newly arrived enthusiast. AlbertTrichology
Yay! I knew you could do it! If the answer works for you, as a good StackOverflow citizen, you should mark the answer as "accepted". Then the responder gets points.Dacy
I have just done that, right? I have ticked the V so that it colors green (sorry did not know about it).Trichology
It worked for me! (Even though some steps failed and I had to do them manually) But only on Python2. Is there a way that it also updates the Python3 one?Peckham
This helped much. For others' reference, I got the compilation to work simply using a normal python setup.py build, and did not need to do the suggested build_static, which failed for me.Geopolitics
Sadly you are unable to enable_load_extension this way in Python3 since pysqlite2 is only for Python2. In order to enable_load_extension, you have to recompile Python3 with the configure --enable-loadable-sqlite-extensions flag. It is really desirable that pip could handle this kind of customizations in Python2 and Python3, something like: pip install pysqlite3 --configure-opts="--enable-loadable-sqlite-extensions"Whey
P
14

I have python 2.7 on a windows machine and the built-in sqlite3.sqlite_version was 3.6.x. By performing the following steps I was able to get it to use sqlite 3.7.9.

  1. Download and unzip the pre-compiled binary DLL ("sqlite-dll-win32-x86-3070900.zip" on http://www.sqlite.org/download.html)
  2. (probably should close all instances of python at this point just to be safe) Go to C:\Python27\DLLs and change filename of "sqlite3.dll" to "sqlite3.dll.old"
  3. Copy the file "sqlite3.dll" to the folder C:\Python27\DLLs
  4. Open python shell and import sqlite3
  5. Verify sqlite3.sqlite_version shows up as '3.7.9'
Pameliapamelina answered 6/1, 2012 at 16:30 Comment(3)
Thanks Will. However I'm wondering if there is no risk of incompatibility between new sqlite3.dll and the python 2.7.? sqlite3 package code. I suppose sqlite3 package was tested against a particular sqlite3.dll and it's API may evolve in time. Maybe you were just lucky with these particular versions and test scenarios... Or is there a test suite?Ashjian
Hi Dim, I didn't do any testing, so it is possible I just got lucky!Pameliapamelina
Seems to work for me also. I went from 3.6.21 to 3.8.8.2, which is 5 years of updates.Emigration
C
4

You need to look at your Python path and make sure that the sqlite you want is installed in an earlier directory than the built-in sqlite.

You can see the path with:

import sys
print(sys.path)

If you want to find out where a module is from, try:

print(sqlite3.__file__)
Charlacharlady answered 9/10, 2009 at 20:19 Comment(1)
Dear Ned: This is very useful - thanks a lot! I feel I am much closer now to completely solving the issue after your comment and the one posted by Ned Deily. --AlbertTrichology
V
3

Instead of the standard Python sqlite3 module, you might be able to use the apsw module, a third-party SQLite module that more closely follows the SQLite API. It includes support for loading extensions, as well as basically anything that is allowed in the SQLite C/C++ API. It also tries as much as possible to keep up to date with any new changes in SQLite.

Valerianaceous answered 6/3, 2014 at 17:5 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.