SQLite3 and multiple processes

Question

How should one ensure correctness when multiple processes access one single SQLite database file?

Eustacia

At least you could have tagged your question with the appropriate programming language. – Topmast

Yeap, or specify that you need a language-agnostic platform-agnostic everything-agnostic approach. – Pinfish

I would try asking again without "show me the code"... – Authorized

Sorry, I'm home blind... Program language of choice is C/C++. – Eustacia

This was my first post here, I will hopefully learn how to make appropriate tags from this mistake. – Eustacia

The SQLite FAQ about exactly this – Heterozygous

Answer 1

First, avoid concurrent access to sqlite database files. Concurrency is one of sqlite's weak points and if you have a highly concurrent application, consider using another database engine.

If you cannot avoid concurrency or drop sqlite, wrap your write transactions in BEGIN IMMEDIATE; ... END;. The default transaction mode in sqlite is DEFERRED which means that a lock is acquired only on first actual write attempt. With IMMEDIATE transactions, the lock is acquired immediately, or you get SQLITE_BUSY immediately. When someone holds a lock to the database, other locking attempts will result in SQLITE_BUSY.

Dealing with SQLITE_BUSY is something you have to decide for yourself. For many applications, waiting for a second or two and then retrying works quite all right, giving up after n failed attempts. There are sqlite3 API helpers that make this easy, e.g. sqlite3_busy_handler() and sqlite3_busy_timeout() but it can be done manually as well.

You could also use OS level synchronization to acquire a mutex lock to the database, or use OS level inter-thread/inter-process messaging to signal when one thread is done accessing the database.

Sacha

I know that one must use transactions within one process. However, my situation is that I have multiple processes, in contrast to multiple threads, accessing the same DB concurrently. Does SQLite transactions really handle such concurrency!?!? – Eustacia

@Tom: Yes, down in the sqlite3 OS-specific porting layer there is functionality for locking that works across processes. See sqlite.org/lockingv3.html for more – Sacha

DEFERRED actually means that the database is not (shared) locked until it's accessed by either a read or write after the BEGIN statement (yes, it should lock on the read). IMMEDIATE means that the database is locked immediately after executing "BEGIN IMMEDIATE TRANSACTION". See sqlite.org/lang_transaction.html – Racing

The word concurrency should probably be interchanged with parallelism. – Eisteddfod

Answer 2

Any SQLite primitive will return SQLITE_BUSY if it tries to access a database other process is accessing at the same time. You could check for that error code and just repeat the action.

Alternatively you could use OS synchronization - mutex on MS Windows or something similar on other OSes. The process will try to acquire the mutex and if someone else already holds it the process will be blocked until the other process finishes the operation and releases the mutex. Care should be taken to prevent cases when the process acquires the mutext and then never releases it.

Pinfish

Hmm, does not SQLite provide locks for protecting the database file access?? – Eustacia

Answer 3

Basically you need to wrap your data access code with transactions. This will keep your data consistent. Nothing else is required.

In SQLite you are using

BEGIN TRANSACTION

COMMIT TRANSACTION

pairs to delimit your transactions. Put your SQL code in between in order to have it execute in a single transaction.

However, as previous people have commented before me - you need to pay close attention for concurrency issues. SQLite can work reasonably fast if it used for read access (multiple readers are not blocked and can run concurrently).

However - the picture changes considerably if your code interleaves write and read access. With SQLite - your entire database file will be locked if even a single writer is active.

Mn