Key/Value database for storing binary data

Question

I am looking for a lightweight, reliable and fast key/value database for storing binary data. Simple without server. Most of popular key/value databases like CDB and BerkeleyDB does not natively store BLOB. What can be the best choice that I missed?

My current choice is SQLite, but it is too advanced for my simple usage.

Pediform

Answer 1

As it was previously pointed out, BerkeleyDB does support opaque values and keys, but I would suggest a better alternative: LevelDB.

LevelDB:

Google is your friend :), so much so that they even provide you with an embedded database: A fast and lightweight key/value database library by Google.

Features:

• Keys and values are arbitrary byte arrays.
• Data is stored sorted by key.
• Callers can provide a custom comparison function to override the sort order.
• The basic operations are Put(key,value), Get(key), Delete(key).
• Multiple changes can be made in one atomic batch.
• Users can create a transient snapshot to get a consistent view of data.
• Forward and backward iteration is supported over the data.
• Data is automatically compressed using the Snappy compression library.
• External activity (file system operations etc.) is relayed through a virtual interface so users can customize the operating system interactions.
• Detailed documentation about how to use the library is included with the source code.

Boise

For people who are reading this answer, LevelDB doesn't support concurrent access, which is a deal breaker in my case. – Judo

Answer 2

What makes you think BerkDB cannot store binary data? From their docs:

Key and content arguments are objects described by the datum typedef. A datum specifies a string of dsize bytes pointed to by dptr. Arbitrary binary data, as well as normal text strings, are allowed.

Also see their examples:

money = 122.45;
key.data = &money;
key.size = sizeof(float);
...
ret = my_database->put(my_database, NULL, &key, &data, DB_NOOVERWRITE);

Gynaecomastia

Answer 3

If you don't need "multiple writer processes" (only multiple readers works), want something small and want something that is available on nearly every linux, you might want to take a look at gdbm, which is like berkeley db, but much simpler. Also it's possibly not as fast.

In nearly the same area are things like tokyocabinet, qdbm, and the already mentioned leveldb.

Berkeley db and sqlite are ahead of those, because they support multiple writers. berkeley db is a versioning desaster sometimes.

The major pro of gdbm: It's already on every linux, no versioning issues, small.

Trajan

Answer 4

Which OS are you running? For Windows you might want to check out ESE, which is a very robust storage engine which ships as part of the OS. It powers Active Directory, Exchange, DNS and a few other Microsoft server products, and is used by many 3rd party products (RavenDB comes to mind as a good example).

Dissimilitude

have you considered the simplistic "filesystem as a db" option? filename=key, file contents = value... – Dissimilitude

@Addys: I like this system a lot, you should post it as a proper answer – Gynaecomastia

Why do people post answers as comments. 很麻煩！ – Moss

Answer 5

You may have a look at http://www.codeproject.com/Articles/316816/RaptorDB-The-Key-Value-Store-V2 my friend of Databases.

Compensate

Answer 6

Using sqlite is now straightforward with the new functions readfile(X) and writefile(X,Y) which are available since 2014-06. For example to save a blob in the database from a file and extract it again

     CREATE TABLE images(name TEXT, type TEXT, img BLOB);
     INSERT INTO images(name,type,img) VALUES('icon','jpeg',readfile('icon.jpg'));
     SELECT writefile('icon-copy2.jpg',img) FROM images WHERE name='icon';

see https://www.sqlite.org/cli.html "File I/O Functions"

Ahmadahmar