Alternative to BerkeleyDB? [closed]

Question

I'm looking for a dbm-like library that I can use in place of Berkeley DB, which I'm currently using. My main reason for switching is the licensing fees for BDB are pretty high (free for open source apps, but my employer does not want to open source this particular app for various reasons).

I've looked briefly at qdbm but it doesn't look like it will fill my needs -- lots of keys (several million) and large data items (> 1-5 megabytes). Before I continue my search I figured I'd ask because it seems there are tons of dbm-like libraries out there.

Ia

Answer 1

C/C++

• http://sphia.org/ a Key/Value Store, Written in C and BSD-Licensed.
• https://code.google.com/p/leveldb/ Written in C++, also BSD-Licensed
• http://luxio.sourceforge.net/ Written in C++, LGPL, faster than Berkeley DB
• http://symas.com/mdb/ Written in C, License is OpenLDAP Public License (BSD-Like)
• https://github.com/davidar/sdbm Written in C, Public Domain, Reimplementation of "ndbm"
• http://repetae.net/computer/vsdb/ Written in C, License unknown, Experimental
• https://github.com/stevedekorte/skipdb Written in C, BSD-Licensed, Key-value Store using Skip-List indexes
• http://unqlite.org/ written in C/C++, Embedded NoSQL Database Engine, BSD-Licensed, Similar to SQLite.
• http://hamsterdb.com/ Written in C/C++, Embedded NoSQL Database Engine, Apache License 2.0.

Java

• https://github.com/dain/leveldb A Java port of LevelDB, Apache License 2.0
• https://github.com/jingwei/krati A former LinkedIn Project, Apache License 2.0
• http://www.mapdb.org/ An Embedded Java Database, Apache License 2.0, supports Key-Value Storage, very efficient Object serialisation
• https://github.com/OpenHFT/Chronicle-Map, LGPL v3, embedded pure Java key-value store, faster than Berkeley DB
• https://github.com/jetbrains/xodus, Apache License 2.0, a Java transactional schema-less embedded database, faster than Berkeley DB
• https://github.com/lmdbjava/lmdbjava, JNI binding for LMDB, Apache License 2.0

Bakery

Answer 2

You could look at Tokyo Cabinet. Its the successor to qdbm/gdbm, and if you decide to scale has a nice network front-end available.

Edit:

Another variant is Kyoto Cabinet; developed by the same person, but supposedly easier to use.

Midmost

Answer 3

SQLite is public domain, meaning you can use it for any purpose whatsoever, and is widely used and supported.

Janot

Good idea, but SQLite apparently isn't recommended for large databases. Ours will be about 50GB; sorry, that probably wasn't clear in the original post. – Ia

The other issue is that BDB is not an SQL DBMS, it's a storage engine. So unless he has written an SQL implementation on top of BDB, it seems there will be a bit more work to port to SQL rather than another storage engine. – Modestomodesty

@Ferruccio, he doesn't have to port to SQL, he can simply implement BDB-like API on top of SQL. easily. – Nylons

sqlite.org/whentouse.html suggestst "a few dozen GB" is fine. You'd want the filesystem to be have at least enough space for a second complete copy of the database, for journals. I've used sqlite on the 2-5GB range without problems; it performs acceptably. – Duodenum

Berkeley DB now supports SQL as well as the key/value API. In fact it uses the SQLite code on top of the Berkeley DB B-Tree so you can now run TB-sized SQLite databases not just in cases where "a few dozen GB" is fine. (Disclaimer: I am a product manager for Berkeley DB at Oracle.) – Marden

Seems like SQLite is claimed to support TB sized databases - sqlite.org/features.html – Densify

I've found that Berkeley DB locking is unreliable compared to SQLite. – Assemblage

@Greg Hewgill Why there is no official tested thing to embed sqlite in java application. Can you throw some light on this issue. It would be of great help. – Loxodrome

Answer 4

You can get much improved performance out of any dbm (even qdbm) and improved parallelism with a simple level of indirection: Simply take your keys and hash them, and use data_dir/H(key)/ as the database to store those keys. Limit the hash output to a small value (say) 255 for best results.

This approach has a number of benefits, easily summarized:

• Conceptually simple
• Easy to implement and test
• Doesn't lock the entire database for updates
• Can support much larger databases
• Easy to replace the DBM component

The hash probably doesn't even need to be cryptographically secure; just mostly uniform. DJB's cdb hash works well in most cases.

Sylviesylvite

Interesting suggestion, but it's a bit orthogonal to the question. The merits of using a hash table vs. a sorted btree-like structure are a different topic. – Oolite

Answer 5

If you are on Windows then you can use the builtin esent database engine. http://blogs.msdn.com/windowssdk/archive/2008/10/23/esent-extensible-storage-engine-api-in-the-windows-sdk.aspx

Schroer

Answer 6

You could try JDBM. It is a free (Apache 2) key-value store with disk persistence. Simple API and high performance

Brunhilda

Answer 7

Postgres or HSQLDB and possible even H2 database

Brandibrandice

Answer 8

db4o is pretty cheap and fast but it can only be used with java or .net

Bede

db4o is absolutely inappropriate alternative to BerkeleyDB. additionally db4o doesn't offer royalty free licensing model. They're charging for each deployment. – Nylons

Answer 9

Firebird is your best friend.

Bistort

Firebird is a very nice SQL database. Not great as a storage DB, but probably better than most due to very good BLOB support. – Crony

Can you elaborate a little more on 'not great as a storage DB'? Also, I would like to hear more on about what Firebird is NOT great (price? size? :) ) – Bistort