java embedded library on-disk key-value database [closed]

Question

What I think I'm looking for is a no-SQL, library-embedded, on disk (ie not in-memory) database, thats accessible from java (and preferably runs inside my instance of the JVM). That's not really much of a database, and I'm tempted to roll-my-own. Basically I'm looking for the "should we keep this in memory or put it on disk" portion of a database.

Our model has grown to several gigabytes. Right now this is all done in memory, meaning we're pushing the JVM for upward of several gigabytes. It's currently all stored in a flat XML file, serialized and deserialized with xstream and compressed with Java'a built in gzip libraries. That's worked well when our model stays under 100MB, but now that its larger than that its becoming a problem.

loosely speaking that model can be broken down as

• Project
  1. configuration component (directed-acyclic-graph), not at all database friendly
  2. a list of a dozen "experiment" structures
     • each containing a list of about a dozen "run-model" structures.
       1. each run-model contains hundreds of megs of data. Once written they are never edited.

What I'd like to do is have something that conforms to a map interface, of guid -> run-model. This mini-database would keep a flat table of these objects. On our experiment model, we would replace the list of run-models with a list of guids, and add, at the application layer, a get call to this map, which would pull it off the disk and into memory.

That means we can keep configuration of our program in XML (which I'm very happy with) and keep a table of the big data in a DBMS that will keep us from consuming multi-GB of memory. On program start and exit I could then load and unload the two portions of our model (the config section in XML, and the run-models in the database format) from an archiving format.

I'm sort've feeling gung-ho about this, and think that I could probably implement it with some of X-Stream's XML inspection strategies and a custom map implementation, but something a voice in the back of my head is telling me I should find a library to do it instead.

Should I roll my own or is there a database that's small enough to fit this bill?

Thanks guys,

-Geoff

Lodhia

im developing an nosql database its jsndb (sourceforge.net/projects/jsdbase). "its in alpha stage" and it is not key/value, but you can get objects by key. give it a try – Almagest

Thanks for asking this question despite it being closed. I too am using plaintext key-value stores and currently rolling my own but haven't figured out a good way to do joins. – Dimitri

Answer 1

http://www.mapdb.org/

Also take a look at this question: Alternative to BerkeleyDB?

Powwow

Thanks a bunch for MapDB. I was thinking about deploying against BerkeleyDB, but it looked like that would require more configuration than I care to do. As it is, I can boot-up, write to, read from, and close a MapDB in about 50 lines of code, which is fantastic. I am a little worried about MapDB's current state of release, but it seems stable enough for our purposes. – Lodhia

Unfortunately MapDB is neither stable nor mature. See changelog, and confirmed bugs that are closed without resolution, at least for Version 1. I truly hope MapDB will be a viable candidate in 2 years. – Istic

Since that last comment we've released version 1.0 of our software on version 1.0 of mapDB, and I've spoken with Jan Kotek, the author of MapDB. We've run into no issues directly caused by MapDB, though there are a number of confirmed bugs. I would encourage anybody reading this to take a look at it. – Lodhia

Answer 2

Since MapDB is a possible solution for your problem, Chronicle Map is also worth consideration. It's an embeddable Java key-value store, optionally persistent, offering a very similar programming model to MapDB: it also via the vanilla java.util.Map interface and transparent serialization of keys and values.

The major difference is that according to third-party benchmarks, Chronicle Map is times faster than MapDB.

Regarding stability, no bugs were reported about the Chronicle Map data storage for months now, while it is in active use in many projects.

Disclaimer: I'm the developer of Chronicle Map.

Screech

Why is the Chronicle Map is pointing to XODUS? – Blessed

@Blessed because Xodus hosts benchmark comparison – Screech

github.com/JetBrains/xodus/tree/master/benchmarks – Copybook