Highest Performance Database in Java

Question

I need ideas to implement a (really) high performance in-memory Database/Storage Mechanism in Java. In the range of storing 20,000+ java objects, updated every 5 or so seconds.
Some options I am open to:

Pure JDBC/database combination

JDO

JPA/ORM/database combination

An Object Database

Other Storage Mechanisms

What is my best option? What are your experiences?

EDIT: I also need like to be able to Query these objects

Connivance

Answer 1

You could try something like Prevayler (basically an in-memory cache that handles serialization and backup for you so data persists and is transactionally safe). There are other similar projects. I've used it for a large project, it's safe and extremely fast.

If it's the same set of 20,000 objects, or at least not 20,000 new objects every 5 seconds but lots of changes, you might be better off cacheing the changes and periodically writing the changes in batch mode (jdbc batch updates are much faster than individual row updates). Depends on whether you need each write to be transactionally wrapped, and whether you'll need a record of the change logs or just aggregate changes.

Edit: as other posts have mentioned Prevayler I thought I'd leave a note on what it does: Basically you create a searchable/serializable object (typically a Map of some sort) which is wrapped in a Prevayler instance, which is serialized to disk. Rather than making changes directly to your map, you make changes by sending your Prevayler instance a serializable record of your change (just an object that contains the change instruction). Prevayler's version of a transaction is to write your serialization changes to disk so that in the event of failure it can load the last complete backup and then replay the changes against that. It's safe, although you do have to have enough memory to load all of your data, and it's a fairly old API, so no generic interfaces, unfortunately. But definitely stable and works as advertised.

Leiker

The documentation on the site is not terribly extensive, but the library download contains demos. The use is pretty straightforward. – Leiker

Answer 2

I highly recommend H2. This is a kind of "second generation" version of HSQLDB done by one of the original authors. H2 allows us to unit-test our DAO layer without requiring an actual PostgreSQL database, which is awesome.

There is an active net group and mailing list, and the author Thomas Mueller is very responsive to queries (hah, little pun there.)

Ranique

Differences on reserved SQL words on H2 and Postgres makes it specially problematic. I am referring to keyword : key – Preach

Answer 3

I don't know if it is the fastest option, but I've been very satisfied with H2 whenever I've used it. It's written by the same person who originally wrote Hypersonic (which later became HSQLDB).

Another option that is allegedly very fast is Prevayler.

Auricula

prevayler is not a database: prevayler.org/wiki.jsp?topic=PrevaylerIsNotADatabase – Giraudoux

Depends on your definition of "database". It's definitely not a relational database, but it would appear to meet the minimum defintion of a database. – Auricula

well, i guess that's true. what i meant was that it holds only as much data as you can have in RAM. – Giraudoux

Answer 4

It is a bit of an old question, but these days there is a whole lot of databases that have a level of performance of 20,000/s. Which database to chose depends on data structure and type of queries you'd like to be making. It also depends on overall volume.

We had similar problem with large volume of time series data, about 300,000 rec/s and we ended up writing a new database, with simple enough API and decent performance. It can do about 2,000,000 object writes/s and we did away without ORM.

It later evolved into QuestDB.

Inconvertible

Answer 5

Try the following, it performs really well with Hibernate and other ORM frameworks

http://hsqldb.org/

Braga

Answer 6

Chronicle Map is an embeddable pure Java persistent database, providing a simple java.util.Map interface. It withstands about 1 million queries/updates per second from a single thread, consistent read/write performance and scales almost linearly to the number of cores in the machine.

Here are some recent performance research with actual numbers:

• Comparison of Jetbrains Xodus, Oracle Berkeley DB JE BTree, MapDB TreeMap, Chronicle Map and H2 MVStore Map
• LmdbJava Benchmarks

Odlo

Answer 7

I would give a try to OrientDB.

Bittner

Answer 8

Terracotta might also be an answer for you. It allows multiple VMs to share objects so you can distribute load etc...

Yoga

Answer 9

You can also check out db4o

Hermetic

Answer 10

hsqldb is quite fast, but it is not ACID transaction-safe. The fastest java-database I know is db4o: benchmarks.

Edit: Please notice that Prevayler is not a database, see http://www.prevayler.org/wiki.jsp?topic=PrevaylerIsNotADatabase. If you're out of RAM, you're out of luck.

Giraudoux

Answer 11

Berkeley DB for Java is a fast in memory database, extremely useful for simple object graphs.

Capsicum

Hmmh. BDB is NOT an in-memory database, but disk-backed one. It can obviously cache much/most of its content in memory, like all production-ready DBs, but I don't know if it even has a memory-only mode. – Backache

Answer 12

H2 is truly fantastic, indeed, in memory, normal server and transactional, you have it all. However It doesn't compare in performance to the object databases, I see Db4o mentioned, I have had much better performance with Neodatis in fact, and everything nicely set up in Maven repositories. Although not very robust, like a Ferrari, fast but not a truck like Oracle.

Vendible

Answer 13

If you want to store all of your data in memory, you might want to look at Prevayler.

I've never used it myself, but it seems like a much better solution than using a relational database for those cases in which all of your data can be stored in memory.

Och

Answer 14

You can try CSQL (available under open source and enterprise version) It provides 30X performance improvement over disk based database systems and provides JDBC interface. It can be configured to work as stand alone main memory database or as a transparent cache to MySQL, Postgres, Oracle databases.

Taffy