How well does UnQLite perform? How does it compare to SQLite (in performance)?

Question

I've researched on what I can about SQLite and UnQLite but there are still a few things that haven't quite been answered yet. UnQLite appears to have been released within the past few years which would attribute to the lack of benchmarks. "Performance" (read/write speed, querying, avg. database size before significant slowdown, etc.) comparisons may be somewhat apples-to-oranges here.

From all that I have seen the two have very few differences comparatively speaking, namely that SQLite is a relational database whereas UnQLite is a key-value pair and document (via Jx9) database. They're both portable, cross-platform, and 32/64-bit friendly, and can have single-write and multi-read connections. Very little can be found on UnQLite benchmarks while SQLite has quite a few with different implementations across various (scripting) languages. SQLite has some varied performance across in-memory databases, indexed data, and read/write modes with varying data size. Overall SQLite appears quick and reliable.

All that I can find on UnQLite are unreliable and confusing. I cannot seem to find anything helpful. What read/writes speeds does UnQLite seem to peak at? What languages are (not) recommended when using UnQLite? What are some known disadvantages and bugs?

---

If it helps at all to explain my intrigue, I'm developing a network utility that will be reading and processing packets with hot-swapping between network interfaces. Since the connections can, though unlikely, reach speeds up to 1 Gbps there will be a lot of raw data being written out to a database. It's still in the early stages of development and I'm having to find a way to balance out performance. There are a lot of factors such as missed packets, how large each write size is, how quickly it can process and move data, how much organization will be required, how many tables will be needed, if I can implement multiprocessing, how reliant each database is on HDD speeds, etc. etc.. My data will need tables but whether or not I have to store them as relational is still in the air. Seeing how the two stack up with their own pros and cons (aside from the usual KVP vs Relational debate) may push me towards either one or, if I'm crazy enough, a mix of both

Kookaburra

You should probably look at BerkeleyDB, too. Is it really that hard to just install it and do your own benchmarks? – Abeyant

@MikeSherrill'CatRecall' I was surprised to see someone respond to this given how "old" it is now. I'll look into BerkeleyDB this week. Thanks for the suggestion! – Kookaburra

BDB is a good suggestion. Kyotocabinet is also a nice key/value store with great performance. – Measureless

Although not directly addressing your question, I would question the need to write directly into a DB - would consider dumping the data into a series of files / directories (perhaps spread over multiple drives if bandwidth is an issue), and then post-processing into a DB as required. – Cress

Answer 1

I've done a bit of fooling around with UnQLite using python bindings I wrote. The Python bindings use cython and are quite fast.

What I've found from my experimentation is that UnQLite's key/value APIs are pretty damn fast, comparable to other DBMs. Things slow down a bit when you start using Jx9 and the document store, though.

Basically depends on what you need...

If you want SQL and ad-hoc querying, I'd suggest using SQLite. It is plenty fast and quite flexible.

If you want just keys and values, I'd use something like leveldb or rocksdb.

If you want a lightweight JSON document store, or key/value with a bit "extra", then UnQLite may be a good fit.

Measureless

Thanks for the response. A lot of requests have gone through on the software that I've been asked to develop since I posted this a month ago. SQLite won't cut it with how large the DB will get and how much information needs to be "mapped," alongside the range of what kind of data to store. I'll be looking into databases again soon so I'll check it out again – Kookaburra

SQLite limitation is not really the data size (up to dozens of GB) or the complexity (it's more capable than most other DBMS there), the main limitation is the number of simultaneous write operations (only one in SQLite). – Drislane

One point not mentioned above, is suitability for the platform in use. I have successfully used SQLite (and so I imagine UnQLite would be similar) in an embedded system with small (code and data) memory footprint, whereas I imagine eg RocksDB would be happier running on a server, given its background. – Cress

@Measureless What would you recommend for a fast k-v for use in a portable Python app in 2017? LevelDB/sophia don't support Windows so I feel they're less relevant, but LMDB and UnQLite do, and their Python packages seem quite complete. Would be happy to hear your current opinion. – Addiction