General-purpose databases that never delete or update data in-place [closed]

Question

I'm very much inspired by the approach to data management advocated by Rich Hickey, and implemented in Datomic, where the data is never mutated in-place, all the versions are always preserved and query-able, and the time is a first-class concept.

Of course, there are specialized databases matching that description, like Git, or any other source control system. The question is if there are any (more or less) general-purpose DBMS-es of relational, graph, hierarchical, document or any other flavor that can be effectively used in, say, an eCommerce Web application. Or is Datomic the only choice then?

Insipience

I think both the BerkeleyDB Java Edition and CouchDB work like that internally. But in both cases, there are "space reclaim" processes that purge old data and I am not sure if the history is really exposed as a first-class concept (as opposed to "just" being used to make transaction isolation work). – Antabuse

That's right. I'm using CouchDB right now. The views' map-reduce functions can't access the old versions. – Insipience

Also there is Git Ketch which is a multi-master Git management system that replicates information across multiple Git servers for resilience and scalability., add here git extensions for large binary files - and get some storage suitable for some types of applications. – Inosculate

Apache HBase does not mutate data in place and previous versions queryable. – Inosculate

I think Google Spanner is such database, i.e. old versions of data are subject to configurable garbage-collection poli- cies; and applications can read data at old timestamps. and F1 maintains a logical history log of all changes, which is written into Spanner itself as part of every transaction. F1 takes full snapshots of data at a timestamp to initialize its data structures, and then reads incremental changes to update them.. Its spinoff CockroachDB may have same characteristics. – Inosculate

Noms is versioned, forkable, syncable, append-only database. It is possible to see the entire history of the database – Inosculate

There's also Bigtable – Rotative

LiteTree SQLite with Branches – Inosculate

github.com/sirixdb/sirix – Inosculate

juxt.pro/crux – Insipience

Answer 1

There is an approach to designing systems with an idea of never deleting or mutating data called Event Sourcing. Basically, the idea is to store events (or facts) that change the system state, instead of snapshots of the state. The history of events can be replayed later on to produce a certain purpose-specific projection of what the state at any point in time looked like. Multiple projections built for different purposes can coexist in the system. More information can found on the following web sites:

• http://martinfowler.com/eaaDev/EventSourcing.html
• http://codebetter.com/gregyoung/2010/02/20/why-use-event-sourcing/

It's in line with what you are describing, but rather than being just a database model, Event Sourcing and Command Query Responsibility Segregation (CQRS) prescribe a special way of designing the whole system including the database and business logic layers.

There are a few frameworks that follow this approach, such as:

• http://www.axonframework.org/
• http://qi4j.org/
• http://en.jdon.com/

While this does not directly answer your question, it may provide a different perspective on the problem.

Tundra

No worries, I'm glad it's useful! – Tundra

Another great article I've found myself is this: nathanmarz.com/blog/how-to-beat-the-cap-theorem.html – Describes a general strategy for storing and querying immutable facts, splitting the data storage into two layers: batch and realtime. Comments are also quite interesting. The author is even writing a book on this topic now: manning.com/marz – Insipience

Are these still relational? In heaven I think there's some sort of prolog/sql aka rule-based/relational, immutable database in JavaScript that can run on the client and server (like PouchDB & CouchDB or Meteor). I send it a transaction, and get a callback on successful (consistency) or collision (simultaneous writes) -- and it does JOINs! But, unfortunately, only in heaven... :P It's a lot to ask for, I know. – Changteh

There is very good description of event sourcing. – Inosculate

Event store is open-source, functional database with Complex Event Processing in JavaScript. if needed idea realized for JavaScript. – Inosculate

Answer 2

Irmin is a distributed database that follows the same design principles as Git.

Inosculate