Automatic persistance of node.js objects in database

Question

While experimenting with some data indexing using node.js objects (arrays, maps...) that takes some time to populate (from DB data) at every startup of the script, I wished my node.js objects could be automatically and transparently persisted in my database.

Having used MongoDB from node.js as well as other databases (including SQL-based) for some time now, I'm quite aware that the query/update mechanisms are very different between javascript objects (synchronous access, no queries) and database records (asynchronous access, queries, ...). However, I'm still hoping that a solution to make a javascript var persisted, at least for indices, can exist and be helpful.

Basically, I'm thinking of something like HTML5's LocalStorage, but for node.js servers.

Do you think this idea is interesting, feasible, or maybe it already exists?

EDIT: Work in progress: https://github.com/adrienjoly/persistent-harmony

Septi

looks like the solution of my dreams would rely on Proxies (ES6): #10666392 – Septi

Answer 1

A first thing to make clear, is that databases serve two purposes: persistence and convenient/efficient querying.

If you only need the first because you absolutely know up front that no other program is going to access your persistent data, you could take a look at the literature on orthogonal persistence, which is exactly the concept that you're suggesting here. An example is the KEN protocol that was successfully implemented in the WaterKen Java server. There is some work to integrate the protocol into Google's V8 JavaScript runtime, which could lead to Nodeken, a Node.js with orthogonal persistence.

One of the difficulties of getting orthogonal persistence right is to map transactional semantics to a f.e. object-oriented programming system. The approach taken by V8-ken is to treat a single event loop execution of your JavaScript runtime as a transaction. In other words, the state of the virtual machine is persisted at the end of each "turn" in response of some event (incoming web request, server reply, user interface event, all asynchronous operations (IO), etc.). This however requires a modified runtime such as V8-ken, but evolutions in ECMAScript, such as proxies, look promising to be able to implement such features more conveniently.

In many cases (think web applications) though, persistent data needs to be accessible by different programs, requiring a "real" database for data to be easily exported, migrated, queried, etc. Hybrid approaches could of course be possible...

Vitavitaceous

Thanks for letting me know about "orthogonal persistence" and V8-Ken, Andoni! I believe that this approach is interesting for embedded apps that may be restarted at anytime, so that they can resume back to their previous state instantly, without code overhead. In my case, I don't need to share the data with other applications, but I would like to specify which variables are to be persisted. It's also important that this persistence is as time-efficient as possible. – Septi

Answer 2

%> npm search persistent storage
closet                JSON persistent storage with methods chainability and callbacks for asynchronous use. =ganglio 2013-01-29 18:41  0.0.7  json persistent storag
ewdDOM                Persistent lightweight DOM using Mumps Global Storage         =robtweed        2013-02-02 14:39  0.0.4
fs-persistent-object  Tiny Node library for persisting small runtime objects on filesystem =oleksiyk 2013-04-09 09:13  0.0.1  persistent tiny storage
headstorage           A persistent storage for Node.js                              =headhsu2568     2012-11-20 13:41  0.0.0  storage
level-store           A streaming storage engine based on LevelDB.                  =juliangruber    2013-06-21 19:55  3.3.2  leveldb levelup stream persistent
node-persist          Super-easy (and fast) persistent data structures in Node.js, modeled after HTML5 localStorage =benmonro 2013-04-09 17:33  0.0.1  node persist
persistent-hash-trie  Pure string:val storage, using structural sharing             =hughfdjackson   2013-05-24 19:24  0.4.1  persistent hash trie pure functional d
perstore              Perstore is a cross-platform JavaScript object store interface for mapping persistent objects to various different storage mediums using an in
shelf.js              A modular, powerful wrapper library for persistent objects in the browser and Node.js =shakty 2013-05-24 08:10  0.4.7  persistance localStorag
stay                  Persistent scuttlebutt instances for browser and node         =juliangruber    2012-12-11 21:54  0.1.0  persistent scuttlebutt persistence loc

Looks like the closest match would be node-persist

=)

EDIT: Here may be a better alternative solution...

@adrienjoly You know prototyping is still fairly high level and may not be (in the long run) as efficient as you are thinking.

You may be better off developing a module in C/C++ exposing a high level API for node.js to take advantage of.

I think I have a post about getting your feet wet with this type of node.js development (it stemmed from an original tutorial I followed here)

I do believe that method is however outdated and a newer method is to use the node-gyp tool. Some additional resources and examples: node-gyp projects, uRSA (I have a small pull request with this one), bcrypt etc..

My assumption in this is that you may bind the module extension to a db api such as oracle or postgres etc., and by writing a low level module linking to a low level API while exposing a high level API for developers to implement the persistent configuration options with API calls for slicing, indices, etc the performance would be optimal vs. trying to have node.js interpret your prototyping shim

Didier

thanks, @jas ! i'm going to have a look at all these modules! however, I believe that node-persist does not allow manipulating persisted objects with native js operators ([], =, etc...). this is a turn off... – Septi

here is a recap of the modules you listed: - the following rely on getters/setters, explicit synchronization functions, or use interval to perform periodic comparisons...: closet, headstorage, node-persist, persistent-hash-trie, perstore, shelf.js - the following are for more specific cases (including too small objects): ewdDOM, level-store, stay, fs-persistent-object – Septi

Answer 3

Maybe this is what you're looking for?

https://github.com/yangli1990/flydb

or

npm install flydb

Now in javascript

var flydb = require('flydb');

flydb.test = "hello world";  //flydb.test will now persist

To run this

node --harmony-proxies <your commands>

e.g

node --harmony-proxies app

Hallvard