per user db (pouchdb/couchdb) & shared data - doable?

Question

I have the following use case / application:

a TODO app where users can: CRUD their on TODOs (I am using pouchdb/couchdb syncing). Pretty much based on Josh Morony's tutorial

Now I want to add ability of users to "share" (post only, there is no "edit"/put) their TODO items with other users, who would be able to just view (read) those (no write access etc).

I was thinking about adding a separate DB (let's call it "shared TODOs DB") where my server can write and all users can only read. So any user could potentially do .find() across that read only db, while posting in there will still be governed by a server upon requests to share their TODO coming from users.

Is there a known pattern (approach) for this? Are there any real apps / examples that already do that?

Sulfapyrazine

You can basically setup replications from user database to a central database. Your facade over the central database would filter the data to show only accessible documents (depending on your permissions/sharing model) – Hent

Hey Alexis, you got me intrigued. But if you could elaborate a bit more on facade specifically - how would you achieve a facade in front of replication target db? I am not sure I understand how such facade could be organized as I thought there can be no middle man? – Sulfapyrazine

I checked documentation at Cloudant so I guess you meant filter functions? like configure user's DBs in such a way that not to be shared docs are not part of replication? – Sulfapyrazine

Filtering only the shared docs can be problematic. Let's say you replicate only the documents with the "shared" field at true. If you decide to change the "shared" field to false, it won't be replicated to the central database but it won't be deleted. This is why you should add some mecanism for your central database. – Hent

So tech stack wise I am confused with what is that mechanism? you mean a service (like node.js app) that mimics a couchdb interface and acts as a proxy that passes what is needed over to the actual central db? – Sulfapyrazine

BTW as for the second comment - in the use case I am thinking about we actually need it all to act exactly as you described - a user can't deleted published doc once shared. But I think there might be other concerns like security etc probably, while lot other area to think about – Sulfapyrazine

Answer 1

So for this kind of project, per-user database is the recommended pattern for your databases.

Idea

So basically, you will have :

• One private database per user (with write/read permissions)
• One central database read-only

As for the central database, you need it to be read-only and you also need to allow shared documents only. You need some kind of application proxy for this. You can basically build an API on top of the central database and allow access to shared documents only.

Then, you can setup replications from each user database to the central database and persist de replication in the _replicator database.

couchperuser

I'm not sure that per-user database plugin is working at the moment with the version 2.X.X but you can do it yourself with some sort of application process (Create the user, then create the database, then manage permissions of the new database)

Defilade

Unfortunately, setting up a continuous replication (potentially one in each direction) for N users scales poorly and will be very expensive unless Cloudant has changed their billing structure. Also, it's unclear how you propose getting around the "unsharing" issue you mention in #46351932 – Angeliqueangelis

I totally agree that there is place for improvement. As for the sharing, it's up to the developer to implement the logic of permissions/sharing and handle it through the application layer. – Hent

Yeah, I think for the central DB a more traditional approach could be simplest if you don't need replication. I've found it ends up being simpler a lot of times to treat CouchDB as any other database — i.e. accessing it through logic in application middleware that has "root access" to the databse, rather than trying to use CouchDB's own permissions and replication stuff :-/ – Angeliqueangelis

There is Envoy - proxy in front of CouchDB that allows using single database for all users as if couchperuser was enabled. And it does not seem hard to build sharing on top of it. See this issue github.com/cloudant-labs/envoy/issues/125 – Levenson

You could look up spiegel and see how it monitors the change feed and then sets up one time replications as needed. – Statistics

Answer 2

CouchDB does not offer a good way to do this, but if your backend has a facility for maintaining lots of shared databases (in addition to single-user ones) I think you could pull it off.

The initial impulse is to use continuous filtered replication to/from a central "master" hub, but this leaves you with two problems:

• Replication can't delete documents unless they are deleted! I.e. filtered replication doesn't determine which documents exist in the target, but rather whether a document's changes will propagate to the target. The effect of this is that you can share, but you can't unshare.
• This sort of replication doesn't scale. For N user databases you need to keep in sync, every change to the central database forces all N replications to process that change independently. Now consider that the amount of changes M that happen on the central database will be almost directly proportional to N (i.e. the more users you have the more frequently all those users will have to process changes to the central database!) You could somewhat mitigate this by adding more layers (fanning out from one central hub to semi-shared hub to individual databases), but that also adds practical complications.

How is your sharing organized? What you might be able to do is set up a shared database for each "share" combination.

When user A wants to share document D with user B, "move" the document into a new database AB. Practically this means: copy the contents of D to a new document D' in database AB, and then delete the original D from database A.

How does this sync?

Keep in mind PouchDB clients can replicate to/from more than one source database, so user A will replicate from A and AB, while user B replicates from B and AB. The trick is to use filtered replication in the opposite direction, back to the server databases:

The now-shared document D' should never go to database A or database B, and likewise an unshared document X should never go to database AB. Something like this:

┌───────────────────────┐                                  
│  server:5984/user_a   │                                  
└───┬───────────────▲───┘                                  
    └─┐           ┌─┘    ┌──────────────────────────────┐  
      │           ●──────│ if (doc.shared_with == null) │  
    ┌─▼───────────┴─┐    └──────────────────────────────┘  
    │     local     │                                      
    └──▲──────────┬─┘                                      
     ┌─┘          └─┐      ┌──────────────────────────────┐
     │              ●──────│ if (doc.shared_with == 'ab') │
 ┌───┴──────────────▼────┐ └──────────────────────────────┘
 │ server:5984/shares_ab │                                 
 └───────────────────────┘              

So, assuming the share database(s) are already set up, when the local client wants to share some data it actually adds _deleted:true to the original (unshared) document and creates a new (shared) document. The deleted original propagates to the user_a database on the server and the created copy propagates to the shares_ab.

Unsharing then works pretty much the same: the client adds _deleted:true to the shared document and recreates the data again in a new unshared copy. From B's perspective, the document had showed up once it was in shares_ab and now disappears because it is deleted from shares_ab.

(Instead of user "pairs" you could extend this to ad-hoc sets of users, or simplify it to specific groups that users are already in, or whatnot. The key idea is to create an actually shared database for each unique sharing context needed.)

Angeliqueangelis

Thanks for the detailed explanations :-) – Sensor