Multiple remote databases, single local database (fancy replication)

Question

I have a PouchDB app that manages users.

Users have a local PouchDB instance that replicates with a single CouchDB database. Pretty simple.

This is where things get a bit complicated. I am introducing the concept of "groups" to my design. Groups will be different CouchDB databases but locally, they should be a part of the user database.

I was reading a bit about "fancy replication" in the pouchDB site and this seems to be the solution I am after.

Now, my question is, how do I do it? More specifically, How do I replicate from multiple remote databases into a single local one? Some code examples will be super.

From my diagram below, you will notice that I need to essentially add databases dynamically based on the groups the user is in. A critique of my design will also be appreciated.

Should the flow be something like this:

1. Retrieve all user docs from his/her DB into localUserDB
2. var groupDB = new PouchDB('remote-group-url'); groupDB.replicate.to(localUserDB);
   (any performance issues with multiple pouchdb instances 0_0?)
3. Locally, when the user makes a change related to a specific group, we determine the corresponding database and replicate by doing something like:
   localUserDB.replicate.to(groupDB) (Do I need filtered replication?)

Outsider

have you figured out the role system? – Autoeroticism

Yeah, basically in the _users database I ended up using one role per database. I defined the database names the user has access to in the user's roles array on each user doc. Let me know if you have any more specific questions, I'll be happy to help since I spent a considerable amount of time on this. – Outsider

how did you set up syncing between the user databases? – Autoeroticism

If you're struggling with this, consider using as many local dbs as there are remote ones. As long as you're not live replicating all of them at once (too many HTTP connections), you should be fine. See sync with multiple databases. – Yttria

Answer 1

Replicate from many remote databases to your local one:

remoteDB1.replicate.to(localDB);
remoteDB2.replicate.to(localDB);
remoteDB3.replicate.to(localDB);
// etc.

Then do a filtered replication from your local database to the remote database that is supposed to receive changes:

localDB.replicate.to(remoteDB1, {
  filter: function (doc) {
    return doc.shouldBeReplicated;
  }
});

Why filtered replication? Because your local database contains documents from many sources, and you don't want to replicate everything back to the one remote database.

Why a filter function? Since you are replicating from the local database, there's no performance gain from using design docs, views, etc. Just pass in a filter function; it's simpler. :)

Hope that helps!

Edit: okay, it sounds like the names of the groups that the user belongs to are actually included in the first database, which is what you mean by "iterate over." No, you probably shouldn't do this. :) You are trying to circumvent CouchDB's built-in authentication/privilege system.

Instead you should use CouchDB's built-in roles, apply those roles to the user, and then use a "database per role" scheme to ensure users only have access to their proper group DBs. Users can always query the _users API to see what roles they belong to. Simple!

For more details, read the pouchdb-authentication README.

Kumamoto

Nolan! To ensure I understand correctly, the flow would be something like this: 1) Retrieve all user docs from his/her DB 2) When a document of type group is encountered: var groupDB = new PouchDB('remote-group-url'); (any performance issues here 0_0 ?) groupDB.replicate.to(localUserDB); 3) Locally, when the user makes a change related to a specific group, we determine the corresponding database and replicate by doing something like: localUserDB.replicate.to(groupDB) (a bit confused about when to use filtered replication in this instance) Does my flow makes sense? Thanks man – Outsider

I apologize for the block of text but stack overflow does not seem to allow line breaks in comments. Also, stack overflow apparently does not let you edit your comments after five minutes 0_0 – Outsider

I added my unreadable block comment above to the question itself underneath the image, – Outsider

I don't really understand your question. :( You definitely don't want to iterate over all docs with allDocs() and try to replicate the documents yourself (e.g. using put() or some other manual method). Just use the replicate() API to replicate between databases, with filtering if necessary... – Kumamoto

If you hop on the Freenode PouchDB IRC channel, it may be easier to help. – Kumamoto