Maximum number of live replications in PouchDB
Asked Answered
C

3

6

I am working on an Ionic app with PouchDB that needs to sync tables with remote CouchDB server. In the constructor of my database.ts provider I have 6 methods:

this.initialiseDB_Ord();
this.initialiseDB_IngProd();
this.initialiseDB_Ing();
this.initialiseDB_Prod();
this.initialiseDB_OrdProd();
this.initialiseDB_Ut();

Each of these methods does the following (I pick the first one as example):

this._DB_Ord = new PouchDB('orders');
this._remoteDB_Ord = this.addressIP + '/orders';
this._syncOpts_Ord = {  live : true,
                        retry : true,
                        continuous : true};
this._DB_Ord.sync(this._remoteDB_Ord, this._syncOpts_Ord)
.on('change',(info) => {
     console.log('Handling syncing change');
     console.dir(info);
}).on('paused',(info)=> {
     console.log('Handling syncing pause');
     console.dir(info);
}).on('active', (info) => {
     console.log('Handling syncing resumption');
     console.dir(info);
}).on('denied', (err) =>{
     console.log('Handling syncing denied');
     console.dir(err);
}).on('complete', (info) =>{
     console.log('Handling syncing complete');
     console.dir(info);
}).on('error', (err)=>{
     console.log('Handling syncing error');
     console.dir(err);
});

then I have a handleSyncing method as follows:

handleSyncingUt() {
  this._DB_Ut.changes({
       since             : 'now',
       live              : true,
       include_docs      : true,
       attachments   : true
  })
  .on('change', (change) =>
  {
     console.log('Handling change');
     console.dir(change);
  })
  .on('complete', (info) =>
  {
     console.log('Changes complete');
     console.dir(info);
  })
  .on('error',  (err) =>
  {
     console.log('Changes error');
     console.log(err);
  });
}

If I have at maximum 5 Databases it works fine. When the sixth DB is added it doesn't synchronize local pouchDB and remote couchDB in real time but only when the app is first opened.

Can someone help me?

Corody answered 4/1, 2018 at 16:31 Comment(2)
What specifically happens when you try to start your 6th live replication? What error or other indication of failure do you see?Proletariat
I don't see any error. If I refresh Fauxton page on my browser it is not updated with last item I posted. With 5 DBs it worksCorody
S
6

@lossleader is right on the money about the max number of connections in a browser/WebView. At Quizster, we have this same problem as Quizster has to sync as many as 10 PouchDB instances simultaneously. This is done using Howler to subscribe to changes for a set of databases. Upon a change to one of these databases, Quizster issues a one-time (not live) sync between the PouchDB instance and the CouchDB cluster.

For a little more background: Quizster has to sync this many PouchDB instances as:

  1. Doing this avoids having to replicate even more data between databases, which leads to lower latency with shared data
  2. Quizster uses a lot of database views to speed up replication and keep the size of the data sets small. And, each view effectively leads to its own PouchDB instance.

I'm planning on open sourcing more of the Quizster stack soon and hope to release some tutorials as well.

Symbolism answered 11/1, 2018 at 17:37 Comment(1)
I figured out some days ago, but I am syncing, when needed, not live.Corody
T
2

Browsers have a maximum number of sockets per domain and per page, and you are exceeding it with live:true connections. Chrome shares the first limit across tabs so you can verify multiple tabs of fewer connections cause the problem in chrome, or by raising the limit in firefox's about:config.

You can't really fix this limit for normal users, and need to adjust how you manage connections (for example a worker can manage one db's sync for many tabs in chrome) and a data design might be changed to observing only one "changes" db constantly to know when to run live:false passes on other dbs.

Tetrachloride answered 4/1, 2018 at 18:34 Comment(2)
So is this a browser problem? I thought that the couchDB server exists even if the browser with Fauxton is closed, or am I wrong?Corody
I still don't understand why you talk about "browsers". I have my mobile app, in my smartphone, and a couch DB in a VPS. There is no Chrome or Mozilla browserCorody
M
0

This 5 connection limitation is expected since you are using Ionic (which uses a browser/WebView to run your app), and this is a pretty crazy limitation right? But don't worry, this limit can be circumvented without any changes in your current code.

Let me first explain why this limit exist and then how you can solve it.

Each PouchDB continuous synchronization you open against a CouchDB server count as a separate HTTP connection. Since CouchDB's integrated web server uses HTTP 1.1 protocol (which is kind of old), you are bizarrely limited by 2 connections by the RFC 2616 or 6 connections (or more, depending on the browser/WebView version).

Check this table for the HTTP 1.1 concurrent connection limits against the same domain:

IE 6 and 7:      2
IE 8:            6
IE 9:            6
IE 10:           8
IE 11:           8
Firefox 2:       2
Firefox 3:       6
Firefox 4 to 46: 6
Opera 9.63:      4
Opera 10:        8
Opera 11 and 12: 6
Chrome 1 and 2:  6
Chrome 3:        4
Chrome 4 to 23:  6
Safari 3 and 4:  4

If you want to dive deeper in the HTTP 1.1 limitation reasons, check this article.

The solution to this problem lies in the HTTP/2, a newer version of the HTTP family protocol. HTTP/2 solves the concurrent HTTP transactions limit by multiplexing them over a single TCP connection.

Since you cannot make CouchDB server speak HTTP/2 by itself, you can place a reverse proxy (HAProxy or NGINX) or API gateway (OpenResty, Kong, Spring Cloud Gateway, Netflix Zuul) in front of it and configure the proxy to serve HTTP/2 endpoints. In a nutshell we have:

IonicApp + PouchDB <--HTTP/2--> Reverse Proxy <--HTTP 1.1--> CouchDB

Configuration sample for HAProxy [1]

Configuration sample for NGINX [1]

Mesocratic answered 20/12, 2021 at 22:15 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.