Pouch DB : database size

Question

I am storing ~2500 images as attachments in same doc in CouchDB. These images occupy roughly about 15MB in hard drive and resultant CouchDB is roughly 17MB.

When I push this document to my client through PouchDB I saw that resultant database is over 40 MB. I made some tests following these steps:

1. upload X images attachments to CouchDB document.

2. Compact CouchDB

3. Clear client cache fully

4. Reload client (in my app I replicate data on reload).

This is the result:

number of attached files | Total size (KB) in HD | Inc | Size in Indexed DB | Inc
17                         129                           207
27                         168.2                   39.2  267                  60
37                         219.6                   51.4  335                  68
47                         275.5                   55.9  414                  79
57                         327.7                   52.2  493                  79
67                         384.9                   57.2  579                  86
77                         428.5                   43.6  654                  75

So, it seems that:

1. PouchDB adds roughly 2K control data to each attachment.

2. This control data increases when more attachments are added. (1.6K -> 2.3K -> 2.6K -> 2.8K...)

Images have content_type:image/png both in CouchDB and PouchDB. I understood that this should prevent storing them as base64. Am I correct?

Has anyone seen this earlier? Has anyone been able to workaround it? This is a big problem when aiming to fit an app within iOS 50MB space limitation.

EDIT

I went ahead and checked the size of some images in pouchDB vs original files:

1. File 1: orig size = 7.4K / PouchDB size = 10.2K

2. File 2: orig size = 5.1K / PouchDB size = 6.8K

So I think the increase of size when storing attachments in PouchDB does not come from any control data (at least it is not relevant) but from the way the binary file is stored in browser IndexedDB (I am using Chrome for these calculations).

So, is anything additional that needs to be done in order to avoid this increase of binary sizes in PouchDB?

Laborsaving

Answer 1

A very good question, but the answer depends on which adapter you're using, which is not clear from the description.

Edit: just realized you did say Chrome, but I'm keeping the original answer for posterity :)

• In Node.js we use LevelDB via LevelUP, which stores the binary data directly on disk.
• In Safari/iOS we use WebSQL, which stores binary blobs, so again no overhead.
• In everything else we use IndexedDB, which takes in Blob objects at the API level in everything but Chrome, since Chrome doesn't support that yet (issue).

I'm going to guess you're testing in Chrome. So the reason you're seeing bad performance is because we have to store everything in base64 as a workaround (source).

On the bright side, that Chromium bug is pretty active (last comment was 48 hours ago), so presumably the Chrome team is on it and will publish the fix shortly. When they do, PouchDB will automatically detect that blob support is available and start using it.

Update: the Chrome team fixed this, and Blobs are supported in PouchDB as of Chrome v43. :)

Eskill

I am mainly testing with Chrome (good tooling). But also tried with iOS Safari. There I saw same issue, when loading whole database the space dedicated to the web app in iOS was ~43MB instead of 20MB or so it should be. So at least this question goes to IndexedDB and WebSQL. – Laborsaving

Did you compact your client-side database? Compacting on the server only affects the client if done before the initial replication. – Eskill

Another thing you can try in Chrome is to use new PouchDB('dbname', {adapter: 'websql'}) so you can see what the sqlite database looks like. Chrome supports both IndexedDB and WebSQL. – Eskill

Ah, it just occurred to me that in iOS the size might be almost exactly doubled (43MB vs 20MB) because iOS/Safari coerces binary data to UTF-16. So for PNG data, you'll effectively have a bonus \u0000 byte for every byte. I don't know for sure, but the SQLite plugin may behave differently, so you might want to try that. – Eskill

Using websql fixes the issue in Chrome. – Laborsaving

websql is no longer supported by pouch, any other suggestion to correct the size bump? – Paba