Store mongodb data in compressed format

Question

I am using mongodb to store raw HTML data of web pages using scrapy framework. In one day of web scraping 25GB disk space is filled up. Is there a way to store raw data in compressed format.

Gearalt

Answer 1

There's nothing built in for compression. Some operating systems offer disk/file compression, but if you want more control, I'd suggest you compress it using a library for whatever programming language you're using and manually control the compression.

For example, NodeJs offers simple convenience methods for this: http://nodejs.org/api/zlib.html#zlib_examples

3.0 Update

If you choose to switch to the new storage engine WiredTiger which ships with 3.0, you can choose between several types of compression as documented here. Of course, you'll want to test this change in production workloads to find if the additional CPU utilization is worth the compression received.

Apollonius

#8507397 – Apollonius

@binit No: jira.mongodb.org/browse/… – Guidebook

@binit -Why wouldn't you want to compress the data with Python? – Apollonius

there is no facility in mongodb as per today. the recommended way is to compress it client side and write it as a binary field. f.ex use somethink like gzip to compress it before writing it. – Nathalie

You can try looking at TokuMX. They do compression on mongodb dataset and also replace the BTree storage engine with a (seemingly) better fractal tree datastructure. tokutek.com/products/tokumx-for-mongodb – Carrnan

Answer 2

Starting with 2.8 version of Mongo, you can use compression. You will have 3 levels of compression with WiredTiger engine, mmap (which is default in 2.6 does not provide compression):

• None
• snappy (by default)
• zlib

Here is an example of how much space will you be able to save for 16 GB of data:

data is taken from this article.

Precautious

Answer 3

You can store your string like this to compress it: myhtml.encode('zlib')

Carpometacarpus

myhtml.encode('zlib') doest always generate unicode, which causes problems when inserting into mongoDB. – Rehearse