Want to do an incremental backup for mongodb. Journaling? Oplog?

Question

I want to daily backups for a single mongodb database, probably with mongodump. To not lose any data, I would like this to be incremental so if something goes wrong in the middle of the day, I need to be able to replay changes for that day up til the point of failure after doing a mongorestore.

Am I understanding correctly that I need to use an oplog for this? Or is journaling the answer? I tried doing the following:

1. Turning my mongo database into a replica set of just one, so that it creates an oplog. (This feels pretty hacky)
2. Restarting mongod with the --oplog option
3. Performing changes that should be recorded in the oplog

However nothing ever gets stored in the oplog. What is the best way to do such incremental backups? I'm basically looking for a similar approach to replaying the mysql binlog.

Thanks

Zarah

Answer 1

MongoDB doesn't provide an incremental backup option out of the box, but it's possible to do a file snapshot and replay the oplog. Did you set up your server as a replica set following the steps outlined in the documentation? http://www.mongodb.org/display/DOCS/Replica+Sets+-+Basics

Could you also tell us the purpose of these backups? Have you considered adding a second node to your replica set for data durability?

If you've walked through the steps to set up the server as a member of a replica set, can you run rs.status() in the shell?

One other note (just to clarify)- journaling is not meant to be a backup strategy; journaling merely ensures that the database can return to a consistent state in the event of failover. Running with journaling enabled is highly recommended.

Here is the MongoDB documentation for backups: http://www.mongodb.org/display/DOCS/Backups

Jubal

Thanks for the clarification, I have looked into this more yesterday and I understand what needs to happen. What I want to do is do a daily mongodump, save it to S3, and every hour also save my oplog to S3. I have two follow up questions if you would be so kind: 1. My oplog is stored in the local database and I'm able to access it from the mongo shell now. To backup the oplog, can I just copy all of my local.(1-N) and local.ns files to s3? 2. What's the best way to replay oplogs? Is mongorestore --oplogReplay enough for this? Thanks so much! – Zarah

mongodump --oplog ensures that any oplog entries created while the dump takes place can be replayed with mongorestore --oplogReplay. You cannot use mongorestore --oplogReplay to replay oplog operations that occurred after the dump finished. One solution that you might investigate is to dump the oplog collection every hour and use the applyOps command: docs.mongodb.org/manual/reference/commands/#applyOps – Jubal

This might be hacky .. but I think using mongorestore --oplogReplay actually worked. What I did was mongodumped the oplog, moved the resulting 'oplog.rs.bson' to 'dump/oplog.bson' and ran mongorestore --oplogReplay. Is this dangerous? If it is I'll definitely go with the applyOps technique. – Zarah

Hi Alex, to be perfectly honest, since MongoDB was not designed to replay the oplog in this manner, I cannot confirm whether or not this technique is advisable. The best and safest solution might be to dump the oplog and write a script that applies these operations. You may want to investigate the mongooplog, a tool that will be available in the next stable release of MongoDB (due very soon): docs.mongodb.org/manual/reference/mongooplog – Jubal

Answer 2

The best way to do the backups is to configure Ops Manager as part of your MongoDB infrastructure, but it does so much more than just backups...

Sardis