How to perform one-time DB sync to another DB in MongoDB?

Question

I have separate development and production MongoDB servers and I want to keep actual data in development server for sometime. What I should use for it: mongodump, mongoimport or something else?

Clarification: I want to copy data from production to development.

Ericaericaceous

Answer 1

You can use the db.copyDatabase(...) or db.cloneDatabase(...) commands:

http://www.mongodb.org/display/DOCS/Copy+Database+Commands

This is faster than mongodump / mongorestore because it skips creating the bson representation on disk.

Lust

mongoDB 4.2: Starting in version 4.2, MongoDB removes the deprecated copydb command and clone command. docs.mongodb.com/manual/release-notes/4.2-compatibility/… – Genethlialogy

Answer 2

If it's a one time-thing

and you want fine control over parameters such as which collections to sync, you should use:

• mongodump to dump bson files of your Production DB to your local machine
• mongorestore to then, retrieve the dumped BSON files in your Local DB

---

Otherwise you should check out mongo-sync

It's a script I wrote for my self when I had to constantly copy my Local MongoDB database to and from my Production DB for a Project (I know it's stupid).

Once you put your DB details in config.yml, you can start syncing using two simple commands:

./mongo-sync push       # Push DB to Remote
./mongo-sync pull       # Pull DB to Local

If you use it inside some project, it's a good idea to add config.yml to .gitignore

Chan

Out of interest why do you say it's stupid? I'm looking at how to handle this situation before I start a project - is there a better way to handle the development/production side of a mongodb? – Outdated

It's stupid because its not a good practice to test/develop on a production database. I wrote that script (in two different languages btw) because that was the requirement of the project and the client; that the data be entered into the db manually by the dev and to be only readable by the users. – Chan

Is that simply due to the inevitable growth and complexity a production database brings, or is there an angle I'm not considering? I'm actually more thinking the other way round though, I'll be developing locally but will want the data to go up with the site (rather than populate the data once in place) – Outdated

That depends on the project requirements I guess, if its an app with users it's usually a good idea to keep dev and prod dbs separate. But if its something like a blog, you can simply populate the db in dev and later push it to prod. – Chan

Very true - my project doesn't involve user generated content, but makes perfect sense that you'd want to maintain a separate dataset there. I'll check out your script though for sure, looks useful! – Outdated

Replication of data from prod to dev makes total sense. If a bug occurs in production and you can't replicate it locally, it's wise to copy the prod data to your dev environment so you have an exact copy of that environment. – Menadione

Answer 3

When you want the dev database to look exactly like the production database, you can just copy the files. I am currently running a setup where I synchronize my MongoDB database between my desktop and my notebook with dropbox - even that works flawless.

Krems

Where is this database files are located in file system? – Ericaericaceous

The default is /data/db relative to the mongod binary. But you usually set the location of the database files with the --dbpath parameter when you start MongoDB. – Krems

Interesting idea. Does this even work if two developers work simultaneously on the same db synchronized over dropbox? Gonna develop an application with a friend over Internet. I thought of having one Master in the web and two local slaves(/masters) for me and my body. However this seems not to work since mongo cannot do a master/master sync or? What happens if i go offline and continue working? My slave becomes master on my local instance. What happens if I go back online? It is not like couch db i guess from reading docs. What happens if you put mongo into dropbox and run it simultaneous? – Dilan

Using dropbox with two people doesn't sound like a good idea. In my setup I was never using both devices simultaneously. – Krems

Answer 4

Fix Mongosync Atlas to Atlas documentation examples to avoid confusion

Description

The documentation for Atlas to Atlas migration via mongosync has confusing examples (generic for on-prem servers) and a wrong example for SRV connection. Documentation

• First example:

cluster0:
mongodb://clusterAdmin:[email protected]:20020,clusterOne02.fancyCorp.com:20020,clusterOne03.fancyCorp.com:20020
cluster1:
mongodb://clusterAdmin:[email protected]:20020,clusterTwo02.fancyCorp.com:20020,clusterTwo03.fancyCorp.com:20020 

• Correction proposal: We should use cluster1Name.ab123.mongodb.net:27017 and cluster2Name.ab123.mongodb.net:27017 as generic name for an Atlas server name for each case and ensure we replace 27017 as default port instead of 20020 for all the examples on the linked documentation.

Using a generic server and a port different from the Atlas default connection port may cause confusion at the time to connect.

• SRV final example:

 mongosync \
   --cluster0 'mongodb+srv://clusterAdmin:[email protected]:20020/' \
   --cluster1 'mongodb+srv://clusterAdmin:[email protected]:20020/'

Correction proposal:

This command is incorrect, the SRV connection string doesn't includes the port in Atlas. The correct command should be:

mongosync \    
--cluster0 'mongodb+srv://clusterAdmin:[email protected]/'\   
--cluster1 'mongodb+srv://clusterAdmin:[email protected]/' 

commands

mongosync --config mongosync.conf

 curl localhost:27182/api/v1/progress -XGET

 curl localhost:27182/api/v1/start -XPOST --data '{"source": "cluster0", "destiantion": "cluster1"}'

Note: run the first command in the background or use another terminal.

Let me know if you need any further clarifications on this.

Form more information: link here

Official Documentation click link

Chemaram