How can I save multiple documents concurrently in Mongoose/Node.js?

Question

At the moment I use save to add a single document. Suppose I have an array of documents that I wish to store as single objects. Is there a way of adding them all with a single function call and then getting a single callback when it is done? I could add all the documents individually but managing the callbacks to work out when everything is done would be problematic.

Indigestible

You need to control the code flow use some kind of async library like async. (there is parallel function and when completed the callback is called) – Shelia

groups.google.com/forum/#!topic/mongoose-orm/IkPmvcd0kds – Guimond

Answer 1

Mongoose doesn't have bulk inserts implemented yet (see issue #723).

Since you know the number of documents you're saving, you could write something like this:

var total = docArray.length
  , result = []
;

function saveAll(){
  var doc = docArray.pop();

  doc.save(function(err, saved){
    if (err) throw err;//handle error

    result.push(saved[0]);

    if (--total) saveAll();
    else // all saved here
  })
}

saveAll();

This, of course, is a stop-gap solution and I would recommend using some kind of flow-control library (I use q and it's awesome).

Telegram

Can you provide the solution using q please? – Lactary

I don't think this is "concurrent". Each save is not invoked until the previous one has completed. – Champaigne

True. A more concurrent approach would be, for example, to fire off all saves, wait for all to call their callback and return an array of results. You could use async for that, or some promise interface. – Telegram

when will this condition if (--total)be false? – Habitable

I think the above answer is very old. There is a method named insertMany() in mongoose. Check mongoosejs.com/docs/api.html#model_Model.insertMany – Marje

Answer 2

Mongoose does now support passing multiple document structures to Model.create. To quote their API example, it supports being passed either an array or a varargs list of objects with a callback at the end:

Candy.create({ type: 'jelly bean' }, { type: 'snickers' }, function (err, jellybean, snickers) {
    if (err) // ...
});

Or

var array = [{ type: 'jelly bean' }, { type: 'snickers' }];
Candy.create(array, function (err, jellybean, snickers) {
    if (err) // ...
});

Edit: As many have noted, this does not perform a true bulk insert - it simply hides the complexity of calling save multiple times yourself. There are answers and comments below explaining how to use the actual Mongo driver to achieve a bulk insert in the interest of performance.

Silverstein

Note: This is not a BULK insert - the underlying mongoose implementation does loops through all of the elements and commits them one by one. – Date

^ that's very much relevant, as it might seriously impact performance for those who use it intensively. – Ear

Response for Aaron Heckman 2011: not really. Model.create(doc1 [, docN], callback) sort of helps here but its still calling model.save for you on each one. If by "faster" you mean "bypass all mongoose hooks and validation" then you could drop down to the native driver and use it directly: Movie.collection.insert(docs, options, callback) github.com/christkv/node-mongodb-native/blob/master/lib/mongodb/… – Guimond

In light of all the comments and new answers on this I've amended mine to point out the objections around bulk insert and performance. However, I feel compelled to note that Hoa never mentioned performance as the reason for asking this question - they simply wanted to avoid waiting for multiple callbacks to succeed. – Silverstein

I want to emphasize that this is NOT the best way to do bulk inserts if you're dealing with a large amount of documents. See https://mcmap.net/q/167642/-mongoose-mongodb-batch-insert which contains a better explanation. – Zeke

(categories = []) => Categories.create(...categories) – Searcy

Answer 3

Mongoose 4.4 added a method called insertMany

Shortcut for validating an array of documents and inserting them into MongoDB if they're all valid. This function is faster than .create() because it only sends one operation to the server, rather than one for each document.

Quoting vkarpov15 from issue #723:

The tradeoffs are that insertMany() doesn't trigger pre-save hooks, but it should have better performance because it only makes 1 round-trip to the database rather than 1 for each document.

The method's signature is identical to create:

Model.insertMany([ ... ], (err, docs) => {
  ...
})

Or, with promises:

Model.insertMany([ ... ]).then((docs) => {
  ...
}).catch((err) => {
  ...
})

Watersoak

Thanks for this. It says it will insert them if they're all valid; does this mean if one fails all will fail? – Reachmedown

It's a bulk operation, but it's not atomic. I'm not sure how Mongoose does it and can't test right now, but it should return the number of successful writes. There are more details in MongoDB's documentation: docs.mongodb.com/manual/reference/method/… – Watersoak

If one fails insertMany doesn't insert anything, I have made the tests – Fortunio

in situations where you need to validate duplicate documents insert many can be a pain. it seems to work fine if the _id is already specified in new documents; it throws duplicate error for those – Iceskate

@Fortunio This is configurable now, when using Model.insertMany(docs [, options]), pass a boolean false for the options parameter so it will not fail fast. – Lianneliao

Answer 4

Bulk inserts in Mongoose can be done with .insert() unless you need to access middleware.

Model.collection.insert(docs, options, callback)

https://github.com/christkv/node-mongodb-native/blob/master/lib/mongodb/collection.js#L71-91

Allison

Response for Aaron Heckman 2011: not really. Model.create(doc1 [, docN], callback) sort of helps here but its still calling model.save for you on each one. If by "faster" you mean "bypass all mongoose hooks and validation" then you could drop down to the native driver and use it directly: Movie.collection.insert(docs, options, callback) github.com/christkv/node-mongodb-native/blob/master/lib/mongodb/… – Guimond

I keep seeing this answer, but this is not really a "mongoose" way of doing things. This bypasses the Mongoose models entirely. If you have default values set up for some fields in the mongoose models, they will be ignored, and not inserted in the DB. – Bolden

How do I use Model.collection.insert in Mongoose? Please provide an example. – Hydrophilic

I know there are some critics to this approach, but this is actually the best (if not the only) answer if you're dealing with a huge amount of documents. This other answer (https://mcmap.net/q/167642/-mongoose-mongodb-batch-insert) explains why it's better and gives an example. – Zeke

can anyone suggest what are the options, maybe? – Basal

Answer 5

Use async parallel and your code will look like this:

  async.parallel([obj1.save, obj2.save, obj3.save], callback);

Since the convention is the same in Mongoose as in async (err, callback) you don't need to wrap them in your own callbacks, just add your save calls in an array and you will get a callback when all is finished.

If you use mapLimit you can control how many documents you want to save in parallel. In this example we save 10 documents in parallell until all items are successfully saved.

async.mapLimit(myArray, 10, function(document, next){
  document.save(next);
}, done);

Redpoll

Interesting - would you mind giving a real-world usable example with an myArray; whereas myArray has 10 Million items. – Hydrophilic

Answer 6

I know this is an old question, but it worries me that there are no properly correct answers here. Most answers just talk about iterating through all the documents and saving each of them individually, which is a BAD idea if you have more than a few documents, and the process gets repeated for even one in many requests.

MongoDB specifically has a batchInsert() call for inserting multiple documents, and this should be used from the native mongodb driver. Mongoose is built on this driver, and it doesn't have support for batch inserts. It probably makes sense as it is supposed to be a Object document modelling tool for MongoDB.

Solution: Mongoose comes with the native MongoDB driver. You can use that driver by requiring it require('mongoose/node_modules/mongodb') (not too sure about this, but you can always install the mongodb npm again if it doesn't work, but I think it should) and then do a proper batchInsert

Moritz

Wrong, Pascal's answer completely misses the point. People who need bulk insert tend to need it because they want to insert 10,000,000 items in one go. Without bulk insert an operation which should take a few seconds can take hours. Model.create is an epic fail, as it pretends to be a bulk insert but under the hood it's just a for loop. – Declivitous

Mongoose seriously needs some revamp then. Also their docs leave A LOT to be desired. – Hydrophilic

I think @Yashua's question addresses it by using the underlying mongodb javascript driver. – Mortgagee

Answer 7

Newer versions of MongoDB support bulk operations:

var col = db.collection('people');
var batch = col.initializeUnorderedBulkOp();

batch.insert({name: "John"});
batch.insert({name: "Jane"});
batch.insert({name: "Jason"});
batch.insert({name: "Joanne"});

batch.execute(function(err, result) {
    if (err) console.error(err);
    console.log('Inserted ' + result.nInserted + ' row(s).');
}

Commentator

Answer 8

Use insertMany function to insert many documents. This sends only one operation to the server and Mongoose validates all the documents before hitting the mongo server. By default Mongoose inserts item in the order they exist in the array. If you are ok with not maintaining any order then set ordered:false.

Important - Error handling:

When ordered:true validation and error handling happens in a group means if one fails everything will fail.

When ordered:false validation and error handling happens individually and operation will be continued. Error will be reported back in an array of errors.

Adamsun

Answer 9

Here is another way without using additional libraries (no error checking included)

function saveAll( callback ){
  var count = 0;
  docs.forEach(function(doc){
      doc.save(function(err){
          count++;
          if( count == docs.length ){
             callback();
          }
      });
  });
}

Joaquin

Answer 10

You can use the promise returned by mongoose save, Promise in mongoose does not have all, but you can add the feature with this module.

Create a module that enhance mongoose promise with all.

var Promise = require("mongoose").Promise;

Promise.all = function(promises) {
  var mainPromise = new Promise();
  if (promises.length == 0) {
    mainPromise.resolve(null, promises);
  }

  var pending = 0;
  promises.forEach(function(p, i) {
    pending++;
    p.then(function(val) {
      promises[i] = val;
      if (--pending === 0) {
        mainPromise.resolve(null, promises);
      }
    }, function(err) {
      mainPromise.reject(err);
    });
  });

  return mainPromise;
}

module.exports = Promise;

Then use it with mongoose:

var Promise = require('./promise')

...

var tasks = [];

for (var i=0; i < docs.length; i++) {
  tasks.push(docs[i].save());
}

Promise.all(tasks)
  .then(function(results) {
    console.log(results);
  }, function (err) {
    console.log(err);
  })

Tribromoethanol

Uncaught TypeError: Promise resolver undefined is not a function in Promise.js – Whippet

Answer 11

Add a file called mongoHelper.js

var MongoClient = require('mongodb').MongoClient;

MongoClient.saveAny = function(data, collection, callback)
{
    if(data instanceof Array)
    {
        saveRecords(data,collection, callback);
    }
    else
    {
        saveRecord(data,collection, callback);
    }
}

function saveRecord(data, collection, callback)
{
    collection.save
    (
        data,
        {w:1},
        function(err, result)
        {
            if(err)
                throw new Error(err);
            callback(result);
        }
    );
}
function saveRecords(data, collection, callback)
{
    save
    (
        data, 
        collection,
        callback
    );
}
function save(data, collection, callback)
{
    collection.save
    (
        data.pop(),
        {w:1},
        function(err, result)
        {
            if(err)
            {               
                throw new Error(err);
            }
            if(data.length > 0)
                save(data, collection, callback);
            else
                callback(result);
        }
    );
}

module.exports = MongoClient;

Then in your code change you requires to

var MongoClient = require("./mongoHelper.js");

Then when it is time to save call (after you have connected and retrieved the collection)

MongoClient.saveAny(data, collection, function(){db.close();});

You can change the error handling to suit your needs, pass back the error in the callback etc.

Koosis

Answer 12

This is an old question, but it came up first for me in google results when searching "mongoose insert array of documents".

There are two options model.create() [mongoose] and model.collection.insert() [mongodb] which you can use. View a more thorough discussion here of the pros/cons of each option:

Mongoose (mongodb) batch insert?

Maxine

Answer 13

Here is an example of using MongoDB's Model.collection.insert() directly in Mongoose. Please note that if you don't have so many documents, say less than 100 documents, you don't need to use MongoDB's bulk operation (see this).

MongoDB also supports bulk insert through passing an array of documents to the db.collection.insert() method.

var mongoose = require('mongoose');

var userSchema = mongoose.Schema({
  email : { type: String, index: { unique: true } },
  name  : String  
}); 

var User = mongoose.model('User', userSchema);


function saveUsers(users) {
  User.collection.insert(users, function callback(error, insertedDocs) {
    // Here I use KrisKowal's Q (https://github.com/kriskowal/q) to return a promise, 
    // so that the caller of this function can act upon its success or failure
    if (!error)
      return Q.resolve(insertedDocs);
    else
      return Q.reject({ error: error });
  });
}

var users = [{email: '[email protected]', name: 'foo'}, {email: '[email protected]', name: 'baz'}];
saveUsers(users).then(function() {
  // handle success case here
})
.fail(function(error) {
  // handle error case here
});

Clothe

which mongoose version is this? – Third