How do I iterate over an entire MongoDB collection using mongojs?

Question

I'm using mongojs and I'm trying to iterate over all elements in a collection

index = 0

db.keys.find({}, {uid: 1, _id: 0}).forEach((err, key) =>
    if err?
        console.log err
    else 
        console.log (++index) + " key: " + key_uid

which logs

1 key: bB0KN
2 key: LOtOL
3 key: 51xJM
4 key: x9wFP
5 key: hcJKP
6 key: QZxnE
.
.
.
96 key: EeW6E
97 key: wqfmM
98 key: LIGHK
99 key: bjWTI
100 key: 2zNGE
101 key: F71mL

and then stops. However when I log into mongo from the terminal and run

> db.keys.count()
2317381

So clearly it should be returning a lot more keys. Do you have any ideas what could be causing this behavior?

Peckham

Answer 1

You need to use the each() method, not forEach(). forEach() is going to iterate over every document in the batch - as you've discovered this defaults to 101. each() will iterate over every document in the cursor. From the documentation:

each

Iterates over all the documents for this cursor. As with {cursor.toArray}, not all of the elements will be iterated if this cursor had been previouly accessed. In that case, {cursor.rewind} can be used to reset the cursor. However, unlike {cursor.toArray}, the cursor will only hold a maximum of batch size elements at any given time if batch size is specified. Otherwise, the caller is responsible for making sure that the entire result can fit the memory.

http://mongodb.github.io/node-mongodb-native/api-generated/cursor.html

Example code:

// Grab a cursor
      var cursor = collection.find();

      // Execute the each command, triggers for each document
      cursor.each(function(err, item) {

        // If the item is null then the cursor is exhausted/empty and closed
        if(item == null) {

          // Show that the cursor is closed
          cursor.toArray(function(err, items) {
            assert.ok(err != null);

            // Let's close the db
            db.close();
          });
        };
      });

Dynamics

Answer 2

You're seeing only first 101 documents because that's the default number of documents MongoDB driver fetched from the server in the first batch.

For most queries, the first batch returns 101 documents or just enough documents to exceed 1 megabyte. Subsequent batch size is 4 megabytes.

You can try to use find and then iterate over documents.

coll.find({}, {uid:1, _id : 0}, function(err, docs){
    if (err) {
        console.log(err);
        return;
    }
    docs.forEach(function(doc, index) { 
        console.log(index + " key: " + doc.uid) 
    });
});

Monstrous