MongoDB query for document older than 30 seconds

Question

Does anyone have a good approach for a query against a collection for documents that are older than 30 seconds. I'm creating a cleanup worker that marks items as failed after they have been in a specific state for more than 30 seconds.

Not that it matters, but I'm using mongojs for this one.

Every document has a created time associated with it.

Exceeding

Answer 1

We are assuming you have a created_at or similar field in your document that has the time it was inserted or otherwise modified depending on which is important to you.

Rather than iterate over the results you might want to look at the multi option in update to apply your change to all documents that match your query. Setting the time you want to look past should be fairly straightforward

In shell syntax, which should be pretty much the same of the driver:

db.collection.update({
     created_at: {$lt: time },
     state: oldstate 
},
{$set: { state: newstate } }, false, true )

The first false being for upserts which does not make any sense in this usage and the second true marking for multi document update.

If the documents are indeed going to be short lived and you have no other need for them afterwards, then you might consider capped collections. You can have a total size or time to live option for these and the natural insertion order favours processing of queued entries.

Gear

Answer 2

If you want to do this using mongo shell:

db.requests.find({created: {$lt: new Date((new Date())-1000*60*60*72)}}).count()

...will find the documents that are older than 72 hours ("now" minus "72*60*60*1000" msecs). 30 seconds would be 1000*30.

Constable

Answer 3

You could use something like that:

var d = new Date();
d.setSeconds(d.getSeconds() - 30);

db.mycollection.find({ created_at: { $lt: d } }).forEach(function(err, doc) {} );

Jibber

I wasn't sure how to get a date object in the right format to use with the query. This worked great. Thanks! – Lacreshalacrimal

Answer 4

The TTL option is also an elegant solution. It's an index that deletes documents automatically after x seconds, see here: https://docs.mongodb.org/manual/core/index-ttl/

Example code would be:

db.yourCollection.createIndex({ created:1 }, { expireAfterSeconds: 30 } )

Remind

For those who will be using TTL indexes, please take note: The TTL index does not guarantee that expired data will be deleted immediately upon expiration. There may be a delay between the time a document expires and the time that MongoDB removes the document from the database. – Immunogenetics