How to set different TTL for every document in MongoDB

Question

As far I know, currently MongoDB TTL are managed by expireAfterSeconds index and it's setting for all documents inside a collection.

Is there any built-in way to set expiration to single document?

Flour

The TTL index is a general index. On that you are totally right. But the time to live is still computed on a per document basis since it's set based on a specifc field against which the index will be evaluated. Is that what you need? Or do you actually want to specify a different length of the ttl on a per document basis? – Hellen

@ManuelZelenka i want more like redis, TTL/expireAfterSeconds specified in single document – Flour

Then you're good to go with Kevins answer. Setting expireAfterSeconds to 0 and then simply defining in the indexed field of the document when it should expire is the way to go. – Hellen

Answer 1

On each documet you can set a field of expires and create the following index:

db.docs.createIndex( { expires:1 }, {expireAfterSeconds: 0 } );

documents inside this collection will get removed once time ticks over the expired field. However, this may take up 60 seconds as the background task that runs to remove documents runs every 60 seconds.

Here's an example though...

So let's insert a bunch of documents adding 60 seconds to each document each time we add them.

> var expires = new Date(); // Gets the current datetime.
> expires.setSeconds(expires.getSeconds() + 60)
1587812623023
> db.docs.insert({expires})
WriteResult({ "nInserted" : 1 })

> expires.setSeconds(expires.getSeconds() + 60)
1587812683023
> db.docs.insert({expires})
WriteResult({ "nInserted" : 1 })

> expires.setSeconds(expires.getSeconds() + 60)
1587812743023
> db.docs.insert({expires})
WriteResult({ "nInserted" : 1 })

> db.docs.find()
{ "_id" : ObjectId("5ea418de00f07c4d6461090b"), "expires" : ISODate("2020-04-25T11:03:43.023Z") }
{ "_id" : ObjectId("5ea418e300f07c4d6461090c"), "expires" : ISODate("2020-04-25T11:04:43.023Z") }
{ "_id" : ObjectId("5ea418e600f07c4d6461090d"), "expires" : ISODate("2020-04-25T11:05:43.023Z") }

Now if we add the TTL index.

db.docs.createIndex( { expires:1 }, { expireAfterSeconds: 0 } )

Then we can monitor our collection each 60 seconds and see each document getting removed.

> new Date()
ISODate("2020-04-25T11:03:28.278Z")
> db.docs.find()
{ "_id" : ObjectId("5ea418de00f07c4d6461090b"), "expires" : ISODate("2020-04-25T11:03:43.023Z") }
{ "_id" : ObjectId("5ea418e300f07c4d6461090c"), "expires" : ISODate("2020-04-25T11:04:43.023Z") }
{ "_id" : ObjectId("5ea418e600f07c4d6461090d"), "expires" : ISODate("2020-04-25T11:05:43.023Z") }

Nothing has been removed yet.

> new Date()
ISODate("2020-04-25T11:04:18.652Z")
> db.docs.find()
{ "_id" : ObjectId("5ea418e300f07c4d6461090c"), "expires" : ISODate("2020-04-25T11:04:43.023Z") }
{ "_id" : ObjectId("5ea418e600f07c4d6461090d"), "expires" : ISODate("2020-04-25T11:05:43.023Z") }

One document has gone.

> new Date()
ISODate("2020-04-25T11:05:17.705Z")
> db.docs.find()
{ "_id" : ObjectId("5ea418e600f07c4d6461090d"), "expires" : ISODate("2020-04-25T11:05:43.023Z") }

Another is gone.

> new Date()
ISODate("2020-04-25T11:06:31.390Z")
> db.docs.find()
>

and we're left with no doucments in our collection.

Westphal

I spent zero time thinking about a solution to this :) Thanks Kevin and thanks SO! – Mylohyoid

In Mongo 5.0+, expireAfterSeconds must be set to 1. If you set it to 0 it will default to disabling TTL expiration. – Mylohyoid

Hey @VladFr, I can't find anywhere in the docs explaining this? can you provide a link please? mongodb.com/docs/manual/tutorial/expire-data/… – Westphal

@KevinSmith, can you please explain, how this solution works? Why don't documents expire right after insertion? Why MongoDB has to wait until time will be greater than expires key? I've looked into the documentation, but could not figure this out. – Quadruplet

@BorisKalinin The expireAfterSeconds on the index is the time it'll wait after the datetime of the field specified, because we set 0 it will expire at the datetime field. This is all also done on a background thread which reads the values of the index then deletes them when expired. – Westphal

Answer 2

Since this is the top search result for "different TTL for every document in pymongo", I'll post this answer here.

One important thing to make documents in the same collection expire at different points in time is to use datetime with the same timezone as whatever timezone your MongoDB instance is running in. This is not a problem in node.js because new Date() returns datetime in UTC.

In my case, I was using the naïve datetime (in Python) but my MongoDB instance was in UTC, so all documents were getting deleted in one go. Using a datetime in UTC fixed the issue.

The following is an example code where every document expires after 60 seconds .

from datetime import datetime, UTC

# create the ttl index
collection.create_index(keys="created_at", expireAfterSeconds=60)

# insert a doc
collection.insert_one({"created_at": datetime.now(UTC)})

Kurys