There are many ways to select random document from a mongodb collection (as discussed in this answer). Comments point out that with mongodb version >= 3.2 then using $sample in the aggregation framework is preferred. However, on a collection with many small documents this seems to extremely slow.
The following code uses mongoengine to simulate the issue and compare it to the "skip random" method:
import timeit
from random import randint
import mongoengine as mdb
mdb.connect("test-agg")
class ACollection(mdb.Document):
name = mdb.StringField(unique=True)
meta = {'indexes': ['name']}
ACollection.drop_collection()
ACollection.objects.insert([ACollection(name="Document {}".format(n)) for n in range(50000)])
def agg():
doc = list(ACollection.objects.aggregate({"$sample": {'size': 1}}))[0]
print(doc['name'])
def skip_random():
n = ACollection.objects.count()
doc = ACollection.objects.skip(randint(1, n)).limit(1)[0]
print(doc['name'])
if __name__ == '__main__':
print("agg took {:2.2f}s".format(timeit.timeit(agg, number=1)))
print("skip_random took {:2.2f}s".format(timeit.timeit(skip_random, number=1)))
The result is:
Document 44551
agg took 21.89s
Document 25800
skip_random took 0.01s
Wherever I've had performance issues with mongodb in the past my answer has always been to use the aggregation framework so I'm surprised $sample is so slow.
Am I missing something here? What is it about this example that is causing the aggregation to take so long?
$samplewas very slow in 3.2.5, but basically instantaneous in 3.2.7. – Unveiling