What is the best way to do ajax pagination with MongoDb and Nodejs?

Question

I have mongodb and NodeJs. The connection is done via mongoosejs.
What is the best way to develop ajax infinity scroll ? Should I use limit and offset ?

Mentalism

Answer 1

"skip and limit" approach is not very efficient when you are paging far into dataset. It is effectively a Shlemiel the Painter's algorithm.

Range queries are much more efficient (when supported by indexes). For example, let's imagine that you're displaying tweets. Your page size is 20 and you're on page 1000 and want to load page 1001.

This query

db.tweets.find().sort({created_at: -1}).skip(1001*20).limit(20)

is much less efficient than

db.tweets.find({created_at: {$lt: last_displayed_date}}).
          sort({created_at: -1}).limit(20);

(provided that you have index on created_at).

You get the idea: when you load a page, note the timestamp of the last tweet and use it to query the next page.

Palmation

What to do if there are some tweets created at the same time by different users but I need to show them ? – Mentalism

Exactly the same millisecond? I don't think so. – Palmation

If that's the case, sort by created_at: -1, user_id: 1 – Palmation

It will not work in case of sorting by another date. For example: 1. User can filter events by date (only by day without hours) 2. There are 300 events created on different dates (created_at different) and they will occur (event_date) on different dates. 3. How to filter them and load by paging ? NOTE: it's impossible to sort by the event_date because this date is rounded and not unique – Mentalism

Man, come up with something. I gave you the basic idea :) – Palmation

I see ) But the 'basic idea' works only with basic problem similar to display items ordered by unique field (like date of creation). It will not work if you need to display ordered (by not unique filed) list of items. – Mentalism

Order by event_date, event_id. – Palmation

sorry for disturbing, my mistake. Thanks one more time – Mentalism

How do I filter results? ie match substring 'foo' in name for example – Catch

@chovy: exactly the same way you'd do if you weren't paginating – Palmation