How to handle pagination when the source data changes frequently

Question

Specifically, I'm using Elasticsearch to do pagination, but this question could apply to any database.

Elasticsearch provides methods to paginate search results with handy from and to parameters.

So I run a query get me the most recent data from result 1 to 10

This works great.

The user clicks "next page" and the query is: get me the most recent data from result 11 to 20

The problem is that in the time between the two queries, 2 new records have been added to the backing database, which means the paginated results will overlap (the last 2 from the first page show up as first two on the second page).

What's the best solution to avoid this? Right now, I'm adding a filter to the query that tell it to only include results later than the last result of the previous query. But it just seems hackish.

Bracy

Answer 1

A filter is not a bad option, if you're already indexing a relevant timestamp. You have to track that timestamp on the client side in order to correctly prepare your queries. You also have to know when to get rid of it. But those aren't insurmountable problems.

The Scroll API is a solid option for this, because it effectively snapshots in time on the Elasticsearch side. The intent of the Scroll API is to provide a stable search query for deep pagination, which has to deal with the exact issue of change that you're experiencing.

You begin a Scrolling Search by supplying your query and the scroll parameter, for which Elasticsearch returns a scroll_id. You then make requests to /_search/scroll supplying that ID, each of which return a page of results and a new scroll_id for the next request.

(Note that you don't want the scan search type here. That's used to extract documents en masse, and does not apply any sorting.)

Compared to filtering, you do still have to track a value: the scroll_id for your next page of results. Whether that's easier than tracking a timestamp depends on your app.

There are other potential downsides to consider. Elasticsearch persists the context for your search on a single node within the cluster. Conceivably these could accumulate in your cluster, depending on how heavily you rely on scrolling search. You'll want to test the performance implications there. And if I recall correctly, scrolling searches also do not persist through a node failure or restart.

The ES documentation for the Scroll API provides good details on all of the above.

Bottom line: filtering by timestamp is actually not a bad choice. The Scroll API is another valid option, designed for a similar use case, but is not without its drawbacks.

Mallorie

Thanks for the detailed explanation. For some reason, I was thinking scan/scroll were the same thing, but that's obviously not true! Scroll seems like a good choice when you don't have something like the latest timestamp to filter on. – Bracy

Would the drawback for timestamp filtering be that if someone were to delete a document, you could potentially skip a document from being returned in the next page of results? – Maelstrom

Note from the Scroll API docs Scrolling is not intended for real time user requests, – Acrefoot

Indeed, newer versions of ES have other, stateless, mechanics for real-time deep pagination, namely search_after in the search request. – Mallorie

Answer 2

Realise this is a bit old but with ElasticSearch 6.3 there's now the search_after feature for the request body which allows for cursor type paging:

https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-search-after.html

It is very similar to the scroll API but unlike it, the search_after parameter is stateless, it is always resolved against the latest version of the searcher.

Malayan

Answer 3

You need to use scan API for this. Scan and scroll API let's you do point in time search and pagination. Scan API -

Haddad

Using the scan API might not be the correct thing to do here. It does not apply sorting on results. – Prosperus