Paging library - Boundary callback for network + db with API taking page and size

Question

Short question:

What is the correct way to handle database + network on the Paging library from Architecture components, using an API that uses page + size to load a new page and the BoundaryCallback class?

Research and explanation

Currently the class BoundaryCallback used in the paging library for architecture components, receives as parameter the instance of an element in the list without actual context of where this element is at. It happens in onItemAtFrontLoaded and onItemAtEndLoaded.

My Api is supposed to receive the page and the size of the page to load the next chunk of data. Boundary callback, added as part of the paged list builder, is supposed to tell you when to load the next page of data based on prefetch distance and page size.

Since the Api needs the page number and the size of the page to provide, I don't see a way to send that to the Api just by receiving one of the elements from the list as offered in onItemAtFrontLoaded and onItemAtEndLoaded. Checking the google examples in this link, they use the name of the last element to get the next one, but that doesn't fit an Api with page + size.

They also have another example with only network that uses PagedKeyedDatasource, but there is no sample or clue on how to mix that with the database and the BoundaryCallback.

Edit: Only solutions I have found so far is to store the last loaded page on the shared preferences, but that sounds like a dirty trick.

Refer to https://github.com/googlesamples/android-architecture-components/issues/252#issuecomment-392119468 for official input on it.

Arouse

The real problem is that if you have new elements on server-side, the page each item was on will shift. Can you reliably cache a pageIndex-based API into DB long-term? – Subtractive

Its possible if the elements are ordered by time (most recent last page), or like in my case, a generated feed that remains untouched once a day. There are multiple applications where the page index based on size + page number makes sense, and I have seen many APIs implementing that pattern – Arouse

Nope, only the one in the official support comment – Arouse

it depends if the API response delivers meta-data, alike page & pages. when it uses keys to get the next page, this seems to be optimized for noSQL - while for SQL this does not necessarily make sense... and when following the logic from the GitHub comments; what if the key had meanwhile be deleted? – Gotthelf

did you find any solution to this since now ? – Bartko

Answer 1

The documentation has this to say on the issue:

If you aren't using an item-keyed network API, you may be using page-keyed, or page-indexed. If this is the case, the paging library doesn't know about the page key or index used in the BoundaryCallback, so you need to track it yourself. You can do this in one of two ways:

Local storage Page key

If you want to perfectly resume your query, even if the app is killed and resumed, you can store the key on disk. Note that with a positional/page index network API, there's a simple way to do this, by using the listSize as an input to the next load (or listSize / NETWORK_PAGE_SIZE, for page indexing). The current list size isn't passed to the BoundaryCallback though. This is because the PagedList doesn't necessarily know the number of items in local storage. Placeholders may be disabled, or the DataSource may not count total number of items.

Instead, for these positional cases, you can query the database for the number of items, and pass that to the network.

In-Memory Page key

Often it doesn't make sense to query the next page from network if the last page you fetched was loaded many hours or days before. If you keep the key in memory, you can refresh any time you start paging from a network source. Store the next key in memory, inside your BoundaryCallback. When you create a new BoundaryCallback when creating a new LiveData/Observable of PagedList, refresh data. For example, in the Paging Codelab, the GitHub network page index is stored in memory.

And links to an example Codelab: https://codelabs.developers.google.com/codelabs/android-paging/index.html#8

Adel

Answer 2

I have a similar API (pageNum + size), I have 2 extra fields in my data class, pageNum and pageSize with defaults 1 and PAGE_SIZE respectively.

If you're using Network+DB then you'll have onZeroItemsLoaded and onItemAtEndLoaded, In onZeroItemsLoaded send pageNum and pageSize as it is, and in onItemAtEndLoaded increment pageSize by 1 and then send.

Let's say you have a method fetchData(pageNum, pageSize) when you receive result in this just update the pageNum and pageSize accordingly in each item of this page.

Regale

And what happens when you kill your app and later open it and app loads data from db? pageNum will not be valid... – Pedro

I have pageNum as a field in the table, so when you kill the app and re-open if Db is empty onZeroItem will be called, use pageNum=1, pageSize=PAGE_SIZE, and if onItemAtEndLoaded is called ( you also get the last item as a param in this method ), use (pageNum+1) from the last item in DB. – Regale

thanks for clarification, i didnt understand at first.In my example that would mean to iterate trough 20 items returned from one api response page and setting pageNum to every one before inserting into db. That dont seem like a very good solution. Im messing with using observeForever listening to separate entity which has page number. link Dont know if it is good approach though... – Pedro

I don't think to iterate over 20 items is expensive when you're doing it asynchronously (along with your API call which is probably async too). But in the end, it depends on you. You can use more methods like auto increment the pageNum field in table and then to calculate pageNum, just take MOD of it with PAGE_SIZE and then plus 1 (might need to adjust here and there). Edit: just saw you're also developing a movies app, you can check mine with same approach I discussed above github.com/Kashish-Sharma/TheMovieDBApp – Regale

Nice, ill keep it for a reference when i get stuck.Thanks. – Pedro

Answer 3

I implement this:

PagedList.BoundaryCallback<Produto> boundaryCallbackNovidades = new PagedList.BoundaryCallback<Produto>(){
    int proxPagina;
    boolean jaAtualizouInicio=false;

    public void onZeroItemsLoaded() {
        requestProdutos(
            webService.pesquisarNovidadesDepoisDe(LocalDateTime.now().format(Util.formatterDataTime), 0, 20));
    }

    public void onItemAtFrontLoaded(@NonNull Produto itemAtFront) {
        if(!jaAtualizouInicio)
            requestProdutos(
                webService.pesquisarNovidadesMaisRecentesQue(itemAtFront.data.format(Util.formatterDataTime)));
        jaAtualizouInicio=true;
    }

    public void onItemAtEndLoaded(@NonNull Produto itemAtEnd) {
        requestProdutos(
            webService.pesquisarNovidadesDepoisDe(LocalDateTime.now().format(Util.formatterDataTime), proxPagina++, 20));
    }
};


public LiveData<PagedList<Produto>> getNovidades(){
    if(novidades==null){
        novidades = new LivePagedListBuilder<>(produtoDao.produtosNovidades(),
                10)
                .setBoundaryCallback(boundaryCallbackNovidades)
                .build();
    }
    return novidades;
}

Metagenesis

Answer 4

Lets say you always fetch N=10 items per page from the server & then store it in db. You can get the number of items in db using the sql query SELECT COUNT(*) FROM tbl & store it in variable count. Now to get the page number that should be requested next, use:

val nextPage: Int = (count / N) + 1

Salbu