Android Paging 3 how to filter, sort and search my data

Question

I'm trying to implement paging I'm using Room and it took me ages to realize that its all done for me 😆 but what I need to do is be able to filter search and sort my data. I want to keep it as LiveData for now I can swap to flow later. I had this method to filter search and sort and it worked perfectly,

    @SuppressLint("DefaultLocale")
    private fun searchAndFilterPokemon(): LiveData<List<PokemonWithTypesAndSpecies>> {
        return Transformations.switchMap(search) { search ->
            val allPokemon = repository.searchPokemonWithTypesAndSpecies(search)
            Transformations.switchMap(filters) { filters ->
                val pokemon = when {
                    filters.isNullOrEmpty() -> allPokemon
                    else -> {
                        Transformations.switchMap(allPokemon) { pokemonList ->
                            val filteredList = pokemonList.filter { pokemon ->
                                pokemon.matches = 0
                                val filter = filterTypes(pokemon, filters)
                                filter
                            }
                            maybeSortList(filters, filteredList)
                        }
                    }
                }
                pokemon
            }
        }
    }

It have a few switchmaps here, the first is responding to search updating

    var search: MutableLiveData<String> = getSearchState()

the second is responding to filters updating

    val filters: MutableLiveData<MutableSet<String>> = getCurrentFiltersState()

and the third is watching the searched list updating, it then calls filterTypes and maybeSortList which are small methods for filtering and sorting

    @SuppressLint("DefaultLocale")
    private fun filterTypes(
        pokemon: PokemonWithTypesAndSpecies,
        filters: MutableSet<String>
    ): Boolean {
        var match = false
        for (filter in filters) {
            for (type in pokemon.types) {
                if (type.name.toLowerCase() == filter.toLowerCase()) {
                    val matches = pokemon.matches.plus(1)
                    pokemon.apply {
                        this.matches = matches
                    }
                    match = true
                }
            }
        }
        return match
    }

    private fun maybeSortList(
        filters: MutableSet<String>,
        filteredList: List<PokemonWithTypesAndSpecies>
    ): MutableLiveData<List<PokemonWithTypesAndSpecies>> {
        return if (filters.size > 1)
            MutableLiveData(filteredList.sortedByDescending {
                Log.d("VM", "SORTING ${it.pokemon.name} ${it.matches}")
                it.matches
            })
        else MutableLiveData(filteredList)
    }

as mentioned I want to migrate these to paging 3 and am having difficulty doing it Ive changed my repository and dao to return a PagingSource and I just want to change my view model to return the PagingData as a live data, so far I have this

@SuppressLint("DefaultLocale")
private fun searchAndFilterPokemonPager(): LiveData<PagingData<PokemonWithTypesAndSpecies>> {
    val pager =  Pager(
        config = PagingConfig(
            pageSize = 50,
            enablePlaceholders = false,
            maxSize = 200
        )
    ){
        searchAndFilterPokemonWithPaging()
    }.liveData.cachedIn(viewModelScope)

    Transformations.switchMap(filters){
        MutableLiveData<String>()
    }

    return Transformations.switchMap(search) {search ->
        val searchedPokemon =
            MutableLiveData<PagingData<PokemonWithTypesAndSpecies>>(pager.value?.filter { it.pokemon.name.contains(search) })
        Transformations.switchMap(filters) { filters ->
            val pokemon = when {
                filters.isNullOrEmpty() -> searchedPokemon
                else -> {
                    Transformations.switchMap(searchedPokemon) { pokemonList ->
                        val filteredList = pokemonList.filter { pokemon ->
                            pokemon.matches = 0
                            val filter = filterTypes(pokemon, filters)
                            filter
                        }
                        maybeSortList(filters, filteredList = filteredList)
                    }
                }
            }
            pokemon
        }
    }
}

but the switchmap is giving me an error that

Type inference failed: Cannot infer type parameter Y in 

fun <X : Any!, Y : Any!> switchMap
(
    source: LiveData<X!>,
    switchMapFunction: (input: X!) → LiveData<Y!>!
 )

which I think I understand but am not sure how to fix it, also the filter and sort methods won't work anymore and I cant see any good method replacements for it with the PageData, it has a filter but not a sort? any help appreciated : LiveData<Y!

UPDATE thanks to @Shadow I've rewritten it to implement searching using a mediator live data but im still stuck on filtering

    init {
        val combinedValues =
            MediatorLiveData<Pair<String?, MutableSet<String>?>?>().apply {
                addSource(search) {
                    value = Pair(it, filters.value)
                }
                addSource(filters) {
                    value = Pair(search.value, it)
                }
            }

        searchPokemon = Transformations.switchMap(combinedValues) { pair ->
            val search = pair?.first
            val filters = pair?.second
            if (search != null && filters != null) {
                searchAndFilterPokemonPager(search)
            } else null
        }
    }



    @SuppressLint("DefaultLocale")
    private fun searchAndFilterPokemonPager(search: String): LiveData<PagingData<PokemonWithTypesAndSpecies>> {
        return Pager(
            config = PagingConfig(
                pageSize = 50,
                enablePlaceholders = false,
                maxSize = 200
            )
        ){
            searchAllPokemonWithPaging(search)
        }.liveData.cachedIn(viewModelScope)

    }

    @SuppressLint("DefaultLocale")
    private fun searchAllPokemonWithPaging(search: String): PagingSource<Int, PokemonWithTypesAndSpecies> {
        return repository.searchPokemonWithTypesAndSpeciesWithPaging(search)
    }

Glidebomb

Answer 1

I do not believe you can sort correctly on the receiving end when you are using a paging source. This is because paging will return a chunk of data from the database in whichever order it is, or in whichever order you specify in its query directly.

Say you have a list of names in the database and you want to display them sorted alphabetically.

Unless you actually specify that sorting in the query itself, the default query will fetch the X first names (X being the paging size you have configured) it finds in whichever order the database is using for the results of that query.

You can sort the results alright, but it will only sort those X first names it returned. That means if you had any names that started with A and they happened to not come in that chunk of names, they will only show when you have scrolled far enough for them to be loaded by the pager, and only then they will show sorted correctly in the present list. You might see names moving around as a result of this whenever a new page is loaded into your list.

That's for sorting, now for search.

What I ended up doing for my search was to just throw away the database's own capability for searching. You can use " LIKE " in queries directly, but unless your search structure is very basic it will be useless. There is also Fts4 available: https://developer.android.com/reference/androidx/room/Fts4

But it is such a PIA to setup and make use of that I ended up seeing absolutely no reward worth the effort for my case.

So I just do the search however I want on the receiving end instead using a Transformations.switchMap to trigger a new data fetch from the database whenever the user input changes coupled with the filtering on the data I receive.

You already have part of this implemented, just take out the contents of the Transformations.switchMap(filters) and simply return the data, then you conduct the search on the results returned in the observer that is attached to the searchAndFilterPokemonPager call.

Filter is the same logic as search too, but I would suggest to make sure to filter first before searching since typically search is input driven and if you don't add a debouncer it will be triggering a new search for every character the user enters or deletes.

In short:

1. sort in the query directly so the results you receive are already sorted
2. implement a switchMap attached to the filter value to trigger a new data fetch with the new filter value taken into account
3. implement a switchMap just like the filter, but for the search input
4. filter the returned data right before you submit to your list/recyclerview adapter
5. same as above, but for search

Tusker

thank you for your answer can this not maybe be achieved using a paging source? – Glidebomb

I honestly have no idea, never thought about it that way, but I assume that whatever works on the receiving end can also work in the paging source. However, for that I'd let someone else more experienced give their input since I have hardly had a need to make my own paging source when using Room integration. – Tusker

so off the back of what you suggested ive implemented search but a little differently so i am using the database with a LIKE query and im now using a mediator live data to return the pager and passing the search word to the pager – Glidebomb

sounds good, just take into account that the LIKE capabilities are very limited, so if you find out that some searches are not returning the results you expect start investigating from there. I had that problem immediately so I went with just the transform switch map instead because I have total control at that point. – Tusker

ive updated the question still not sure how id do filtering – Glidebomb

a filter is nothing more than a search, it is essentially the same thing with the only difference being you specify what the filters values are while in the search the user is the one entering the search string. try to do the same thing like you did with the search, but remember to do it in the same query string, not as 2 separate queries. – Tusker

for example, if your filter is electric for electric pokemons and the user searches for pikachu your query would be something along the lines of select * from pokemon where type=:filter and name like :name and the Dao would receive two arguments, the filter and the name, with those values being "electric" and "pikachu" – Tusker

yh but i also want it to surface results higher if they match more than one filter – Glidebomb

That would still be pure query, from my point of view. if you use where type in (:filters) it will match any types that are in the list of filters, this way if the user selects the filters electric and water it will match in the query any type results that are either electric or water w3schools.com/sql/sql_in.asp all you have to pass is a list of strings for the respective Dao argument instead of a single string. – Tusker

yh but then i want to sort them by ID and how many filters they matched so if i select fire and flying charizard would be first moltres second and then a bunch of only fire pokemon and only flying pokemon, – Glidebomb

also my types in a different table and im not sure how to write the join query – Glidebomb

That's more than 1 type per pokemon then, that will result in a n-n relation where a single pokemon can have 1 or more types, and that means it can match 1 or more types in your query results. This will result in a complex query, and I have not had to make something like this before to instruct you better. If I had to do it for me I would resort to using a RawQuery where each filter match would be an AND concatenated to the query, this way you can have a multiple filters working, but each one will be checked individually against the list of filters each pokemon has, but... – Tusker

But you also have to pull the type relation table into the query before you can conduct that filter properly. Something like select * from pokemon left join pokemonTypes on pokemon.id = pokemonTypes.id AND pokemonTypes.type = :filter1 AND pokemonTypes.type =: filter2 AND ... as many "ands" as the user selects and you generate these ands in your rawquery. The result will be a list of each pokemon that makes at least one of those filters or more and if you want to group them and sort them you do that after all the "ands" – Tusker

so it would be easier to filter them after i get those results but im returning a PagingSource from the dao as the docs instruct – Glidebomb

i might give up on paging – Glidebomb

correct, to me that would be way more easier, to modify the data returned by the query. I understand your frustration, the documentation for something that appears to be so basic like this is absolutely non-existent and I already messaged them trying to explain that this will make people give up adopting new things because they don't know how to since the documentation is lacking, but so far they do not appear very concerned with it, at least they haven't showed it. – Tusker

yh, thanks again, i wanted to implement this to conserve memory, im going to go through and create new data classes that only return the data needed and excludes fields that dont need to be returned for this exact query (which i know sounds sane enough but will soon desend into madness) but do you have any other tips for conserving memory? – Glidebomb

Not really, the paging is the way to go and you were on the right track when you decided to use Paging since it only loads portions of data on demand instead of pulling entire lists into memory straight away. – Tusker

the most frustrating thing is that if this was coming from the network i could manipulate the data – Glidebomb

Looking through the bugs tracker on paging 3 I've found alot of people using flows to achieve this I might have something working I'll update this if I do – Glidebomb

@Glidebomb Please do, I'd be most interested in seeing what you come up with and I am sure more developers trying to adopt Paging 3 will also be very interested in reading it. – Tusker

added an answer still not sure about sorting – Glidebomb

Answer 2

so this is at least partially possible using flows but sorting isnt possible see here https://issuetracker.google.com/issues/175430431, i havent figured out sorting yet but searching and filtering are possible, so for filtering i have it very much the same the query is a LIKE query that triggers by some livedata (the search value) emitting new data heres the search method

    @SuppressLint("DefaultLocale")
    private fun searchPokemonPager(search: String): LiveData<PagingData<PokemonWithTypesAndSpeciesForList>> {
        return Pager(
            config = PagingConfig(
                pageSize = 50,
                enablePlaceholders = false,
                maxSize = 200
            )
        ) {
            searchAllPokemonWithPaging(search)
        }.liveData.cachedIn(viewModelScope)
    }


    @SuppressLint("DefaultLocale")
    private fun searchAllPokemonWithPaging(search: String): PagingSource<Int, PokemonWithTypesAndSpeciesForList> {
        return repository.searchPokemonWithTypesAndSpeciesWithPaging(search)
    }

and the repo is calling the dao, now to do the filtering @Shadow suggested using the pager this works but is much easier with flow using combine, my filters are live data and flow has some convenient extensions like asFlow so it becomes as easy as

    @SuppressLint("DefaultLocale")
    private fun searchAndFilterPokemonPager(search: String): Flow<PagingData<PokemonWithTypesAndSpeciesForList>> {
        val pager = Pager(
            config = PagingConfig(
                pageSize = 50,
                enablePlaceholders = false,
                maxSize = 200
            )
        ) {
            searchAllPokemonWithPaging(search)
        }.flow.cachedIn(viewModelScope).combine(filters.asFlow()){ pagingData, filters ->
            pagingData.filter { filterTypesForFlow(it, filters) }
        }
        return pager
    }

obviously here I've changed the return types so we also have to fix our mediator live data emitting our search and filters as it expects a live data, later ill fix this so it all uses flow

    init {
        val combinedValues =
            MediatorLiveData<Pair<String?, MutableSet<String>?>?>().apply {
                addSource(search) {
                    value = Pair(it, filters.value)
                }
                addSource(filters) {
                    value = Pair(search.value, it)
                }
            }

        searchPokemon = Transformations.switchMap(combinedValues) { pair ->
            val search = pair?.first
            val filters = pair?.second
            if (search != null && filters != null) {
                searchAndFilterPokemonPager(search).asLiveData()
            } else null
        }
    }

here we just use the asLiveData extesnion

Glidebomb

sorting will be trickier because if I understood right you want to sort by type, but each result can have more than 1 type so you have to come up with how you want to interpret sorting multiple types, for example: sort by whichever has the most types matched first and then alphabetically or just sort alphabetically regardless of the number of matched types or some other order you find to be the best – Tusker

yh i want it to sort by matched types and by ID ive started an issue here issuetracker.google.com/issues/175430431 we'll see what happens, thanks again – Glidebomb

Just a suggestion, include some data examples in your issue so it is easier to understand what you are trying to achieve with something that is more tangible than just a description of it, helps a ton. – Tusker

@Tusker didn't need to they've said they won't be fixing this – Glidebomb

It basically came back to what I said before; the sorting has to be done before the paging receives the data, and it makes sense. Paging is meant to "split this huge list into X chuncks of size Y" and that is all it can do. – Tusker