RTK Query - Infinite Scrolling, retaining existing data
Asked Answered
M

2

21

I am attempting to implement infinite scrolling functionality in our current application;

We first fetch the first 5 'Posts' for a page. Upon scrolling to the bottom of the page, we then fetch the next 5 Posts.

This works nicely, however using the same query means that the existing data (the first 5 posts) has been replaced by the new data.

Is it possible to merge the existing data with the new data?

I could merge them in place, for example with something like; const posts = [newPosts, oldPosts] but then we lose the data invalidation provided by RTK Query if the existing data is modified.

What is the recommended approach for this case?

Mendenhall answered 7/6, 2022 at 11:6 Comment(3)
Don't know how traffic sensitive your application is, but as workaround you could certainly fetch all posts once and only change the visibility upon scrolling.Grout
@JohannesGriebenow We do have lazyloading in place, though fetching all posts at once would be quite significant. It might be an option to simply increase the count upon reaching maximum scroll depth, thereby achieving some level of pagination, but I'm wondering if there's an alternative.Mendenhall
You can pick some ideas from this thread github.com/reduxjs/redux-toolkit/discussions/1163Balbinder
R
39

In RTK 1.9 it is now possible to use the merge option to merge newly fetched data with the data that currently lives inside the cache. Make sure you use the option together with serializeQueryArgs or forceRefetch to keep a cache entry for the data.

createApi({
  baseQuery: fetchBaseQuery({ baseUrl: '/' }),
  endpoints: (build) => ({
    listItems: build.query<string[], number>({
      query: (pageNumber) => `/listItems?page=${pageNumber}`,
      // Only have one cache entry because the arg always maps to one string
      serializeQueryArgs: ({ endpointName }) => {
        return endpointName
      },
      // Always merge incoming data to the cache entry
      merge: (currentCache, newItems) => {
        currentCache.push(...newItems)
      },
      // Refetch when the page arg changes
      forceRefetch({ currentArg, previousArg }) {
        return currentArg !== previousArg
      },
    }),
  }),
})

Source: RTK Documenation on the merge option

Using this you can easily implement infinite scroll. Changing the pageNumber parameter of your query, will automatically fetch new data and concat it with the data that was already in the cache.

To illustrate this, I've created a working example on CodeSandbox.

enter image description here

Rizzio answered 18/12, 2022 at 22:1 Comment(11)
this saved a lot of logic and many lines of code, thanks a lotJepson
even though, if you try to update this data with a post request, the cache data will not be override, do you know how you could solve this ?Jepson
Unfortunately this doesn't work nicely with providesTags and invalidatesTags at this point. I currently set the page parameter back to 0 after the invalidating call is finsihed. This refetches the first entries automatically. On top of that I have some logic in my merge function that wipes the cache whenever page 0 is requested. It is similar to what is described here: github.com/reduxjs/redux-toolkit/issues/2874.Rizzio
it's a pain in the a**, they already did everything with RTK but they just don't want to fix this problem, I don't get why they don't want to fix this problem and make like like react-queryJepson
@DaanKlijn can you please explain this forceRefetch option. So how much I understood is (in the example in answer), if the new argument is different then it will refetch the data, but what does that actually mean, will it overwrite the whole cache also (but that is not happening) and if not, isn't it just the normal data fetching ??Invertebrate
forceRefetch, refetches the data and normally overwrites the original cache. But, whenever the merge option is provided like I've done it doesn't overwrite the orignal cache, but instead merges it with the already existing cache.Rizzio
The thing you now might be wondering is: what if I actually want to overwrite the full cache then? Not beautiful, but you can add some logic to the merge function s.t. it doesn't merges the new and old cache in some cases. For my usecase I have added an if statement that checks wether the user is requesting the first page (e.g. pageNumber === 0). Whenever the first page is requested I replace the full cache with the first page data. Whenever another page is requested (e.g. pageNumber > 0) I will merge it into the exising cache.Rizzio
@Jepson : highly belated, but what do you mean by "don't want to fix the problem"? Which problem specifically? Note that we can't monitor every SO question for ideas. If you have suggestions, it's best to file an issue on our repo.Destiny
How do you use this with providesTags and invalidatesTags to make it refetch everything if you mutate the data e.g. with posting a new Pokémon in this example?Jehiah
@DaanKlijn You mentioned: "I currently set the page parameter back to 0 after the invalidating call is finsihed" How do I accomplish this? I need to reset the page to zero when the query is invalidated by another endpoint.Dripdry
@Jehiah Did you figure out how to make it refetch everything?Dripdry
H
1

Here is a workaround for having infinite loading with caching benefits of rtk-query

in my example backend response type is

{ items: anyDTO[], count: number /* totalCount */ }

in order to make it work properly when invalidating tag I had to fetch first page with hook and handle the rest in useEffect.

import { useCallback, useEffect, useState } from 'react';
import { useDispatch } from 'react-redux';
import * as R from 'ramda';
import { ApiEndpointQuery } from '@reduxjs/toolkit/dist/query/core/module';
import { QueryHooks } from '@reduxjs/toolkit/dist/query/react/buildHooks';

interface UseLazeyInfiniteDataProps<T, N> {
  api: T;
  /** any rtk-query api: passing the whole enpoint so we have access to api utils to invalidate provided tags */
  apiEndpointName: N;
  /** apiEndpoint name to retrieve correct apiEndpoint query which will have 'initiate' and 'useQuery' */
  apiArgs: { [key: string]: any; params: object };
  /** apiArgs are the query arguments it should have a params objec */
  limit?: number;
  /** limit or page-size per request (defaults 20) */
  invalidatesTags?: any[];
}
/**
 * This hook is for having infinite loading experience with caching posibility of rtk-query
 * it's storing the data comming from rtk-q to local useState throgh a useEffect hook
 * in orther to make it work when invalidating tags it makes the first page request through rtk-query hook
 * and whenever it changes it will refetch the rest data
 */
const useLazyInfiniteData = <
  T extends { endpoints: any; util: any },
  N extends keyof T['endpoints'],
>({
  api,
  apiEndpointName,
  apiArgs,
  limit = 20,
  invalidatesTags,
}: UseLazeyInfiniteDataProps<T, N>) => {
  const dispatch = useDispatch<any>();
  const [pageNumber, setPageNumber] = useState(1); // first load only page 1
  const [maxPage, setMaxPage] = useState(0); // we don't know how many pages could exists yet
  const [accData, setAccData] = useState<any[]>([]);
  const [isFetchingMore, setIsFetchingMore] = useState(false);

  const apiEndpoint: ApiEndpointQuery<any, any> & QueryHooks<any> =
    api.endpoints[apiEndpointName];
  // we need this extra hook to automate refetching when invalidating tag
  // this will make the useEffect rerender if the first page data changes
  const {
    currentData: firstPageData,
    isLoading,
    isFetching,
    refetch: refetch_,
  } = apiEndpoint.useQuery({
    ...apiArgs,
    params: R.mergeRight(apiArgs.params, { offset: 0, limit }),
  });

  const refetch = useCallback(() => {
    if (invalidatesTags) {
      dispatch(api.util.invalidateTags());
    }
    refetch_();
  }, [api.util, dispatch, invalidatesTags, refetch_]);

  /** when params change like changing filters in the params then we reset the loading pages to 1 */
  useEffect(
    function resetPageLoadDataForSinglePage() {
      setPageNumber(1);
    },
    [apiArgs.params],
  );

  useEffect(
    function loadMoreDataOnPageNumberIncrease() {
      if (firstPageData)
        setMaxPage(Math.ceil((firstPageData as any).count / limit));

      if (pageNumber === 1) {
        setAccData((firstPageData as any)?.items ?? []);
      }
      if (pageNumber > 1) {
        setIsFetchingMore(true);
        const promises = R.range(1, pageNumber).map((page) =>
          dispatch(
            apiEndpoint.initiate({
              ...apiArgs,
              params: R.mergeRight(apiArgs.params, {
                offset: page * limit,
                limit,
              }),
            }),
          ).unwrap(),
        );

        Promise.all(promises)
          .then((data: any[]) => {
            const items = R.chain(R.propOr([], 'items'), [
              firstPageData,
              ...data,
            ]);
            setAccData(items);
          })
          .catch(console.error)
          .finally(() => {
            setIsFetchingMore(false);
          });
      }
    },
    [apiEndpoint, apiArgs, dispatch, firstPageData, limit, pageNumber],
  );

  /** increasing pageNumber will make the useEffect run */
  const loadMore = useCallback(() => {
    setPageNumber(R.inc);
  }, []);

  return {
    data: accData,
    loadMore,
    hasMore: pageNumber < maxPage,
    isLoading,
    isFetching,
    isFetchingMore,
    refetch,
  };
};

export default useLazyInfiniteData;

usage: Assuming you have rtk query API:

const extendedApi = emptySplitApi.injectEndpoints({ 
  endpoints: (build) => ({ 
    example: build.query({
      query: ({x, params: { offset, limit }}) => 'test'
    })
  }),
})

You can use it like:

useLazyInfiniteData({ 
  api: extendedApi,
  apiEndpointName: 'example',
  apiArgs: { x }, // better to be a memorized value
})
Hungary answered 27/10, 2022 at 13:13 Comment(3)
Can you share how can I use this hook in my component?Dripdry
How to use this please provide an exampleBroida
Sorry for the late reply in advance. assuming you have rtk query API: ``` const extendedApi = emptySplitApi.injectEndpoints({ endpoints: (build) => ({ example: build.query({ query: ({x, params: { offset, limit }}) => 'test', }), }), }) ``` you can use it like ``` useLazyInfiniteData({ api: extendedApi, apiEndpointName: 'example', apiArgs: { x }, // better to be a memorized value }) ```Hungary

© 2022 - 2024 — McMap. All rights reserved.