How to fetch data over multiple pages?

Question

My project is based on React, redux, redux-saga, es6 and I try to fetch data from this API:

http://api.dhsprogram.com/rest/dhs/data/BD,2000,2004,2007?&returnFields=CharacteristicLabel,Indicator,IndicatorId,Value&f=json

As you can see, this specific API call shows data with a limit of 100 data per page spread on 40 pages.

According to this answer: http://userforum.dhsprogram.com/index.php?t=msg&th=2086&goto=9591&S=Google it says that you can extend the limit to a maximum of 3000data per page.

However, in some cases I would do an API call that exceeds that limit which means I would not receive all my data doing it like this:

export function fetchMetaData(countryCode: string, surveyYears: string) {
return (fetch('http://api.dhsprogram.com/rest/dhs/data/' + countryCode + ',' + surveyYears + '?returnFields=CharacteristicLabel,Indicator,IndicatorId,Value&f=json')
    .then(response => response.json())
    .then(json => json.Data.map(survey => survey)))
} 

So my question is; what is the best way to get all data from this API given that I know the total pages of data. The answer in the forum link suggest to loop through the API. However, I can't find the right syntax usage to do this.

My idea would be doing one api call to get the total number of pages. Then store this in a state using redux+redux-saga. Then do a new request sending the total pages as parameter and fetch this total number of pages times. And by doing this I can't figure out the syntax to store the data for each iteration.

Koh

A couple of questions: 1. Why would you need to save the total number of pages in redux store? You probably wan't to encapsulate fetching all data into one action (using several async fetches). 2. You more likely meant "...sending the start page as parameter...", right? 3. What specifically is your problem, what have you tried so far? Iterating over arrays? Aggregating? Handling several async calls or promises? – Mannerheim

1. Storing total number of pages in redux store was just a suggestion to my own problem and what you propose sounds more logical. 2. Correct. 3. What I have done so far is using redux-saga to handle async fetches. When my component request data by dispatching an action, my watcher in saga receive the action and request data from the function above. What I receive is data from 1 page which I store in the action body and add it to my reducer. My problem is that I can't get all data from every pages using the function above. Is there a smart way to do this? @Mannerheim – Koh

Okay, we're getting closer to the point. I never used redux-saga and don't know if this lib offers a dedicated and preferred way to solve this. But I don't think this shouldn't stop you in solving your problem in a typical way using promises (or async/await) and functions that deal with building/mapping arrays and objects. If I got you right, you're looking for a way to execute several asynchronous calls to one API, aggregate the gathered data and then hand it over to the store/reducer for creating new state. May I ask: How familiar are you with JavaScript in general? – Mannerheim

Answer to your question, imo after doing some projects using javascript I would say I understand Javascript/ES6 basic well. However, I have not touched a lot of ES7like async/await or data fetching. This is the first time in my project. – Koh

Answer 1

A possible solution -the idea is to get the number of pages first, then make the appropriate number of API calls, pushing the promise from each call into an array. We then wait for all the promises to resolve, and do something with the returned data.

async function fetchMetaData() {

    const response = await fetch('apiUrlToGetPageNumber');

    const responses = await Promise.all(
        Array.from(
            Array(resp.data.pagesRequired),
            (_, i) => fetch(`apiUrlToSpecificPage?page=${i}`)
        )
    );
    
    // do something with processedResponses here

}
            
            

Windproof

Thank you very much! (I upvoted answer but I have less then 15rep). However, by using .reduce on processedResponses; the function can' recall that prev is defined. But I get the idea, so I will try to figure it out! – Koh

Good point on the reduce, I will edit into something that works. – Windproof

I would appreciate that =) – Koh

I have tried to make this works ever since. I just learned that Javascript is not running each line/statement sequential (which is new to me). In my case "processedResponses" are supposed to get all the resolved data, but since every statement is racing for the finish line my return value is an empty array. I wonder if you (or anyone) know a solution to this? I am currently researching generators in ES6 in hope that may be the solution. But I can't get it to work yet. – Koh

Answer 2

Here is another possible solution using async/await. The beauty of this is that the total_pages count is dynamic, so that if it increases while you're processing your request, it'll make sure you get it all.

async function fetchMetaData() {
  let allData = [];
  let morePagesAvailable = true;
  let currentPage = 0;

  while(morePagesAvailable) {
    currentPage++;
    const response = await fetch(`http://api.dhsprogram.com/rest/dhs/data?page=${currentPage}`)
    let { data, total_pages } = await response.json();
    data.forEach(e => allData.unshift(e));
    morePagesAvailable = currentPage < total_pages;
  }

  return allData;
}

Parasitology

What would this look like with a required Rest-User-Token? – Dumfries

@Dumfries you'd need to pass in options into the fetch call with the headers included. It'd look something like: fetch('http://api.dhs...', { headers: { "Rest-User-Token": "your-token" } }). Docs for supplying request options to fetch – Parasitology