Recursively calling an asynchronous API call

Question

I'm trying to fetch data from an API that only returns 1000 items per call, and I want to do this recursively until I have all the data.

I don't know beforehand how many items there are in total, so after every call I have to check

If the call was synchronous, I'd use something like this:

function fetch(all, start) {
    const newData = getData(start, 1000);
    all = all.concat(newData);
    return (newData.length === 1000) ? fetch(all, all.length) : all;
}

However, the getData() call here is asynchronous. Using a Promise.all() doesn't work because I don't know beforehand how many calls I need so I can't prepare an array of calls.

I have the feeling I could solve this with generators or with async/await but I don't know how. Can anyone point me in the right direction?

In case it makes a difference, I'm using Angular 4.

Overmaster

Answer 1

This depends on how things are done in your case. Considering that getData returns a promise, it is:

async function fetch(all, start) {
    const newData = await getData(start, 1000);
    all = all.concat(newData);
    return (newData.length === 1000) ? await fetch(all, all.length) : all;
}

Averell

Amazing, it was as simple as that. Thanks! – Overmaster

You're welcome. Yes, that's why async/await is great. – Averell

Answer 2

You can implement this with async/await without recursion:

let fetch = async () {
  try {
    let start = 0;
    let all = [];
    let newData;
    do {
      newData = await getData(start, 1000);
      start += newData.length;
      all = all.concat(newData);
    } while (newData.length > 0);

    return all;
  } catch (err) {
    console.log(err);
  }
}

Episode

You solution works just as well as @estus's, but his is just a bit cleaner. Thanks for your answer though, I hadn't realized this could be so simple. – Overmaster

Be careful with recursive functions, also it can be a problem how to handle errors thrown in async function getData. – Episode

Thanks, I'll keep that in mind. I'm inclinced towards recursion because it keeps code more functional (less state variables), but I agree it does introduce the challenge of handling errors. – Overmaster