Using node.js and promise to fetch paginated data

Question

Please keep in mind that I am new to node.js and I am used with android development.

My scenario is like this:

1. Run a query against the database that returns either null or a value
2. Call a web service with that database value, that offers info paginated, meaning that on a call I get a parameter to pass for the next call if there is more info to fetch.
3. After all the items are retrieved, store them in a database table
4. If everything is well, for each item received previously, I need to make another web call and store the retrieved info in another table
5. if fetching any of the data set fails, all data must be reverted from the database

So far, I've tried this:

getAllData: function(){
  self.getMainWebData(null)
     .then(function(result){
       //get secondary data for each result row and insert it into database
   }
}


getMainWebData: function(nextPage){
    return new Promise(function(resolve, reject) {

      module.getWebData(nextPage, function(errorReturned, response, values) {

    if (errorReturned) {
          reject(errorReturned);
        }

        nextPage = response.nextPageValue;
        resolve(values);
      })
    }).then(function(result) {

    //here I need to insert the returned values in database     

      //there's a new page, so fetch the next set of data
      if (nextPage) {
          //call again getMainWebData? 
          self.getMainWebData(nextPage)
      }

    })

There are a few things missing, from what I've tested, getAllData.then fires only one for the first set of items and not for others, so clearly handling the returned data in not right.

LATER EDIT: I've edited the scenario. Given some more research my feeling is that I could use a chain or .then() to perform the operations in a sequence.

Inconsonant

Here: #44784297 you can find an example of implementing a transaction with pg-promise that pulls data asynchronously. – Larcher

Thank you @Larcher but since I am new to node it's hard for me to understand. I've used knex for postgres insert and worked fine. I need to study your answer from the other question in more detail before understanding it – Inconsonant

You probably used knex outside of a transaction. This time it is was trickier. You need a transaction, to be able to roll the changes back, and you need to run an infinite asynchronous sequence. And that's what that example does. – Larcher

@Larcher Indeed, you have a point, as if something fails during the fetching, I need to be able to rollback all the changes. Could you post an answer here related to my question? Thank you for taking your time to answer me. – Inconsonant

Best is to read this: github.com/vitaly-t/pg-promise/wiki/Data-Imports. It is too much for re-posting it here. – Larcher

@Larcher I'm looking over your answer to the other question and your link. My question is how do you handle the index as in t.sequence(index? For instance, on each web call, I get the nextPage value but I don't understand how you actually call getNextData(index) where I don't see index being save or sent anywhere since it may change on each getNextData or be null. – Inconsonant

In my case since nextPage is independent from the actual result, how do I pass it foward from getNextData? – Inconsonant

A deeper look into your sample, seems that index increments, I want to have the nextPage value there which is not 0,1,2... it could be a String for instance. – Inconsonant

That's easy to do. See the complete syntax for method sequence. The second parameter we pass in is the previously resolved data. So you can append .then(data => {...inserting}).then(()=>{return whatever-string});, then use it as getNextData(pageIndex, previousString){}. – Larcher

@Larcher please see my open question related to this #44926539 – Inconsonant

Answer 1

Yes it is happening as you are resolving the promise on the first call itself. You should put resolve(value) inside an if statement which checks if more data is needed to be fetched. You will also need to restructure the logic as node is asynchronous. And the above code will not work unless you do change the logic.

Solution 1:

You can either append the paginated response to another variable outside the context of the calls you are making. And later use that value after you are done with the response.

getAllData: function(){
  self.getMainWebData(null)
  .then(function(result){
       // make your database transaction if result is not an error
   }
}

function getList(nextpage, result, callback){
  module.getWebData(nextPage, function(errorReturned, response, values) {
    if(errorReturned)
      callback(errorReturned);
    result.push(values);
    nextPage = response.nextPageValue;
    if(nextPage)
      getList(nextPage, result, callback);
    else
      callback(null, result);
  })
}
getMainWebData: function(nextPage){
  return new Promise(function(resolve, reject) {
    var result = [];
    getList(nextpage, result, function(err, results){
      if(err)
        reject(err);
      else{
        // Here all the items are retrieved, you can store them in a database table
        //  for each item received make your web call and store it into another variable or result set 
        // suggestion is to make the database transaction only after you have retrieved all your data 
        // other wise it will include database rollback which will depend on the database which you are using
        // after all this is done resolve the promise with the returning value
        resolve(results); 
      }
    });
  })    
}

I have not tested it but something like this should work. If problem persists let me know in comments.

Solution 2:

You can remove promises and try the same thing with callback as they are easier to follow and will make sense to the programmers who are familiar with structural languages.

Courtyard

Thank you for taking your time to respond me. In your answer I don't see the promise used anywhere and it's a request to use it, as the title says. – Inconsonant

@Inconsonant You changed the tags after I answered the question. Updated the answer. It should work I guess. – Courtyard

I've edited the scenario. Could you please update your answer if the case? Thank you for your time. – Inconsonant

I have tried to be elaborate with comments. Please mention specifically where you are stuck then maybe I will update the answer accordingly? – Courtyard

One short question, why getList doesn't uses Promise? Any reason for that? – Inconsonant

Because it is only being invoked inside the new Promise(** context **)'s context. It helps the promise to resolve itself depending on the condition which you decide. It's like putting data in and getting the result back, getting the promise resolved in this case. – Courtyard

Answer 2

Looking at your problem, I have created a code that would loop through promises. and would only procede if there is more data to be fetched, the stored data would still be available in an array. I hope this help. Dont forget to mark if it helps.

let fetchData = (offset = 0, limit= 10) => {
    let addresses = [...Array(100).keys()];
    return Promise.resolve(addresses.slice(offset, offset + limit))
}
// o => offset & l => limit
let o = 0, l = 10;
let results = [];
let process = p => {
  if (!p) return p;
  return p.then(data => {
    // Process with data here;
    console.log(data);
    // increment the pagination
    o += l;
    results = results.concat(data);
    // while there is data equal to limit set then fetch next page 
    // otherwise return the collected result
    return (data.length == l)? process(fetchAddress(o, l)).then(data => data) : results;
  })
}
process(fetchAddress(o, l))
.then(data => {
    // All the fetched data will be here
}).catch(err => {
// Handle Error here.
// All the retrieved data from database will be available in "results" array 
});

if You want to do it more often I have also created a gist for reference. If You dont want to use any global variable, and want to do it in very functional way. You can check this example. However it requires little more complication.

Eurythermal