async.waterfall in a For Loop in Node.js

Question

When using async.waterfall within a for loop, it appears that the for loop iterates before the nested async.waterfall finishes all its steps. How can this be avoided?

for(var i = 0; i < users.length; i++ ) {

    console.log(i)

    async.waterfall([
        function(callback) {
            callback(null, 'one', 'two');
        },
        function(arg1, arg2, callback) {
          // arg1 now equals 'one' and arg2 now equals 'two'
            callback(null, 'three');
        },
        function(arg1, callback) {
            // arg1 now equals 'three'
            callback(null, 'done');
        }
    ], function (err, result) {
        // result now equals 'done'
        console.log('done')
    });


}

Output

0
1
2
3
4
done
done
done
done
done

Desired Output

0
done
1
done
2
done
3
done
4
done

Vespucci

The for-loop is a block of sync work, which posts async work on the eventloop queue -- so why does it need to work in a different order? Maybe a clarification of your actual problem would get you a better answer. – Inert

Just don't use a for loop, but rather async.map (or whatever you need) – Marijuana

Answer 1

You can use like this with async's forEachLimit

var async = require("async")
var users = []; // Initialize user array or get it from DB

async.forEachLimit(users, 1, function(user, userCallback){

    async.waterfall([
        function(callback) {
            callback(null, 'one', 'two');
        },
        function(arg1, arg2, callback) {
            // arg1 now equals 'one' and arg2 now equals 'two'
            callback(null, 'three');
        },
        function(arg1, callback) {
            // arg1 now equals 'three'
            callback(null, 'done');
        }
    ], function (err, result) {
        // result now equals 'done'
        console.log('done')
        userCallback();
    });


}, function(err){
    console.log("User For Loop Completed");
});

Counterspy

Thanks! Here's a follow up question: #31816417 – Vespucci

Answer 2

You would need to apply a recursive pattern. My suggestion would be something like this

function foo(properties, callback){
/*Initialize inputs w/ useful defaults.  Using a properties object simply because it looks nicer, you can use individual values if you'd like.*/
        properties.start = properties.start || 0;
        properties.end = properties.end || 10; //or any default length
        properties.data = properties.data || [];

        async.waterfall([
          function(callback) {
        },
        function(arg1, arg2, callback) {
          /* arg1 now equals 'one' and arg2 now equals 'two' you can do something with that before continuing processing */
        },
        function(arg1, callback) {
            /* arg1 now equals 'three', you can do something with that before continuing processing */

        }
    ], function (err, result) {
        // result now equals 'done' 
        // now that we have fully processed one record, we need to either finish or recurse to the next element.  

     if(properties.index >= properties.end){
       callback();
     }
     else{
       properties.index++;
       foo(properties, callback);
     }

    })}

I've passed the callback into each of the function callbacks. You could chose to end the recursion early this way if you wanted to, or you could do anything elsein that spot that you wanted to. This is similar to a question that I asked recently: Patterns for asynchronous but sequential requests which has some other interesting solutions to the problem as well.

Minion

Why use recursion when the async library is already available? – Marijuana

Personal preference. The async library might actually be more performant (this example is pretty inefficient but I wanted to illustrate the concept of the recursion more than I wanted to show off my ability) if it has been optimized on the language-level. Having not actually dug into the code in that library, it's hard for me to say. – Minion

I doubt async.js would be more performant, and the library code is horrible indeed, but the code that uses it would be far more readable. – Marijuana

I don't know about more readable, this is simply one call to perform the task of looping through an array. That said, the onus is on the developer to maintain this and see it through any future changes. That's definitely a good enough reason to use the async library. TBH, I didn't know the async library could do this on a data-structure level like this. I've not taken the time to learn it yet, as I mentioned earlier. – Minion