What is a simple implementation of async.waterfall?

Question

I'm using some functions from the async library, and want to make sure I understand how they're doing things internally; however, I'm stuck on async.waterfall (implementation here). The actual implementation uses other functions from within the library, and without much experience, I'm finding it difficult to follow.

Could someone, without worrying about optimization, provide a very simple implementation that achieves waterfall's functionality? Probably something comparable to this answer.

From the docs, waterfall's description:

Runs the tasks array of functions in series, each passing their results to the next in the array. However, if any of the tasks pass an error to their own callback, the next function is not executed, and the main callback is immediately called with the error.

An example:

async.waterfall([
    function(callback) {
        callback(null, 'one', 'two');
    },
    function(arg1, arg2, callback) {
      // arg1 now equals 'one' and arg2 now equals 'two'
        callback(null, 'three');
    },
    function(arg1, callback) {
        // arg1 now equals 'three'
        callback(null, 'done');
    }
], function (err, result) {
    // result now equals 'done'    
});

Outgo

Answer 1

Well, here is a simple implementation for chaining functions by queuing them.

First of all, the function:

function waterfall(arr, cb){} // takes an array and a callback on completion

Now, we need to keep track of the array and iterate it:

function waterfall(arr, cb){
    var fns = arr.slice(); // make a copy
}

Let's start with handling passed and empty array, by adding an extra parameter so we can pass results around called result:

function waterfall(arr, cb, result){ // result is the initial result
    var fns = arr.slice(); // make a copy
    if(fns.length === 0){
        process.nextTick(function(){ // don't cause race conditions
            cb(null, result); // we're done, nothing more to do
        });
    }
}

That's nice:

waterfall([], function(err, data){
    console.log("Done!");
});

Now, let's handle actually having stuff in:

function waterfall(arr, cb, result){ // result is the initial result
    var fns = arr.slice(1); // make a copy, apart from the first element
    if(!arr[0]){ // if there is nothing in the first position
        process.nextTick(function(){ // don't cause race conditions
            cb(null, result); // we're done, nothing more to do
        });
        return;
    }
    var first = arr[0]; // get the first function
    first(function(err, data){ // invoke it
         // when it is done
         if(err) return cb(err); // early error, terminate entire call
         // perform the same call, but without the first function
         // and with its result as the result
         waterfall(fns, cb, data); 
    });
}

And that's it! We overcome the fact we can't loop with callbacks by using recursion basically. Here is a fiddle illustrating it.

It's worth mentioning that if we were implementing it with promises we could have used a for loop.

Krissie

Here is a fiddle that allows functions to pass a varying number of arguments along – Outgo

Answer 2

For those who like to keep it short :

function waterfall(fn, done){
   fn.length ? fn.shift()(function(err){ err ? done(err) : waterfall(fn, done) }) :  done();
}

Piggin

will it work with extra args as suggested in question ? – Ilke