How to sync JavaScript callbacks?

Question

I've been developing in JavaScript for quite some time but net yet a cowboy developer, as one of the many things that always haunts me is synching JavaScript's callbacks.

I will describe a generic scenario when this concern will be raised: I have a bunch of operations to perform multiple times by a for loop, and each of the operations has a callback. After the for loop, I need to perform another operation but this operation can only execute successfully if all the callbacks from the for loop are done.

Code Example:

 for ... in ... {
   myFunc1(callback); // callbacks are executed asynchly
 }

 myFunc2(); // can only execute properly if all the myFunc1 callbacks are done

Suggested Solution:

Initiate a counter at the beginning of the loop holding the length of the loop, and each callback decrements that counter. When the counter hits 0, execute myFunc2. This is essentially to let the callbacks know if it's the last callback in sequence and if it is, call myFunc2 when it's done.

Problems:

1. A counter is needed for every such sequence in your code, and having meaningless counters everywhere is not a good practice.
2. If you recall how thread conflicts in classical synchronization problem, when multiple threads are all calling var-- on the same var, undesirable outcomes would occur. Does the same happen in JavaScript?

Ultimate Question:

Is there a better solution?

Heydon

I think Promises will help you out here. Take a look at here or the other SO question where the answer is Promises. – Samara

Should it happen in parallel or one after the other? – Thermit

If he wanted it one after the other then he wouldn't be using async tasks. – Samara

@Samara Not necessarily, asynch tasks may need to be serialized too. – Thermit

@Jack the callbacks happen in parallel, but the operation after needs to happen after all the callbacks are done. – Heydon

possible duplicate of Design pattern for managing multiple asynchronous JavaScript operations – Metalliferous

Answer 1

The good news is that JavaScript is single threaded; this means that solutions will generally work well with "shared" variables, i.e. no mutex locks are required.

If you want to serialize asynch tasks, followed by a completion callback you could use this helper function:

function serializeTasks(arr, fn, done)
{
    var current = 0;

    fn(function iterate() {
        if (++current < arr.length) {
            fn(iterate, arr[current]);
        } else {
            done();
        }
    }, arr[current]);
}

The first argument is the array of values that needs to be passed in each pass, the second argument is a loop callback (explained below) and the last argument is the completion callback function.

This is the loop callback function:

function loopFn(nextTask, value) {
    myFunc1(value, nextTask);
}

The first argument that's passed is a function that will execute the next task, it's meant to be passed to your asynch function. The second argument is the current entry of your array of values.

Let's assume the asynch task looks like this:

function myFunc1(value, callback)
{
  console.log(value);
  callback();
}

It prints the value and afterwards it invokes the callback; simple.

Then, to set the whole thing in motion:

serializeTasks([1,2, 3], loopFn, function() {
    console.log('done');
});

Demo

To parallelize them, you need a different function:

function parallelizeTasks(arr, fn, done)
{
    var total = arr.length,
    doneTask = function() {
      if (--total === 0) {
        done();
      }
    };

    arr.forEach(function(value) {
      fn(doneTask, value);
    });
}

And your loop function will be this (only parameter name changes):

function loopFn(doneTask, value) {
    myFunc1(value, doneTask);
}

Demo

Cerelia

Thx @Jack, I'll need to go through this in detail later, have to sleep now :) – Heydon

@Heydon I've added a parallel version too :) – Thermit

Thanks for the effort and thank everybody for making suggestions. All posts are great but only this post answered my 2nd question. Most of the suggested solutions involved the use of a counter of some sort, but the promise sounds promising. I think I'll explore that a bit more :) – Heydon

@Jack, thank you for warping my brain and showing me how to solve my problem. – Ideology

Answer 2

The second problem is not really a problem as long as every one of those is in a separate function and the variable is declared correctly (with var); local variables in functions do not interfere with each other.

The first problem is a bit more of a problem. Other people have gotten annoyed, too, and ended up making libraries to wrap that sort of pattern for you. I like async. With it, your code might look like this:

async.each(someArray, myFunc1, myFunc2);

It offers a lot of other asynchronous building blocks, too. I'd recommend taking a look at it if you're doing lots of asynchronous stuff.

Dunlavy

Interesting library; does it always assume the function takes two parameters, the second being the callback? – Thermit

Thanks for sharing, but I think the 2nd problem MIGHT be a problem because all of the functions are decrementing the same shared variable (in my case, the counter) but not their own local variables. – Heydon

@Xavier_Ex: As long as your counter is declared in the same function as the for loop, it's the only counter (with that name) in the function, and when you refer to it in the callback, you're referring to that counter, you should be fine. – Dunlavy

@Dunlavy thanks, but I don't quite understand your explanation. Even though the counter is a local variable in a function, but it is passed to all of the callbacks functions so they are essentially modifying a shared variable. Would you mind making an example of what will be a problem then? – Heydon

@Xavier_Ex: Sure, it's shared among all of those callbacks, but it won't get mixed up with other functions. There is no problem; I only called it a 'problem' because you labeled it as a 'problem' in your question. – Dunlavy

Answer 3

You can achieve this by using a jQuery deferred object.

var deferred = $.Deferred();
var success = function () {
    // resolve the deferred with your object as the data
    deferred.resolve({
        result:...;
    });
};

Millimicron

Thanks for sharing, I may need to take a look at how jquery 'deferred' works first. – Heydon

I think this is the same solution as using promises. Need to read up promise then. – Heydon

Answer 4

With this helper function:

function afterAll(callback,what) {
  what.counter = (what.counter || 0) + 1;
  return function() {
    callback(); 
    if(--what.counter == 0) 
      what();
  };
}

your loop will look like this:

function whenAllDone() { ... }
for (... in ...) {
  myFunc1(afterAll(callback,whenAllDone)); 
}

here afterAll creates proxy function for the callback, it also decrements the counter. And calls whenAllDone function when all callbacks are complete.

Consol

Thanks for sharing, but this is ultimately the same way with counter. I don't know about the general public but I do find meaningless counters annoying... – Heydon

@Heydon I do not understand your comment, any solution of the task will use counters of some sort. Consider afterAll as library function and use it places where you need it. – Consol

yes that's what I want to find out, if using counter is the only way. I know some concurrency problems can be solved by co-routines thus want to explore the possibilities here. Ultimately counter is not my preferred solution, be it the only way or not. – Heydon

Answer 5

single thread is not always guaranteed. do not take it wrong.

Case 1: For example, if we have 2 functions as follows.

var count=0;
function1(){
  alert("this thread will be suspended, count:"+count);
}
function2(){
  //anything
  count++;
  dump(count+"\n");
}

then before function1 returns, function2 will also be called, if 1 thread is guaranteed, then function2 will not be called before function1 returns. You can try this. and you will find out count is going up while you are being alerted.

Case 2: with Firefox, chrome code, before 1 function returns (no alert inside), another function can also be called.

So a mutex lock is indeed needed.

Buerger

Answer 6

There are many, many ways to achieve this, I hope these suggestions help!

First, I would transform the callback into a promise! Here is one way to do that:

function aPromise(arg) {
    return new Promise((resolve, reject) => {
        aCallback(arg, (err, result) => {
            if(err) reject(err);
            else resolve(result);
        });
    })
}

Next, use reduce to process the elements of an array one by one!

const arrayOfArg = ["one", "two", "three"];
const promise = arrayOfArg.reduce(
    (promise, arg) => promise.then(() => aPromise(arg)), // after the previous promise, return the result of the aPromise function as the next promise
    Promise.resolve(null) // initial resolved promise
    );
promise.then(() => {
    // carry on
});

If you want to process all elements of an array at the same time, use map an Promise.all!

const arrayOfArg = ["one", "two", "three"];
const promise = Promise.all(arrayOfArg.map(
    arg => aPromise(arg)
));
promise.then(() => {
    // carry on
});

If you are able to use async / await then you could just simply do this:

const arrayOfArg = ["one", "two", "three"];
for(let arg of arrayOfArg) {
    await aPromise(arg); // wow
}

// carry on

You might even use my very cool synchronize-async library like this:

const arrayOfArg = ["one", "two", "three"];
const context = {}; // can be any kind of object, this is the threadish context

for(let arg of arrayOfArg) {
    synchronizeCall(aPromise, arg); // synchronize the calls in the given context
}

join(context).then(() => { // join will resolve when all calls in the context are finshed
    // carry on
});

And last but not least, use the fine async library if you really don't want to use promises.

const arrayOfArg = ["one", "two", "three"];
async.each(arrayOfArg, aCallback, err => {
    if(err) throw err; // handle the error!
    // carry on
});

Guttery