Handling interdependent and/or layered asynchronous calls
Asked Answered
T

6

11

As an example, suppose I want to fetch a list of files from somewhere, then load the contents of these files and finally display them to the user. In a synchronous model, it would be something like this (pseudocode):

var file_list = fetchFiles(source);

if (!file_list) {
    display('failed to fetch list');

} else {
        for (file in file_list) { // iteration, not enumeration
        var data = loadFile(file);

        if (!data) {
            display('failed to load: ' + file);
        } else {
            display(data);
        }
    }
}

This provides decent feedback to the user and I can move pieces of code into functions if I so deem necessary. Life is simple.

Now, to crush my dreams: fetchFiles() and loadFile() are actually asynchronous. The easy way out is to transform them into synchronous functions. But this is not good if the browser locks up waiting for calls to complete.

How can I handle multiple interdependent and/or layered asynchronous calls without delving deeper and deeper into an endless chain of callbacks, in classic reductio ad spaghettum fashion? Is there a proven paradigm to cleanly handle these while keeping code loosely coupled?

Twentytwo answered 18/1, 2013 at 21:14 Comment(6)
Do the two async functions take callbacks as their 2nd arguments?Edgington
Do you want to load all the files in parallel? And display data / alert error separately for each?Resnatron
Could you please comment on the existing answers as to why they are not acceptable. Feedback can help us help you.Mitrailleuse
@BenFelda I will investigate the existing answers further, but when I started the bounty only the jQuery answer was available.Twentytwo
@Resnatron This is really just an example, what I want is a clean way to structure code with lots asynchronous calls.Twentytwo
I'd say you probably want to use events. It's a great model for loosely coupling asynchronous code.Bryology
D
6

Deferreds are really the way to go here. They capture exactly what you (and a whole lot of async code) want: "go away and do this potentially expensive thing, don't bother me in the meantime, and then do this when you get back."

And you don't need jQuery to use them. An enterprising individual has ported Deferred to underscore, and claims you don't even need underscore to use it.

So your code can look like this:

function fetchFiles(source) {
    var dfd = _.Deferred();

    // do some kind of thing that takes a long time
    doExpensiveThingOne({
        source: source,
        complete: function(files) {
            // this informs the Deferred that it succeeded, and passes
            // `files` to all its success ("done") handlers
            dfd.resolve(files);

            // if you know how to capture an error condition, you can also
            // indicate that with dfd.reject(...)
        }
    });

    return dfd;
}

function loadFile(file) {
    // same thing!
    var dfd = _.Deferred();

    doExpensiveThingTwo({
        file: file,
        complete: function(data) {
            dfd.resolve(data);
        }
    });

    return dfd;
}

// and now glue it together
_.when(fetchFiles(source))
.done(function(files) {
    for (var file in files) {
        _.when(loadFile(file))
        .done(function(data) {
            display(data);
        })
        .fail(function() {
            display('failed to load: ' + file);
        });
    }
})
.fail(function() {
    display('failed to fetch list');
});

The setup is a little wordier, but once you've written the code to handle the Deferred's state and stuffed it off in a function somewhere you won't have to worry about it again, you can play around with the actual flow of events very easily. For example:

var file_dfds = [];
for (var file in files) {
    file_dfds.push(loadFile(file));
}

_.when(file_dfds)
.done(function(datas) {
    // this will only run if and when ALL the files have successfully
    // loaded!
});
Dive answered 23/1, 2013 at 21:21 Comment(3)
+1. Though it should be noted that this will schedule each loadFile() at nearly the same time, causing parallel downloads ... which depending on the situation may not be desired.Leban
The beauty of deferreds is that you can stagger them by time or progress, or even run them serially, without touching either the code that performs the download or the code that handles the data.Dive
But to create the promise objects, you would typically also perform the Ajax call is what I meant.Leban
B
3

Events

Maybe using events is a good idea. It keeps you from creating code-trees and de-couples your code.

I've used bean as the framework for events.

Example pseudo code:

// async request for files
function fetchFiles(source) {

    IO.get(..., function (data, status) {
        if(data) {
            bean.fire(window, 'fetched_files', data);
        } else {
            bean.fire(window, 'fetched_files_fail', data, status);
        } 
    });

}

// handler for when we get data
function onFetchedFiles (event, files) {
    for (file in files) { 
        var data = loadFile(file);

        if (!data) {
            display('failed to load: ' + file);
        } else {
            display(data);
        }
    }
}

// handler for failures
function onFetchedFilesFail (event, status) {
    display('Failed to fetch list. Reason: ' + status);
}

// subscribe the window to these events
bean.on(window, 'fetched_files', onFetchedFiles);
bean.on(window, 'fetched_files_fail', onFetchedFilesFail);

fetchFiles();

Custom events and this kind of event handling is implemented in virtually all popular JS frameworks.

Burley answered 29/1, 2013 at 19:4 Comment(0)
U
2

Sounds like you need jQuery Deferred. Here is some untested code that might help point you in the right direction:

$.when(fetchFiles(source)).then(function(file_list) { 
  if (!file_list) {
    display('failed to fetch list');
  } else {
    for (file in file_list) {
      $.when(loadFile(file)).then(function(data){
        if (!data) {
          display('failed to load: ' + file);
        } else {
          display(data);
        }
      });
    }
  }
});

I also found another decent post which gives a few uses cases for the Deferred object

Uzzia answered 18/1, 2013 at 21:52 Comment(2)
one note - you'll have to modify "fetchFiles" and "loadFile" so that they return a jQuery Deferred object. $.ajax does this, for exampleUzzia
you can use done and fail to handle success and failure separately, instead of checking the argument passed to thenDive
S
2

If you do not want to use jQuery, what you could use instead are web workers in combination with synchronous requests. Web workers are supported across every major browser with the exception of any Internet Explorer version before 10.

Web Worker browser compatability

Basically, if you're not entirely certain what a web worker is, think of it as a way for browsers to execute specialized JavaScript on a separate thread without impacting the main thread (Caveat: On a single-core CPU, both threads will run in an alternating fashion. Luckily, most computers nowadays come equipped with dual-core CPUs). Usually, web workers are reserved for complex computations or some intense processing task. Just keep in mind that any code within the web worker CANNOT reference the DOM nor can it reference any global data structures that have not been passed to it. Essentially, web workers run independent of the main thread. Any code that the worker executes should be kept separate from the rest of your JavaScript code base, within its own JS file. Furthermore, if the web workers need specific data in order to properly work, you need to pass that data into them upon starting them up.

Yet another important thing worth noting is that any JS libraries that you need to use to load the files will need to be copied directly into the JavaScript file that the worker will execute. That means these libraries should first be minified(if they haven't been already), then copied and pasted into the top of the file.

Anyway, I decided to write up a basic template to show you how to approach this. Check it out below. Feel free to ask questions/criticize/etc.

On the JS file that you want to keep executing on the main thread, you want something like the following code below in order to invoke the worker.

function startWorker(dataObj)
{
    var message = {},
        worker;

      try
      {
        worker = new Worker('workers/getFileData.js');
      } 
      catch(error) 
      {
        // Throw error
      }

    message.data = dataObj;

    // all data is communicated to the worker in JSON format
    message = JSON.stringify(message);

    // This is the function that will handle all data returned by the worker
    worker.onMessage = function(e)
    {
        display(JSON.parse(e.data));
    }

    worker.postMessage(message);
}

Then, in a separate file meant for the worker (as you can see in the code above, I named my file getFileData.js), write something like the following...

function fetchFiles(source)
{
    // Put your code here
    // Keep in mind that any requests made should be synchronous as this should not
    // impact the main thread
}

function loadFile(file)
{
    // Put your code here
    // Keep in mind that any requests made should be synchronous as this should not
    // impact the main thread
}

onmessage = function(e)
{
    var response = [],
        data = JSON.parse(e.data),
        file_list = fetchFiles(data.source),
        file, fileData;

    if (!file_list) 
    {
        response.push('failed to fetch list');
    }
    else 
    {
        for (file in file_list) 
        { // iteration, not enumeration
            fileData = loadFile(file);

            if (!fileData) 
            {
                response.push('failed to load: ' + file);
            } 
            else 
            {
                response.push(fileData);
            }
        }
    }

    response = JSON.stringify(response);

    postMessage(response);

    close();
}

PS: Also, I dug up another thread which would better help you understand the pros and cons of using synchronous requests in combination with web workers.

Stack Overflow - Web Workers and Synchronous Requests

Simeon answered 23/1, 2013 at 17:27 Comment(0)
E
1

async is a popular asynchronous flow control library often used with node.js. I've never personally used it in the browser, but apparently it works there as well.

This example would (theoretically) run your two functions, returning an object of all the filenames and their load status. async.map runs in parallel, while waterfall is a series, passing the results of each step on to the next.

I am assuming here that your two async functions accept callbacks. If they do not, I'd require more info as to how they're intended to be used (do they fire off events on completion? etc).

async.waterfall([
  function (done) {
    fetchFiles(source, function(list) {
      if (!list) done('failed to fetch file list');
      else done(null, list);
    });
    // alternatively you could simply fetchFiles(source, done) here, and handle
    // the null result in the next function.
  },

  function (file_list, done) {
    var loadHandler = function (memo, file, cb) {
      loadFile(file, function(data) {
        if (!data) {
          display('failed to load: ' + file);
        } else {
          display(data);
        }
        // if any of the callbacks to `map` returned an error, it would halt 
        // execution and pass that error to the final callback.  So we don't pass
        // an error here, but rather a tuple of the file and load result.
        cb(null, [file, !!data]);
      });
    };
    async.map(file_list, loadHandler, done);
  }
], function(err, result) {
  if (err) return display(err);
  // All files loaded! (or failed to load)
  // result would be an array of tuples like [[file, bool file loaded?], ...]
});

waterfall accepts an array of functions and executes them in order, passing the result of each along as the arguments to the next, along with a callback function as the last argument, which you call with either an error, or the resulting data from the function.

You could of course add any number of different async callbacks between or around those two, without having to change the structure of the code at all. waterfall is actually only 1 of 10 different flow control structures, so you have a lot of options (although I almost invariably end up using auto, which allows you to mix parallel and series execution in the same function via a Makefile like requirements syntax).

Edgington answered 23/1, 2013 at 18:8 Comment(0)
M
1

I had this issue with a webapp I'm working on and here's how I solved it (with no libraries).

Step 1: Wrote a very lightweight pubsub implementation. Nothing fancy. Subscribe, Unsubscribe, Publish and Log. Everything (with comments) adds up 93 lines of Javascript. 2.7kb before gzip.

Step 2: Decoupled the process you were trying to accomplish by letting the pubsub implementation do the heavy lifting. Here's an example:

// listen for when files have been fetched and set up what to do when it comes in
pubsub.notification.subscribe(
    "processFetchedResults", // notification to subscribe to
    "fetchedFilesProcesser", // subscriber

    /* what to do when files have been fetched */ 
    function(params) {

        var file_list = params.notificationParams.file_list;

        for (file in file_list) { // iteration, not enumeration
        var data = loadFile(file);

        if (!data) {
            display('failed to load: ' + file);
        } else {
            display(data);
        }
    }
);    

// trigger fetch files 
function fetchFiles(source) {

   // ajax call to source
   // on response code 200 publish "processFetchedResults"
   // set publish parameters as ajax call response
   pubsub.notification.publish("processFetchedResults", ajaxResponse, "fetchFilesFunction");
}

Of course this is very verbose in the setup and scarce on the magic behind the scenes. Here's some technical details:

  1. I'm using setTimeout to handle triggering subscriptions. This way they run in a non-blocking fashion.

  2. The call is effectively decoupled from the processing. You can write a different subscription to the notification "processFetchedResults" and do multiple things once the response comes through (for example logging and processing) while keeping them in very separate, tiny and easily-managed code blocks.

  3. The above code sample doesn't address fallbacks or run proper checks. I'm sure it will require a bit of tooling to get to production standards. Just wanted to show you how possible it is and how library-independent your solution can be.

Cheers!

Mukluk answered 29/1, 2013 at 16:30 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.