nodejs async: multiple dependant HTTP API calls

Question

I'm working on a project that involves making multiple HTTP GET requests to different APIs, each requiring information from the last. I'm trying to avoid nested-callaback-and-counter-hell, and have been trying to get it working with the async module.

This is what I need to do: I have an array of 1..n course identifiers (['2014/summer/iat/100/d100', '2014/spring/bisc/372/d100']). For each course in the array, I need to fetch its course outline via a HTTP GET.

The resulting outline looks something like this:

{
  "info": {
    "nodePath": "2014/spring/bisc/372/d100",
    "number": "372",
    "section": "D100",
    "title": "Special Topics in Biology",
    "term": "Spring 2014",
    "description": "Selected topics in areas not currently offered...",
    "name": "BISC 372 D100",
    "dept": "BISC",
 },
 "instructor": [
    {
      "lastName": "Smith",
      "commonName": "Frank",
      "phone": "1 555 555-1234",
      "email": "[email protected]",
      "name": "Frank Smith",
      "roleCode": "PI"
    },
    {
      "lastName": "Doe",
      "commonName": "John",
      "phone": "1 555 555-9876",
      "email": "[email protected]",
      "name": "John Doe",
      "roleCode": "PI"
    }
  ]
}

(a bunch of non-relevant fields omitted)

Each outline object may contain an instructor property which is an array of 0..n instructor objects for the course. For each member of the instructor array, I need to then call another API to get additional data. When that call returns, I need to insert it into the right instructor object.

Finally, when everything is done, the data gets passed to a template for express to render and return to the client.

I've tried getting this working using async and had some success with async.waterfall when doing a proof-of-concept with only getting one of the instructor profiles (e.g. not looping over the array, just getting instructor[0]). The async module's docs are comprehensive, but pretty dense and I'm having a hard time determining what I actually need to do. I had a Frankenstein combination of various nested async calls which still didn't work.

I don't really care how I accomplish the task - flow-control, promises, magic pixie dust, whatever. Any hints greatly appreciated.

Liable

Answer 1

Using Q for promises, you can probably do something like this:

return Q
.all(course_ids.map(function(course) {
    return HTTP.GET(course); // Assuming this returns a promise
}))
.then(function(course_data) {
    var instructors = [];

    course_data.forEach(function(course) {
        var p = Q
            .all(course.instructor.map(function(instructor) {
                return HTTP.GET(instructor.id);
            }))
            .then(function(instructors) {
                course.instructors_data = instructors;

                return course;
            });

        promises.push(p);
    });

    return Q.all(promises);
});

Will resolve with an array containing the courses, each of which contains an array of instructor data in its instructors_data value.

Factfinding

I've decided to try promises for this. I've got the first step, fetching the outlines for the courses, working; that was stupid-easy. I'm having trouble figuring out how to promisify the next part, grabbing the profiles for each instructor for a course and stuffing them into the right place. I've got it working the longhand way, using callbacks and counters. gist.github.com/grahamb/11306874 – Liable

Answer 2

You could use async.each(), which would do the API requests in parallel (assuming there is no concurrent API request limits on the server side, if that is the case, use async.eachLimit() instead):

async.each(instructors, function(instructor, callback) {

  // call API here, store result on `instructor`,
  // and call `callback` when done

}, function(err){
  if (err)
    console.log('An error occurred while processing instructors');
  else
    console.log('All instructors have been processed successfully');
});

Lewak