Make Requests in Sequential Order Node.js

Question

If I need to call 3 http API in sequential order, what would be a better alternative to the following code:

http.get({ host: 'www.example.com', path: '/api_1.php' }, function(res) { 
  res.on('data', function(d) { 

    http.get({ host: 'www.example.com', path: '/api_2.php' }, function(res) { 
      res.on('data', function(d) { 

        http.get({ host: 'www.example.com', path: '/api_3.php' }, function(res) { 
          res.on('data', function(d) { 


          });
        });
        }
      });
    });
    }
  });
});
}

Coeval

other than cleaning that up, i don't think you can do better than that. – Bistre

Why do they need to be in order? – Enthuse

@Enthuse You might need some data from api_1 before you know what to send to api_2 – Eveleen

helpful answer: https://mcmap.net/q/37476/-javascript-promise-node-js – Eames

It's worth mentioning that Futures is pretty deprecated, consider using a newer library like Bluebird or Q. – Overexert

The title and question contradict each other. You're not describing a synchronous request in your question but a sequence of requests, which would normally each occur asynchronously. Big difference - a synchronous call blocks, and a sequence of asynchronous actions does not block (block the UI, block the server from handling other requests). There is an answer below mentioning the sync-request library, which is a good answer to the title of this question, but not an answer for what the question's code implies. The answer below about Promises is a better answer for that. Which did you mean? – Romilda

Answer 1

Using deferreds like Futures.

var sequence = Futures.sequence();

sequence
  .then(function(next) {
     http.get({}, next);
  })
  .then(function(next, res) {
     res.on("data", next);
  })
  .then(function(next, d) {
     http.get({}, next);
  })
  .then(function(next, res) {
    ...
  })

If you need to pass scope along then just do something like this

  .then(function(next, d) {
    http.get({}, function(res) {
      next(res, d);
    });
  })
  .then(function(next, res, d) { })
    ...
  })

Enthuse

Please try IcedCoffeScript to which provides await and defer for nodejs. – Ghostwrite

Is this non blocking? I mean it is blocking for the next function in line but this won't block execution of other async functions, will it? – Cowans

Yes, deferred methods are non-blocking / async. – Riendeau

the ES6 Promise API should effectively replace this, even according to the author of "Futures" – Game

Futures is very old and deprecated. See q instead. – Kiersten

Answer 2

The actual need in 2023 is handled by natives promises in almost all cases. The way it generally works, is that almost any callback-based API can be turned into a promise (pretty trivially in normal code, but also via util.promisify in node core: https://nodejs.org/api/util.html#utilpromisifyoriginal)

Promises can be used to control flow in node via async/await to make things look much more desirable:

Basic example

// async here tells the javascript runtime that whats returned here
// is a promise, and we'll also be encountering async/await syntax
// inside this block
async function main() {
  // in serial
  const result1 = await longRunningTask1();
  const result2 = await longRunningTask2();
  const result3 = await longRunningTask1();
  const result4 = await longRunningTask2();

  // in parallel, can also use .allSettled to handle failures
  // gracefully, .race to only take the first return
  const results = await Promise.all([
    longRunningTask1(), 
    longRunningTask2(), 
    longRunningTask3(), 
    longRunningTask4(), 
  ]);
}

There are also several libraries these days for managing limiting concurrent promises (p-limit and bottleneck are both options circa 2023).

Be aware, top-level await is also something that is still actively being worked on. At this moment in time, to use "top level await" which means a block that doesn't require an async function wrapper, you need to use the mjs extension, the support of which is still pretty dubious, especially in the mixed ESM/CommonJS ecosystem that currently exists.

For posterity, old answer...

I like Raynos' solution as well, but I prefer a different flow control library.

https://github.com/caolan/async

Depending on whether you need the results in each subsequent function, I'd either use series, parallel, or waterfall.

[Series][1] when they have to be serially executed, but you don't necessarily need the results in each subsequent function call.

[Parallel][2] if they can be executed in parallel, you don't need the results from each during each parallel function, and you need a callback when all have completed.

[Waterfall][3] if you want to morph the results in each function and pass to the next

endpoints = 
 [{ host: 'www.example.com', path: '/api_1.php' },
  { host: 'www.example.com', path: '/api_2.php' },
  { host: 'www.example.com', path: '/api_3.php' }];

async.mapSeries(endpoints, http.get, function(results){
    // Array of results
});

[1]: https://github.com/caolan/async#series [2]: https://github.com/caolan/async#parallel [3]: https://github.com/caolan/async#waterfall

Steiner

var http = require('http'); – Spigot

Hah. example.com is actually a domain designed for this sort of thing. Wow. – Mastigophoran

The async.series code doesn't work, at least as of async v0.2.10. series() only takes up to two arguments and will execute the first argument's elements as functions, so async throws an error trying to execute the objects as functions. – Oospore

You can do something similar to what is intended with this code using forEachAsync (github.com/FuturesJS/forEachAsync). – Oospore

This does exactly what I wanted. Thank you! – Pyrogenous

Answer 3

sync-request

By far the most easiest one I've found and used is sync-request and it supports both node and the browser!

var request = require('sync-request');
var res = request('GET', 'http://google.com');
console.log(res.body.toString('utf-8'));

That's it, no crazy configuration, no complex lib installs, although it does have a lib fallback. Just works. I've tried other examples here and was stumped when there was much extra setup to do or installs didn't work!

Notes:

The example that sync-request uses doesn't play nice when you use res.getBody(), all get body does is accept an encoding and convert the response data. Just do res.body.toString(encoding) instead.

Representation

I found that sync-request is very slow.. I've ended up using another one github.com/dhruvbird/http-sync which is 10 times faster in my case. – Beldam

i haven't had any slow runs for it. This spawns a child process. How many cpus does your system use and what version of node are you using? I'd love to know to determine if I need to switch or not. – Representation

I agree with Filip, this is slow. – Damick

Same thing I've asked flip but got no response: How many cpus does your system use and what version of node are you using? – Representation

this uses a serious amount of CPU, not recommended for production use. – Alston

Do you have test cases and results to prove? In node, this module will create child processes so that it does not block the main event loop. In the browser, it simply blocks the main event loop. If you find that you're creating to many child processes, then you should be using async requests. You can promisify the request. If it's in the browser, it's not being cpu intensive, it's simply blocking the event loop as expected and to you it appears cpu intensive. Sync-request is useful for window.unload in the browser. – Representation

@YehudaClinton Node doesn't reserve 'res' for output. Plus, this method will return a response object. Which is why the variable is set to res, just short for response. Name it however you want, just helping you understand why it's named like so. – Representation

Totally agree with @jem I am node.js novice and sync-request is by far the easiest and straightforward way to get what we are looking for. – Farhi

Answer 4

You could do this using my Common Node library:

function get(url) {
  return new (require('httpclient').HttpClient)({
    method: 'GET',
      url: url
    }).finish().body.read().decodeToString();
}

var a = get('www.example.com/api_1.php'), 
    b = get('www.example.com/api_2.php'),
    c = get('www.example.com/api_3.php');

Boding

crap, i upvoted thinking it would work and it doesnt :( require(...).HttpClient is not a constructor – Alston

Answer 5

I'd use a recursive function with a list of apis

var APIs = [ '/api_1.php', '/api_2.php', '/api_3.php' ];
var host = 'www.example.com';

function callAPIs ( host, APIs ) {
  var API = APIs.shift();
  http.get({ host: host, path: API }, function(res) { 
    var body = '';
    res.on('data', function (d) {
      body += d; 
    });
    res.on('end', function () {
      if( APIs.length ) {
        callAPIs ( host, APIs );
      }
    });
  });
}

callAPIs( host, APIs );

edit: request version

var request = require('request');
var APIs = [ '/api_1.php', '/api_2.php', '/api_3.php' ];
var host = 'www.example.com';
var APIs = APIs.map(function (api) {
  return 'http://' + host + api;
});

function callAPIs ( host, APIs ) {
  var API = APIs.shift();
  request(API, function(err, res, body) { 
    if( APIs.length ) {
      callAPIs ( host, APIs );
    }
  });
}

callAPIs( host, APIs );

edit: request/async version

var request = require('request');
var async = require('async');
var APIs = [ '/api_1.php', '/api_2.php', '/api_3.php' ];
var host = 'www.example.com';
var APIs = APIs.map(function (api) {
  return 'http://' + host + api;
});

async.eachSeries(function (API, cb) {
  request(API, function (err, res, body) {
    cb(err);
  });
}, function (err) {
  //called when all done, or error occurs
});

Burdened

This is the method I employed as I have a variable list of requests to make (600 items and growing). That said, there is a problem with your code: the 'data' event will be emitted multiple times per request if the API output is greater than the chunk size. You want to "buffer" the data like so: var body = ''; res.on('data',function(data){ body += data; }).on('end',function(){ callback(body); if (APIs.length) callAPIs(host, APIs);}); – Unbacked

Updated. I was just wanting to show how the problem could be make simpler/more flexible through recursion. Personally I always use the request module for this sort of thing since it lets you skip the multiple callbacks with ease. – Burdened

@generalhenry, how would i do this if i wanted to use the request module? Can you offer a code snippet that achieves the above using request? – Dovetail

I added a request version and a request/async version. – Burdened

Answer 6

As of 2018 and using ES6 modules and Promises, we can write a function like that :

import { get } from 'http';

export const fetch = (url) => new Promise((resolve, reject) => {
  get(url, (res) => {
    let data = '';
    res.on('end', () => resolve(data));
    res.on('data', (buf) => data += buf.toString());
  })
    .on('error', e => reject(e));
});

and then in another module

let data;
data = await fetch('http://www.example.com/api_1.php');
// do something with data...
data = await fetch('http://www.example.com/api_2.php');
// do something with data
data = await fetch('http://www.example.com/api_3.php');
// do something with data

The code needs to be executed in an asynchronous context (using async keyword)

Thermosphere

This answer needs more upvotes, urgently. I would just tweak the return a bit to resolve([res, data]) which allows callers to get the return status code with res.statusCode. – Twostep

Answer 7

Another possibility is to set up a callback that tracks completed tasks:

function onApiResults(requestId, response, results) {
    requestsCompleted |= requestId;

    switch(requestId) {
        case REQUEST_API1:
            ...
            [Call API2]
            break;
        case REQUEST_API2:
            ...
            [Call API3]
            break;
        case REQUEST_API3:
            ...
            break;
    }

    if(requestId == requestsNeeded)
        response.end();
}

Then simply assign an ID to each and you can set up your requirements for which tasks must be completed before closing the connection.

const var REQUEST_API1 = 0x01;
const var REQUEST_API2 = 0x02;
const var REQUEST_API3 = 0x03;
const var requestsNeeded = REQUEST_API1 | REQUEST_API2 | REQUEST_API3;

Okay, it's not pretty. It is just another way to make sequential calls. It's unfortunate that NodeJS does not provide the most basic synchronous calls. But I understand what the lure is to asynchronicity.

Bayne

Answer 8

It seems solutions for this problem is never-ending, here's one more :)

// do it once.
sync(fs, 'readFile')

// now use it anywhere in both sync or async ways.
var data = fs.readFile(__filename, 'utf8')

http://alexeypetrushin.github.com/synchronize

Adjudicate

Though the library you linked DOES offer a solution to the OP's problem, in your example, fs.readFile is always sync. – Leishaleishmania

Nope, you can supply callback explicitly and use it as asynchronous version if you wish. – Adjudicate

the example was for http requests though, not file system communication. – Presumably

Answer 9

use sequenty.

sudo npm install sequenty

or

https://github.com/AndyShin/sequenty

very simple.

var sequenty = require('sequenty'); 

function f1(cb) // cb: callback by sequenty
{
  console.log("I'm f1");
  cb(); // please call this after finshed
}

function f2(cb)
{
  console.log("I'm f2");
  cb();
}

sequenty.run([f1, f2]);

also you can use a loop like this:

var f = [];
var queries = [ "select .. blah blah", "update blah blah", ...];

for (var i = 0; i < queries.length; i++)
{
  f[i] = function(cb, funcIndex) // sequenty gives you cb and funcIndex
  {
    db.query(queries[funcIndex], function(err, info)
    {
       cb(); // must be called
    });
  }
}

sequenty.run(f); // fire!

Lodestone

Answer 10

Using the request library can help minimize the cruft:

var request = require('request')

request({ uri: 'http://api.com/1' }, function(err, response, body){
    // use body
    request({ uri: 'http://api.com/2' }, function(err, response, body){
        // use body
        request({ uri: 'http://api.com/3' }, function(err, response, body){
            // use body
        })
    })
})

But for maximum awesomeness you should try some control-flow library like Step - it will also allow you to parallelize requests, assuming that it's acceptable:

var request = require('request')
var Step    = require('step')

// request returns body as 3rd argument
// we have to move it so it works with Step :(
request.getBody = function(o, cb){
    request(o, function(err, resp, body){
        cb(err, body)
    })
}

Step(
    function getData(){
        request.getBody({ uri: 'http://api.com/?method=1' }, this.parallel())
        request.getBody({ uri: 'http://api.com/?method=2' }, this.parallel())
        request.getBody({ uri: 'http://api.com/?method=3' }, this.parallel())
    },
    function doStuff(err, r1, r2, r3){
        console.log(r1,r2,r3)
    }
)

Hurleigh

Answer 11

There are lots of control flow libraries -- I like conseq (... because I wrote it.) Also, on('data') can fire several times, so use a REST wrapper library like restler.

Seq()
  .seq(function () {
    rest.get('http://www.example.com/api_1.php').on('complete', this.next);
  })
  .seq(function (d1) {
    this.d1 = d1;
    rest.get('http://www.example.com/api_2.php').on('complete', this.next);
  })
  .seq(function (d2) {
    this.d2 = d2;
    rest.get('http://www.example.com/api_3.php').on('complete', this.next);
  })
  .seq(function (d3) {
    // use this.d1, this.d2, d3
  })

Waits

Answer 12

This has been answered well by Raynos. Yet there have been changes in the sequence library since the answer has been posted.

To get sequence working, follow this link: https://github.com/FuturesJS/sequence/tree/9daf0000289954b85c0925119821752fbfb3521e.

This is how you can get it working after npm install sequence:

var seq = require('sequence').Sequence;
var sequence = seq.create();

seq.then(function call 1).then(function call 2);

Stubblefield

Answer 13

Here's my version of @andy-shin sequently with arguments in array instead of index:

function run(funcs, args) {
    var i = 0;
    var recursive = function() {
        funcs[i](function() {
            i++;
            if (i < funcs.length)
                recursive();
        }, args[i]);
    };
    recursive();
}

Houk

Answer 14

...4 years later...

Here is an original solution with the framework Danf (you don't need any code for this kind of things, only some config):

// config/common/config/sequences.js

'use strict';

module.exports = {
    executeMySyncQueries: {
        operations: [
            {
                order: 0,
                service: 'danf:http.router',
                method: 'follow',
                arguments: [
                    'www.example.com/api_1.php',
                    'GET'
                ],
                scope: 'response1'
            },
            {
                order: 1,
                service: 'danf:http.router',
                method: 'follow',
                arguments: [
                    'www.example.com/api_2.php',
                    'GET'
                ],
                scope: 'response2'
            },
            {
                order: 2,
                service: 'danf:http.router',
                method: 'follow',
                arguments: [
                    'www.example.com/api_3.php',
                    'GET'
                ],
                scope: 'response3'
            }
        ]
    }
};

Use the same order value for operations you want to be executed in parallel.

If you want to be even shorter, you can use a collection process:

// config/common/config/sequences.js

'use strict';

module.exports = {
    executeMySyncQueries: {
        operations: [
            {
                service: 'danf:http.router',
                method: 'follow',
                // Process the operation on each item
                // of the following collection.
                collection: {
                    // Define the input collection.
                    input: [
                        'www.example.com/api_1.php',
                        'www.example.com/api_2.php',
                        'www.example.com/api_3.php'
                    ],
                    // Define the async method used.
                    // You can specify any collection method
                    // of the async lib.
                    // '--' is a shorcut for 'forEachOfSeries'
                    // which is an execution in series.
                    method: '--'
                },
                arguments: [
                    // Resolve reference '@@.@@' in the context
                    // of the input item.
                    '@@.@@',
                    'GET'
                ],
                // Set the responses in the property 'responses'
                // of the stream.
                scope: 'responses'
            }
        ]
    }
};

Take a look at the overview of the framework for more informations.

Lunchroom

Answer 15

I landed here because I needed to rate-limit http.request (~10k aggregation queries to elastic search to build an analytical report). The following just choked my machine.

for (item in set) {
    http.request(... + item + ...);
}

My URLs are very simple so this may not trivially apply to the original question but I think it's both potentially applicable and worth writing here for readers that land here with issues similar to mine and who want a trivial JavaScript no-library solution.

My job wasn't order dependent and my first approach to bodging this was to wrap it in a shell script to chunk it (because I'm new to JavaScript). That was functional but not satisfactory. My JavaScript resolution in the end was to do the following:

var stack=[];
stack.push('BOTTOM');

function get_top() {
  var top = stack.pop();
  if (top != 'BOTTOM')
    collect(top);
}

function collect(item) {
    http.request( ... + item + ...
    result.on('end', function() {
      ...
      get_top();
    });
    );
}

for (item in set) {
   stack.push(item);
}

get_top();

It looks like mutual recursion between collect and get_top. I'm not sure it is in effect because the system is asynchronous and the function collect completes with a callback stashed for the event at on.('end'.

I think it is general enough to apply to the original question. If, like my scenario, the sequence/set is known, all URLs/keys can be pushed on the stack in one step. If they are calculated as you go, the on('end' function can push the next url on the stack just before get_top(). If anything, the result has less nesting and might be easier to refactor when the API you're calling changes.

I realise this is effectively equivalent to the @generalhenry's simple recursive version above (so I upvoted that!)

Pauper

Answer 16

Super Request

This is another synchronous module that is based off of request and uses promises. Super simple to use, works well with mocha tests.

npm install super-request

request("http://domain.com")
    .post("/login")
    .form({username: "username", password: "password"})
    .expect(200)
    .expect({loggedIn: true})
    .end() //this request is done 
    //now start a new one in the same session 
    .get("/some/protected/route")
    .expect(200, {hello: "world"})
    .end(function(err){
        if(err){
            throw err;
        }
    });

Representation

Answer 17

This code can be used to execute an array of promises synchronously & sequentially after which you can execute your final code in the .then() call.

const allTasks = [() => promise1, () => promise2, () => promise3];

function executePromisesSync(tasks) {
  return tasks.reduce((task, nextTask) => task.then(nextTask), Promise.resolve());
}

executePromisesSync(allTasks).then(
  result => console.log(result),
  error => console.error(error)
);

Hardnett

Answer 18

sync-request-curl

Following on from @jemiloii's answer about sync-request, but addressing the performance and CPU issues within the answer's replies.

I wrote a new library called sync-request-curl, which

• contains a subset of the features in sync-request
• leverages node-libcurl for better performance on NodeJS
• has more recent documentation (e.g. ES6 syntax)

The usage is the same as sync-request for the most part. For example,

import request from 'sync-request-curl';

const res = request(
  'GET',
  'https://ipinfo.io/json'
);
console.log('Status Code:', res.statusCode);
const jsonBody = JSON.parse(res.body.toString());
console.log('Returned JSON object:', jsonBody);

Note that this library relies on libcurl being installed, so naturally it will not work on a browser (only NodeJS).

Conditional

Answer 19

I actually got exactly what you (and me) wanted, without the use of await, Promises, or inclusions of any (external) library (except our own).

Here's how to do it:

We're going to make a C++ module to go with node.js, and that C++ module function will make the HTTP request and return the data as a string, and you can use that directly by doing:

var myData = newModule.get(url);

ARE YOU READY to get started?

Step 1: make a new folder somewhere else on your computer, we're only using this folder to build the module.node file (compiled from C++), you can move it later.

In the new folder (I put mine in mynewFolder/src for organize-ness):

npm init

then

npm install node-gyp -g

now make 2 new files: 1, called something.cpp and for put this code in it (or modify it if you want):

#pragma comment(lib, "urlmon.lib")
#include <sstream>
#include <WTypes.h>  
#include <node.h>
#include <urlmon.h> 
#include <iostream>
using namespace std;
using namespace v8;

Local<Value> S(const char* inp, Isolate* is) {
    return String::NewFromUtf8(
        is,
        inp,
        NewStringType::kNormal
    ).ToLocalChecked();
}

Local<Value> N(double inp, Isolate* is) {
    return Number::New(
        is,
        inp
    );
}

const char* stdStr(Local<Value> str, Isolate* is) {
    String::Utf8Value val(is, str);
    return *val;
}

double num(Local<Value> inp) {
    return inp.As<Number>()->Value();
}

Local<Value> str(Local<Value> inp) {
    return inp.As<String>();
}

Local<Value> get(const char* url, Isolate* is) {
    IStream* stream;
    HRESULT res = URLOpenBlockingStream(0, url, &stream, 0, 0);

    char buffer[100];
    unsigned long bytesReadSoFar;
    stringstream ss;
    stream->Read(buffer, 100, &bytesReadSoFar);
    while(bytesReadSoFar > 0U) {
        ss.write(buffer, (long long) bytesReadSoFar);
        stream->Read(buffer, 100, &bytesReadSoFar);
    }
    stream->Release();
    const string tmp = ss.str();
    const char* cstr = tmp.c_str();
    return S(cstr, is);
}

void Hello(const FunctionCallbackInfo<Value>& arguments) {
    cout << "Yo there!!" << endl;

    Isolate* is = arguments.GetIsolate();
    Local<Context> ctx = is->GetCurrentContext();

    const char* url = stdStr(arguments[0], is);
    Local<Value> pg = get(url,is);

    Local<Object> obj = Object::New(is);
    obj->Set(ctx,
        S("result",is),
        pg
    );
    arguments.GetReturnValue().Set(
       obj
    );

}

void Init(Local<Object> exports) {
    NODE_SET_METHOD(exports, "get", Hello);
}

NODE_MODULE(cobypp, Init);

Now make a new file in the same directory called something.gyp and put (something like) this in it:

{
   "targets": [
       {
           "target_name": "cobypp",
           "sources": [ "src/cobypp.cpp" ]
       }
   ]
}

Now in the package.json file, add: "gypfile": true,

Now: in the console, node-gyp rebuild

If it goes through the whole command and says "ok" at the end with no errors, you're (almost) good to go, if not, then leave a comment..

But if it works then go to build/Release/cobypp.node (or whatever its called for you), copy it into your main node.js folder, then in node.js:

var myCPP = require("./cobypp")
var myData = myCPP.get("http://google.com").result;
console.log(myData);

..

response.end(myData);//or whatever

Kileykilgore