error handling in asynchronous node.js calls

Question

I'm new to node.js although I'm pretty familiar with JavaScript in general. My question is regarding "best practices" on how to handle errors in node.js.

Normally when programming web servers, FastCGI servers or web pages in various languages I'm using Exceptions with blocking handlers in a multi-threading environment. When a request comes in I usually do something like this:

function handleRequest(request, response) {
  try {

    if (request.url=="whatever")
      handleWhateverRequest(request, response);
    else
      throw new Error("404 not found");

  } catch (e) {
    response.writeHead(500, {'Content-Type': 'text/plain'});
    response.end("Server error: "+e.message);
  }
}

function handleWhateverRequest(request, response) {
  if (something) 
    throw new Error("something bad happened");
  Response.end("OK");
}

This way I can always handle internal errors and send a valid response to the user.

I understand that with node.js one is supposed to do non-blocking calls which obviously leads to various number of callbacks, like in this example:

var sys    = require('sys'),
    fs     = require('fs');

require("http").createServer(handleRequest).listen(8124);

function handleRequest(request, response) {

  fs.open("/proc/cpuinfo", "r",
    function(error, fd) {
      if (error)
        throw new Error("fs.open error: "+error.message);

      console.log("File open.");

      var buffer = new require('buffer').Buffer(10);
      fs.read(fd, buffer, 0, 10, null,
        function(error, bytesRead, buffer) {

          buffer.dontTryThisAtHome();  // causes exception

          response.end(buffer);
        }); //fs.read

    }); //fs.open

}

This example will kill the server completely because exceptions aren't being catched. My problem is here that I can't use a single try/catch anymore and thus can't generally catch any error that may be raised during the handling of the request.

Of course I could add a try/catch in each callback but I don't like that approach because then it's up to the programmer that he doesn't forget a try/catch. For a complex server with lots of different and complex handlers this isn't acceptable.

I could use a global exception handler (preventing the complete server crash) but then I can't send a response to the user since I don't know which request lead to the exception. This also means that the request remains unhandled/open and the browser is waiting forever for a response.

Does someone have a good, rock solid solution?

Zoroastrianism

Answer 1

Node 0.8 introduces a new concept called "Domains". They are very roughly analogousness to AppDomains in .net and provide a way of encapsulating a group of IO operations. They basically allow you to wrap your request processing calls in a context specific group. If this group throws any uncaught exceptions then they can be handled and dealt with in a manner which gives you access to all the scope and context specific information you require in order to successfully recover from the error (if possible).

This feature is new and has only just been introduced, so use with caution, but from what I can tell it has been specifically introduced to deal with the problem which the OP is trying to tackle.

Documentation can be found at: http://nodejs.org/api/domain.html

Ammoniac

Be warned, Domain has since been deprecated. nodejs.org/api/domain.html#domain_domain – Composition

Answer 2

Checkout the uncaughtException handler in node.js. It captures the thrown errors that bubble up to the event loop.

http://nodejs.org/docs/v0.4.7/api/process.html#event_uncaughtException_

But not throwing errors is always a better solution. You could just do a return res.end('Unabled to load file xxx');

Coriander

Thank you for your answer, but as written in the description, uncaughtException won't help me as I can't end the request in there, because I have no request or response reference. And you can't completely avoid exceptions (and after all I love them and use them a lot). – Zoroastrianism

Answer 3

This is one of the problems with Node right now. It's practically impossible to track down which request caused an error to be thrown inside a callback.

You're going to have to handle your errors within the callbacks themselves (where you still have a reference to the request and response objects), if possible. The uncaughtException handler will stop the node process from exiting, but the request that caused the exception in the first place will just hang there from the user point of view.

Branton

Answer 4

I give an answer to my own question... :)

As it seems there is no way around to manually catch errors. I now use a helper function that itself returns a function containing a try/catch block. Additionally, my own web server class checks if either the request handling function calls response.end() or the try/catch helper function waitfor() (raising an exception otherwise). This avoids to a great extent that request are mistakenly left unprotected by the developer. It isn't a 100% error-prone solution but works well enough for me.

handler.waitfor = function(callback) {
  var me=this;

  // avoid exception because response.end() won't be called immediately:
  this.waiting=true;

  return function() {
    me.waiting=false;
    try {
      callback.apply(this, arguments);

      if (!me.waiting && !me.finished)
        throw new Error("Response handler returned and did neither send a "+
          "response nor did it call waitfor()");

    } catch (e) {
      me.handleException(e);
    }
  }
}

This way I just have to add a inline waitfor() call to be on the safe side.

function handleRequest(request, response, handler) {
  fs.read(fd, buffer, 0, 10, null, handler.waitfor(
    function(error, bytesRead, buffer) {

      buffer.unknownFunction();  // causes exception
      response.end(buffer);
    }
  )); //fs.read
}

The actual checking mechanism is a little more complex, but it should be clear how it works. If someone is interested I can post the full code here.

Zoroastrianism

When downvoting my solution, please add a comment and describe the reason. I'm already using this method in a number of projects. It works like a charm and I can't see a better alternative, yet. – Zoroastrianism

It wasn't me who downvoted you, but I do have a question mark about this. What about chained callbacks? In other words, you have a callback which calls another function that takes a callback, and so on. Unless I'm mistaken, it looks here like you'd have to wrap every callback in a waitfor function. Is that the case? – Lx

Basically, yes. The waitfor() function just makes sure that "any" exception thrown within the callback can be catched and handled in the right context instead of ending as a generic unhandled exception. As said, the solution unfortunately is not perfect, but I have yet to see a better one. – Zoroastrianism

+1 to counter downvotes given without any explanation whatsoever. This solution does look a bit inelegant, but if an inelegant solution is the best that is available, then we just have to live with it. – Lx

Answer 5

Very good question. I'm dealing with the same problem now. Probably the best way, would be to use uncaughtException. The reference to respone and request objects is not the problem, because you can wrap them into your exception object, that is passed to uncaughtException event. Something like this:

var HttpException = function (request, response, message, code) {

  this.request = request;
  this.response = response;  
  this.message = message;    
  this.code = code || 500;

}

Throw it:

throw new HttpException(request, response, 'File not found', 404);

And handle the response:

process.on('uncaughtException', function (exception) {
  exception.response.writeHead(exception.code, {'Content-Type': 'text/html'});
  exception.response.end('Error ' + exception.code + ' - ' + exception.message);
});

I haven't test this solution yet, but I don't see the reason why this couldn't work.

Shaftesbury

I'm successfully using a solution like the one I described in my own answer now and it works great. A custom HttpException works only for self-generated exceptions, but fails for exceptions generated elsewhere, like a plugin or a 3rd party script, which aren't HttpException. The only way to catch any exception while keeping context info is using try .. catch – Zoroastrianism

...and, exception can also be caused by typos, like misspelled function names, of course. – Zoroastrianism

You want to be very careful with using this solution, as per the node documentation. See my answer for a better way of dealing with this situation. "Don't use it, use domains instead. If you do use it, restart your application after every unhandled exception!" source: nodejs.org/api/process.html – Ammoniac

Doesn't this leak like crazy? – Trackandfield

Answer 6

One idea: You could just use a helper method to create your call backs and make it your standard practice to use it. This does put the burden on the developer still, but at least you can have a "standard" way of handling your callbacks such that the chance of forgetting one is low:

var callWithHttpCatch = function(response, fn) {
    try {
        fn && fn();
    }
    catch {
        response.writeHead(500, {'Content-Type': 'text/plain'}); //No
    }
}

<snipped>
      var buffer = new require('buffer').Buffer(10);
      fs.read(fd, buffer, 0, 10, null,
        function(error, bytesRead, buffer) {

          callWithHttpCatch(response, buffer.dontTryThisAtHome());  // causes exception

          response.end(buffer);
        }); //fs.read

    }); //fs.open

I know that probably isn't the answer you were looking for, but one of the nice things about ECMAScript (or functional programming in general) is how easily you can roll your own tooling for things like this.

Wormeaten

Answer 7

At the time of this writing, the approach I am seeing is to use "Promises".

http://howtonode.org/promises
https://www.promisejs.org/

These allow code and callbacks to be structured well for error management and also makes it more readable. It primarily uses the .then() function.

someFunction().then(success_callback_func, failed_callback_func);

Here's a basic example:

  var SomeModule = require('someModule');

  var success = function (ret) {
      console.log('>>>>>>>> Success!');
  }

  var failed = function (err) {
    if (err instanceof SomeModule.errorName) {
      // Note: I've often seen the error definitions in SomeModule.errors.ErrorName
      console.log("FOUND SPECIFIC ERROR");
    }
    console.log('>>>>>>>> FAILED!');
  }

  someFunction().then(success, failed);
  console.log("This line with appear instantly, since the last function was asynchronous.");

Adalineadall

Promise don't help here. If a promise do an async call that throw an exception, the NodeJS process stop as well. Try to setTimeout(function(){ throw new Exception()}) in your someFunction to see what happens. – Arnaud

Answer 8

Two things have really helped me solve this problem in my code.

1. The 'longjohn' module, which lets you see the full stack trace (across multiple asyncronous callbacks).

2. A simple closure technique to keep exceptions within the standard callback(err, data) idiom (shown here in CoffeeScript).

   ferry_errors = (callback, f) ->
     return (a...) ->
       try f(a...)
       catch err
         callback(err)
   

Now you can wrap unsafe code, and your callbacks all handle errors the same way: by checking the error argument.

Jazmin

Answer 9

I've recently created a simple abstraction named WaitFor to call async functions in sync mode (based on Fibers): https://github.com/luciotato/waitfor

It's too new to be "rock solid".

using wait.for you can use async function as if they were sync, without blocking node's event loop. It's almost the same you're used to:

var wait=require('wait.for');

function handleRequest(request, response) {
      //launch fiber, keep node spinning
      wait.launchFiber(handleinFiber,request, response); 
}

function handleInFiber(request, response) {
  try {
    if (request.url=="whatever")
      handleWhateverRequest(request, response);
    else
      throw new Error("404 not found");

  } catch (e) {
    response.writeHead(500, {'Content-Type': 'text/plain'});
    response.end("Server error: "+e.message);
  }
}

function handleWhateverRequest(request, response, callback) {
  if (something) 
    throw new Error("something bad happened");
  Response.end("OK");
}

Since you're in a fiber, you can program sequentially, "blocking the fiber", but not node's event loop.

The other example:

var sys    = require('sys'),
    fs     = require('fs'),
    wait   = require('wait.for');

require("http").createServer( function(req,res){
      wait.launchFiber(handleRequest,req,res) //handle in a fiber
  ).listen(8124);

function handleRequest(request, response) {
  try {
    var fd=wait.for(fs.open,"/proc/cpuinfo", "r");
    console.log("File open.");
    var buffer = new require('buffer').Buffer(10);

    var bytesRead=wait.for(fs.read,fd, buffer, 0, 10, null);

    buffer.dontTryThisAtHome();  // causes exception

    response.end(buffer);
  }
  catch(err) {
    response.end('ERROR: '+err.message);
  }

}

As you can see, I used wait.for to call node's async functions in sync mode, without (visible) callbacks, so I can have all the code inside one try-catch block.

wait.for will throw an exception if any of the async functions returns err!==null

more info at https://github.com/luciotato/waitfor

Reverberator

Answer 10

Also in synchronous multi-threaded programming (e.g. .NET, Java, PHP) you can't return any meaningful information to the client when a custom unkown Exception is caught. You may just return HTTP 500 when you have no info regarding the Exception.

Thus, the 'secret' lies in filling a descriptive Error object, this way your error handler can map from the meaningful error to the right HTTP status + optionally a descriptive result. However you must also catch the exception before it arrives to process.on('uncaughtException'):

Step1: Define a meaningful error object

function appError(errorCode, description, isOperational) {
    Error.call(this);
    Error.captureStackTrace(this);
    this.errorCode = errorCode;
    //...other properties assigned here
};

appError.prototype.__proto__ = Error.prototype;
module.exports.appError = appError;

Step2: When throwing an Exception, fill it with properties (see step 1) that allows the handler to convert it to meannigul HTTP result:

throw new appError(errorManagement.commonErrors.resourceNotFound, "further explanation", true)

Step3: When invoking some potentially dangerous code, catch errors and re-throw that error while filling additional contextual properties within the Error object

Step4: You must catch the exception during the request handling. This is easier if you use some leading promises library (BlueBird is great) which allows you to catch async errors. If you can't use promises than any built-in NODE library will return errors in callback.

Step5: Now that your error is caught and contains descriptive information about what happens, you only need to map it to meaningful HTTP response. The nice part here is that you may have a centralized, single error handler that gets all the errors and map these to HTTP response:

    //this specific example is using Express framework
    res.status(getErrorHTTPCode(error))
function getErrorHTTPCode(error)
{
    if(error.errorCode == commonErrors.InvalidInput)
        return 400;
    else if...
}

You may other related best practices here

Burglary