Node.js spawn EMFILE

Question

I am trying to run a command inside a async.forEach loop using ChildProcess.exec in my node job. here is the code

async.forEach( docPaths, function(docPath, callback) { 
 var run = [];
 // some command using docPath variable here..
 run.push(command);
debugger;
 exec(run.join(' '), function(error, stdout, stderr){
    callback();
  });
 }, callback);

Here is the error

"stack":"Error: spawn EMFILE\
at errnoException (child_process.js:478:11)\
at ChildProcess.spawn (child_process.js:445:11)\
at child_process.js:343:9\
at Object.execFile (child_process.js:253:15)\
at child_process.js:220:18\

a quick google shows i need to set ulimit value to increase the number of file descriptors can be open. some thing like "ulimit -n 10000".. (from link below)

https://groups.google.com/forum/#!topic/nodejs/jeec5pAqhps

where can i increase this..? or is there any other solution to circumvent the issue?

Appreciate your help.. Thanks much !!

Zarathustra

Answer 1

First of all its not advisable to mess with ulimit, as it may have system wide impacts.

Instead since you are already using async, it comes with a limit paramater which you can use to limit the number of parallely executions.

async.eachLimit( docPaths, 100, function(docPath, callback) { 
 var run = [];
 // some command using docPath variable here..
 run.push(command);
debugger;
 exec(run.join(' '), function(error, stdout, stderr){
    callback();
  });
 }, callback);

Please do trial and error and replace 100 with suitable value.

Jeffreys

As far as I know, ulimit on file descriptors was set so purely to track down FD leaks in classic c++ apps. So it should be safe to set it to unlimited for node apps since it doesn't make much sense in async world anyway. – Cementum

@alex, it is not safe to set the ulimit to unlimited, please refer linuxhowtos.org/Tips%20and%20Tricks/ulimit.htm to see the consequences.. – Jeffreys

i have tried both eachLimit and eachSeries to get pass the error. any of the two worked well for me. Though eachSeries does not process simultaneously, it kind of solved the issue i am running into. Thanks much – Zarathustra

@Sriharsha, the article is about max user processes (-u), we're talking about open files (-n), that's a different thing. – Cementum