Node.js EMFILE error with increasing traffic

Question

I have very recently received a lot of traffic to my site that runs Node.js. With the increasing traffic it has started to crash a lot, which has not happened before. I get the following error in my log:

{ [Error: connect EMFILE] code: 'EMFILE', errno: 'EMFILE', syscall: 'connect' }
Error: connect EMFILE
    at errnoException (net.js:670:11)
    at connect (net.js:548:19)
    at net.js:607:9
    at Array.0 (dns.js:88:18)
    at EventEmitter._tickCallback (node.js:192:40)

Anyone that have an idea why it crash? And ideas how to solve it?

I'm using Express.js and Socket.io. It runs on Ubuntu.

Avestan

EMFILE error means that the OS is denying your program to open more files/sockets, have a look at: https://mcmap.net/q/110561/-how-do-i-change-the-number-of-open-files-limit-in-linux-closed/511300 – Rectify

I'd say, this Q&A is definitely worth MORE views, even as many as Hello World. Developers open concurrent sockets on demand without any limitation, however, they don't know the fact that it means to allow massive requests for opening sockets to make their application crash and mess up all the things. None warns this, even though it may be a serious vulnerability. Developers had to be informed this before they learned how to use forever. – Nutter

To prevent your application from crashing by massive connections, firstly, ascertain no resource leakage ― your application not making and keeping unnecessary connections; then secondly, do either, limit your available concurrent sockets, or increase the OS's opened-socket limit. – Nutter

Answer 1

EMFILE error means that the OS is denying your program to open more files/sockets.

Have a look at: How do I change the number of open files limit in Linux?

Rectify

Increasing your socket limit won't fix a leak though, it'll only keep your app from crashing for a little while longer. – Mesic

@Mesic what do you suggest to prevent leak? I've exactly the same problem and I don't want fix it by increasing ulimit because, as you wrote, it's temporarilly fix. – Handtohand

@Robert Sometimes increasing the ulimit is actually the solution, like if it was at 256 and you had high traffic, though I'm my case I was repeatedly opening file steams and never closing them, I caught this leak with lsof -p <nodes pid>. If that fails you can use izaacs graceful-fs module – Mesic

Unfortunatelly graceful-fs module is not fix for me. I reach limit because of sockets, not because of opening to much files with fs but thanks for help. – Handtohand

sockets and files are both file descriptors, ulimit modifies the file descriptor limit – Mesic

@RobertJagoda were you able to solve the problem with sockets? I couldn't find a solution when dealing with http.get requests, resulting in EMERROR! :( – Rhodos

@Rhodos yes but I've had somewhere else wrong code causing troubles. You can check my thread: #18377811 If you have to many connections check if there is everything ok in your code. Maybe something creates too many of connections? Check also ulimit on server. Some servers have it unlimited, others have limits. – Handtohand

Yes, found the error. It was indeed creating too many connections inside a setInterval block. I had to put the initializing code outside it. – Rhodos

Answer 2

EMFILE means error maximum files which indicates that the OS is denying your program to open more file descriptors.

Note that open files in the system include disk files, named pipes, network sockets and devices opened by all processes.

Your operating system specifies the open file limit per process.

To check open file limit of your system use ulimit -a command. Generally on unix like system it is default to 1024.

If the limitation was 1024, means you/process can open maximum 1024 files. if you exceed this limit means open, pipe and dup system calls will fail and yield EMFILE error.

When you get EMFILE it mostly indicates the leak in your code. Increasing the ulimit to a higher value is not a good solution in case there is leak in your program.

You should try to find out the cause of leak. Following can be used to debug in unix like operating system :

1. lsof meaning list open files command gives the list of open files. Assume your nodeJS program is having leak so to find out total number of file descriptors opened by your program :

   lsof | grep node | wc -l

2. To find out the file descriptors for sockets, use:

   lsof -n -i -P | grep node

Using this we can find out where the leak is and then we can correct it.

Haviland

Answer 3

In my case, we already had the ulimit set and kern.maxfiles configuration highest it could go on a Mac.

What fixed it was limiting the number of max sockets globally. In later versions of node the limit is Infinity.

Try adding these lines of code:

var https = require('https');
var http = require('http');

https.globalAgent.maxSockets = 5;
http.globalAgent.maxSockets = 5;

In the case of using the request library, you can configure these settings.

See: https://github.com/request/request#request---simplified-http-client

Here is more on maxSockets: https://nodejs.org/api/http.html#http_agent_maxsockets

Roil