Node.js SSL server frozen, high CPU, not crashed but no connections

Question

I hope anyone could help me with this issue.

In our company we are setting up a node.js server, connected to a Java Push server.

I'm using https module instead of http and SLL certificates.

The connection between node and clients is made by socket.io, in server and client.

At the same time the node.js server is client of the java server, this connection is being made with regular sockets (net.connect).

The idea is that users connect to the server, join some channels, and when some data arrive from java server, this is dispatched to the corresponding users.

Everything seems to work fine, but after a while, like randomly, having like between 450 and 700 users, the server's CPU reaches 100%, all the connections are broken, but the server is not crashed. The thing is that if you go to the https://... in the browser, you are not getting 404 or something like that but SSL connection error, and its really fast.

I tried to add logs everywhere, but there's not something like a pattern, its like random.

If anybody have the same problem or could bring me a clue, or a tip to debug better, I'll appreciate anything.

Thanks a lot.

Centenarian

Answer 1

Okay, the problem is solved. It is a problem that will occur in every Linux server. So, if you are working with one of these, you need to read this.

The reason was the default limit of files the Linux server had per each process.

Seems that ever single linux server comes with this limitation of 1024 files opened by each process, you can check your limit with:

# ulimit -n

To increase this number

# ulimit -n 5000 (for example)

Each socket creates a new virtual file.

For some reason my server was not displaying any error, the server just got frozen, stopping the log and no signal or evidence of anything. It was when I set up a copy of the server in another machine, when it started to send

warn: error raised: Error: accept EMFILE
warn: error raised: Error: accept EMFILE
warn: error raised: Error: accept EMFILE
...

Be careful because if you are not root, you will only change this for the current session and not permanently.

Trick: If you want to cound the number of files, in this case, the number of files opened by your node process, take note of your process id and call this command.

# ls -l /proc/XXXXX/fd | wc -l

Where XXXXX is the process id. This will help you to know if this is your problem, once you launch your node server, you can use this command to check if it reaches a top, and it stops growing after it gets frozen. (by default 1024 or "ulimit -n").

If you only want to check which files are open by the process:

# ls -l /proc/XXXXX/fd

Hope this can help you. Any way if you are setting up a node js server I'm pretty sure you want to do that to be sure it won't melt.

Finally if you need help in future errors without log, you can try to straceing or dtrussing process

# strace -p <process-id> 

should do the job.

Centenarian

Thanks, that helped me a lot! Although I have another problem with that. You wrote that each socket create new virtual file. Now, for example if You have one user and he sends new actions through sockets then every one action creates new file. When You have 1000 users, each one send 1 socket then You will have 1000 new files? So more actions === more files and finally server must reach the limit. How did You solved that? Or maybe I do something in wrong way? For me every socket.emit action creates new files. – Kassel

What do you mean with actions?, each user maintains a connection thus a file. But there'll be a new connection per tab in the browser. – Centenarian

Oh, I forgot to delete my comment after resolving my issue. I've had a bug- almost every action created connection with memcache without closing it so server's console showed me that every socket emit action create new file. After I added closing memcache connection everything is ok. – Kassel