What are the most important statistics to look at when deploying a Node.js web-application?

Question

First - a little bit about my background: I have been programming for some time (10 years at this point) and am fairly competent when it comes to coding ideas up. I started working on web-application programming just over a year ago, and thankfully discovered nodeJS, which made web-app creation feel a lot more like traditional programming. Now, I have a node.js app that I've been developing for some time that is now running in production on the web. My main confusion stems from the fact that I am very new to the world of the web development, and don't really know what's important and what isn't when it comes to monitoring my application.

I am using a Joyent SmartMachine, and looking at the analytics options that they provide is a little overwhelming. There are so many different options and configurations, and I have no clue what purpose each analytic really serves. For the questions below, I'd appreciate any answer, whether it's specific to Joyent's Cloud Analytics or completely general.

QUESTION ONE

Right now, my main concern is to figure out how my application is utilizing the server that I have it running on. I want to know if my application has the right amount of resources allocated to it. Does the number of requests that it receives make the server it's on overkill, or does it warrant extra resources? What analytics are important to look at for a NodeJS app for that purpose? (using both MongoDB and Redis on separate servers if that makes a difference)

QUESTION TWO

What other statistics are generally really important to look at when managing a server that's in production? I'm used to programs that run once to do something specific (e.g. a raytracer that finishes running once it has computed an image), as opposed to web-apps which are continuously running and interacting with many clients. I'm sure there are many things that are obvious to long-time server administrators that aren't to newbies like me.

QUESTION THREE

What's important to look at when dealing with NodeJS specifically? What are statistics/analytics that become particularly critical when dealing with the single-threaded event loop of NodeJS versus more standard server systems?

I have other questions about how databases play into the equation, but I think this is enough for now...

Narcis

Very constructive question. Nice! – Sheerness

Answer 1

We have been running node.js in production nearly an year starting from 0.4 and currenty 0.8 series. Web app is express 2 and 3 based with mongo, redis and memcached.

Few facts.

• node can not handle large v8 heap, when it grows over 200mb you will start seeing increased cpu usage
• node always seem to leak memory, or at least grow large heap size without actually using it. I suspect memory fragmentation, as v8 profiling or valgrind shows no leaks in js space nor resident heap. Early 0.8 was awful in this respect, rss could be 1GB with 50MB heap.
• hanging requests are hard to track. We wrote our middleware to monitor these especially as our app is long poll based

My suggestions.

• use multiple instances per machine, at least 1 per cpu. Balance with haproxy, nginx or such with session affinity
• write midleware to report hanged connections, ie ones that code never responded or latency was over threshold
• restart instances often, at least weekly
• write poller that prints out memory stats with process module one per minute
• Use supervisord and fabric for easy process management

Monitor cpu, reported memory stats and restart on threshold

Foretop

so does this imply that if I have a 2GB RAM machine running 10 processes across 10 cores (hypothetical), it would be a lot more effective to add a new machine of equal capacity as opposed to doubling the RAM and CPU power? – Narcis

I don't know. All I know that you need to have several small instances of node per server to utilize the machine capability. Scaling is dependent on your app and network design. – Foretop

Answer 2

Whichever the type of web app, NodeJS or otherwise, load testing will answer whether your application has the right amount of server resources. A good website I recently found for this is Load Impact.

The real question to answer is WHEN does the load time begin to increase as the number of concurrent users increase? A tipping point is reached when you get to a certain number of concurrent users, after which the server performance will start to degrade. So load test according to how many users you expect to reach your website in the near future.

How can you estimate the amount of users you expect?

Installing Google Analytics or another analytics package on your pages is a must! This way you will be able to see how many daily users are visiting your website, and what is the growth of your visits from month-to-month which can help in predicting future expected visits and therefore expected load on your server.

Even if I know the number of users, how can I estimate actual load?

The answer is in the F12 Development Tools available in all browsers. Open up your website in any browser and push F12 (or for Opera Ctrl+Shift+I), which should open up the browser's development tools. On Firefox make sure you have Firebug installed, on Chrome and Internet Explorer it should work out of the box. Go to the Net or Network tab and then refresh your page. This will show you the number of HTTP requests, bandwidth usage per page load!

So the formula to work out daily server load is simple:

Number of HTTP requests per page load X the average number of pages load per user per day X Expected number of concurrent users = Total HTTP Requests to Server per Day

And...

Number of MBs transferred per page load X the average number of pages load per user per day X Expected number of concurrent users = Total Bandwidth Required per Day

I've always found it easier to calculate these figures on a daily basis and then extrapolate it to weeks and months.

Flavory

thanks for this answer - i gave the other one the bounty though because it addresses node.js specifically, which is more of what I'm interested in. – Narcis

@Narcis You gave it to the right person! :) - I just never had the time to address node.js specifically – Flavory

if you'd like to, i just put up a new bounty for 150 ;) – Narcis

Answer 3

Node.js is single threaded so you should definitely start a process for every cpu your machine has. Cluster is by far the best way to do this and has the added benefit of being able to restart died workers and to detect unresponsive workers.

You also want to do load testing until your requests start timing out or exceed what you consider a reasonable response time. This will give you a good idea of the upper limit your server can handle. Blitz is one of the many options to have a look at.

I have never used Joyent's statistics, but NodeFly and their node-nodefly-gcinfo is a great tools to monitor node processes.

Rushy

Thanks for sharing your thoughts. This unfortunately doesn't add much beyond what has already been said. – Narcis