Inter-child-process communication options in node.js cluster

Question

So I'm working on a node.js game server application at the moment, and I've hit a bit of a wall here. My issue is that I'm using socket.io to accept inbound connections from game clients. These clients might be connected to one of several Zones or areas of the game world.

The basic architecture is displayed below. The master process forks a child process for each Zone of the game which runs the Zone Manager process; a process dedicated to maintaining the Zone data (3d models, positions of players/entities, etc). The master process then forks multiple "Communication Threads" for each Zone Manager it creates. These threads create an instance of socket.io and listen on the port for that Zone (multiple threads listening on a single port). These threads will handle the majority of the game logic in their own process as well as communicate with the database backing the game server. The only issue is that in some circumstances they may need to communicate with the Zone Manager to receive information about the Zone, players, etc.

As an example: A player wants to buy/sell/trade with a non-player character (NPC) in the Zone. The Zone Communication thread needs to ask the Zone Manager thread if the player is close enough to the NPC to make the trade before it allows the trade to take place.

The issue I'm running into here is that I was planning to make use of the node.js cluster functionality and use the send() and on() methods of the processes to handle passing messages back and forth. That would be fine except for one caveat I've run into with it. Since all child processes spun off with cluster.fork() can only communicate with the "master" process. The node.js root process becomes a bottleneck for all communication. I ran some benchmarks on my system using a script that literally just bounced a message back and forth using the cluster's Inter-process Communication (IPC) and kept track of how many relays per second were being carried out. It seems that eventually node caps out at about 20k per second in terms of how many IPC's it can relay. This number was consistent on both a Phenom II 1.8ghz quad core laptop, and an FX-8350 4.0ghz 8-core desktop.

Now that sounds pretty decently high, except that this basically means that regardless of how many Zones or Communication Threads there are, all IPC is still bottlenecking through a single process that acts as a "relay" for the entire application. Which means that although it seems each individual thread can relay > 20k IPCs per second, the entire application as a whole will never relay more than that even if it were on some insane 32 core system since all the communication goes through a single thread.

So that's the problem I'm having. Now the dilemma. I've read a lot about the various other options out there and read like 20 different questions here on stack about this topic and I've seen a couple things popping up regularly:

Redis: I'm actually running Redis on my server at the moment and using it as the socket.io datastore so that socket.io in multiple threads can share connection data so that a user can connect to any of N number of socket.io threads for their Zone so the server can sort of automatically load balance the incoming connections.

My concern with this is that it runs through the network stack. Hardly ideal for communication between multiple processes on the same server. I feel like the latency would be a major issue in the long run.

0MQ (zeromq/zmq): I've never used this one for anything before, but I've been hearing a bit about it lately. Based on the reading I've done, I've found a lot of examples of people using it with TCP sockets, but there's not a lot of buzz about people using it for IPC. I was hoping perhaps someone here had worked with 0MQ for IPC before (possibly even in node.js?) and could shine some light on this option for me.

dnode: Again I've never used this, but from what I've seen it looks like it's another option that is designed to work over TCP which means the network stack gets in the way again.

node-udpcomm: Someone linked this in another question on here (which I can't seem to find again unfortunately). I've never even heard of it, and it looks like a very small solution that opens up and listens on UDP connections. Although this would probably still be faster than TCP options, we still have the network stack in the way right? I'm definitely like about a mile outside of my "programmer zone" as is here and well into the realm of networking/computer architecture stuff that I don't know much about lol

Anyway the bottom line is that I'm completely stuck here and have no idea what the best option would be for IPC in this scenario. I'm assuming at the moment that 0MQ is the best option of the ones I've listed above since it's the only one that seems to offer an "IPC" option for communication protocol which I presume means it's using a UNIX socket or something that's not going through the network stack, but I can't confirm that or anything.

I guess I'm just hoping some people here might know enough to point me in the right direction or tell me I'm already headed there. The project I'm working on is a multiplayer game server designed to work "out of the box" with a multiplayer game client both powering their 3D graphics/calculations with Three.js. The client and server will be made open source to everyone once I get them all working to my satisfaction, and I want to make sure that the architecture is as scalable as possible so people don't build a game on this and then go to scale up and end up hitting a wall.

Anyway thanks for your time if you actually read all this :)

Mortensen

Answer 1

I think that 0MQ would be a very good choice, but I admit that I don't know the others :D

For 0MQ it's transparent what transport you decide to use, the library calls are the same. It's just about choosing particular endpoint (and thus transport) during the call to zmq_bind and zmq_connect at the beginning. There are basically 4 paths you may decide to take:

1. "inproc://<id>" - in-process communication endpoint between threads via memory
2. "ipc://<filepath>" - system-dependent inter-process communication endpoint
3. "tcp://<ip-address>" - clear
4. "pgm://..." or "epgm://..." - an endpoint for Pragmatic Reliable Multicast

So to put is simply, the higher in the list you are, the faster it is and the less problems considering latency and reliability you have to face. So you should try to keep as high as possible. Since your components are processes, you should go with the IPC transport, naturally. If you later on need to change something, you can just change the endpoint definition and you are fine.

Now what is in fact more important than the transport you choose is the socket type, or rather pattern you decide to use. You situation is a typical request-response kind of communication, so you can either do

1. Manager: REP socket, Threads: REQ socket; or
2. Manager: ROUTER, Threads: REQ or DEALER

The Threads would then connect their sockets to the single Manager socket assigned to them and that's all, they can start to send their requests and wait for responses. All they have to decide on is the path they use as the endpoint.

But to describe in details what all those socket types and patterns mean is definitely out of the scope of this post, but you can and should read more about it in the ZeroMQ Guide. There you can not only learn about all the socket types, but also about many different ways how to connect your components and let them talk to each other. The one I mentioned is just a very simple one. Once you understand, you can build arbitrary hierarchies. It's like LEGO ;-)

Hope it helped a bit, cheers!

Zicarelli

Brilliant! Thanks :) The only other thing I'm wondering here is could two separate threads of node.js utilize the inproc://<id> transport method with a req pattern to enable bi-directional communication via memory? Of course performance is the number 1 concern here, but I also want to make sure I'm using libraries and methods that will work on *NIX/Windows/Mac basically anything that runs node.js. So 0MQ looks great, I just want to make sure I'm using a solid transport/pattern :) – Mortensen

inproc has just one "limitation" compared to the other transports - you have to bind the socket before others try to connect (other transports can connect before someone binds). On the other hand, it's by far the fastest transport available since it's going directly through memory and no I/O thread is used. So if you use just inproc, you can create your context with 0 I/O threads. – Zicarelli

So just to make sure I'm clear on that last bit about the bind/connect order. I'm hearing that as I need to call var sock = require('zmq').socket('req').bindSync('inproc://my_uid'); in each of the Threads before i can call var sock = require('zmq').socket('res').bindSync('inproc://my_uid'); in the Manager process multiple times to connect a callback function to each of the Threads? Or do I have that all kinds of backwards? lol – Mortensen

Don't make it more complex than it is, it's actually really simple :-) The only thing I wanted to say is that inproc really requires the typical client-server model, i.e. server binds, clients then connect. Other transports allow to connect before the server binds and as long as the queues don't overflow you are fine. Since I guess there are no queues in the inproc, or rather only on the server side, it has to be created before any client connects. So Manager: socket(RES).bindSync(ID), then all the Threads can do socket(REQ).connect(ID). It's like a web server, in a way... – Zicarelli

Yeah I kinda played around with it and figured that out. At the moment I'm using a little test script with 1 "manager" and 2 "threads" attempting to connect to it. I have the manager binding and then relaying a message through the node.js cluster messaging system so the threads can connect. The problem is I'm getting "Connection Refused" when the threads call connect. If I change the transport from inproc to ipc it works fine, so there must be something I'm missing here :\ – Mortensen

You mentioned that The master process then forks multiple "Communication Threads". When I mentioned Threads I meant your Threads, i.e. processes. Then you must, logically, use the ipc transport... In any case, threads don't really make sense within node.js I guess. – Zicarelli

Hmm yeah it does look like node's cluster.fork is actually spawning new processes, so I'm guessing that means the inproc option isn't going to work and I'd have to use ipc. I'll have to run some benchmarks on this with IPC then because the one I just ran actually shows 0MQ to be much much slower than relaying the communication through the built-in IPC of node.js cluster. Which kind of makes sense when you think about it. I'll do some more testing on this and see what's going to be fastest. Thanks for all the help :) – Mortensen

Btw if you use ipc, you don't have to wait for the server to come up, you can just connect and 0MQ will take care of it when the server comes online. So I guess you don't have to relay messages through node.js cluster to sync... – Zicarelli

Btw check also the ROUTER socket instead of REP one for the Manager. I am not even sure how REP can work in node.js. I've never tried, though... – Zicarelli

Ah, ok, it's synchronous, that's why it kinda works. Chmm. In any case it needs some research to get things right in node.js, it's possible that there is a way to make it faster if it's slow for you, but for that you need to understand 0MQ and node.js bindings more I guess... – Zicarelli

Actually, I stand corrected. I implemented the benchmark on the manager on the node test and on the threads in 0MQ. So that's why I was seeing about a 50% loss lol. Anyway, with the new (correct) benchmark I'm showing about a 1% performance loss in going from node.js IPC to 0MQ IPC. That's absolutely tolerable, especially given that using 0MQ means communication for the entire application doesn't relay through a single thread. There's no bottleneck this way. Brilliant :) – Mortensen

Cool. Btw what do you measure, latency? Or throughput? Because with node.js you are going to hit a limit of throughput at one point because it's just a single thread after all. You seem to care about this a lot, so it could make sense to perhaps implement your Managers in a language that supports real parallelism. – Zicarelli

Well node.js actually can multithread now using the webworker API addition to JS. You can call require('cluster').fork() in your code and it will start up a second instance of node.js running the same script, but with an altered environment. What I was measuring was the number of IPCs that could be echoed back and forth. Basically having the threads saturate the manager with as many IPCs as possible. So given that there was a mere 1% loss in throughput to use 0MQ, I'd definitely say that's a solid option here ;) – Mortensen

It does not do multithreading, it forks new node.js instances. I just mean that you may be a bit too concerned about the actual transport and not about how the Manager performs. If you have 100 Threads asking the Manager for work, I barely think that the way of transportation will be that much of an issue. The Manager just won't be able to keep up since node.js is single threaded... – Zicarelli

Yeah that's correct. There won't ever be more than a couple IO listener threads per manager. The idea just being that the manager should have its own process to allow it to run without choking up the IO channel and vice versa. I think it'll be fine given the benchmarks with 0MQ :) – Mortensen

Let us know how your game turned out! Been making a similar backend system just like how you explained in your post for my gameserver as well. @Mortensen – Serval

@NiCkNewman Unfortunately I stopped developing on it because I couldn't find anyone to work on the non-programming stuff (3d models, sounds, world design, etc). Not for lack of a sound architecture behind the server though. I think it would have been nice to see it through, but for now it's dead in a ditch. Good luck with your project though! – Mortensen