Using Linux Virtual Server for load balancing of zones in MMO game
Asked Answered
R

4

5

I'm a developer of a MMO game and currently we're at my company facing some scalability issues which, I think, can be resolved with proper clustering of the game world.

I don't really want to reinvent the wheel that's why I think Linux Virtual Server could be a good choice especially with some Level 7 load balancing technique.

I'm currently looking at ktcpvs as a load balancing solution and wonder if it's a proper choice.

The main idea is to have a number of zones("locations" in terms of my game) running on dedicated servers. When a player decides to go to some specific location the load balancer decides which zone server will be actually serving the player(that's actually why I need a Level 7 load balancer)

What do you folks think about all said above?

Update: I posted the same question to LVS users mailing list http://marc.info/?l=linux-virtual-server&m=124976265209769&w=2

Update: I also started the similar topic on the gamedev.net forum http://www.gamedev.net/community/forums/topic.asp?topic_id=544386

Romance answered 8/8, 2009 at 14:1 Comment(4)
Interesting but I think you'll get better help on serverfault.comSpectrogram
I'm aware of this site but I think it's rather dedicated to server administration rather programming...Romance
Sounds like the (an?) answer to your question is one and the same with how Blizzard solved this problem with World of Warcraft.Mickey
And how exactly did they do it? Got the link? ;)Romance
S
10

In order to address your question we need to understand whether you need volume or response, but it is difficult to get both at the same time.

Layer 7 load balancing - is data based application level balancing, so the data content of the network packet needs to be routed to an end-point. You can achieve volume (more users) by implementing routing at the application level, service level or kernel level.

Scalability - I assume you are running out of memory, CPU resources and network bandwidth.

  • Application level - your application logic receives an application packet and routes accordingly.

  • Service level - your system framework (front end service of some kind) receives the packet and through a module - performs the routing (think of custom apache module, even network driver modules - like writing a network filter)

  • Kernel level - Performs routing at network packet level.

The closer you move to the metal, the better your response will be. I suggest using dedicated linux server up-front to perform the routing - go native, not virtual. Use multiple or teamed network adapters for the WAN and a dedicated adapter for each end-point (one+ wan, one each for each connected app server)

If response time is important then you need a kernel/supervisor state solution, it will save you a few context switches but be aware that you need to limit hops at all costs and could better be served by fewer, larger machines and your scalability will always be limited. There is a risk in using KTCPVS, it is quite old and not actively updated. If you judge that it works for you great, otherwise consider writing something akin to a network filter as long as it runs in system state.

If volume is important but response time is secondary, implement a custom built high-speed socket switch built in C++ running in problem/user state. It is the easiest to maintain and will offer the best scalability.

You will need to build some prototypes to figure out what suits your needs best.

Final thoughts -

Before doing any of the above first ensure that you have optimized your game design. You may know most of this, I list it here for the benefit of all.

(a) messages should fit comfortably within one network packet, less than 1500 bytes for most home routers

(b) Try to fit the logic of the routing in your game client instead of your servers. A simple download of a small table with zones and IP addresses to a client will allow you to forego all of the above.

(c) Try to limit zone visibility by to the clients, they should know about their zones and adjacent zones only (if you implement the point b above)

Hope this helps, sorry I cannot be more specific regarding KTCPVS.

Sheehan answered 12/8, 2009 at 20:33 Comment(3)
Thank you very much for the lenghty reply :) It's a pity stackoverflow commenting system sucks and I have to split my reply into several posts.Romance
Yes, I heard KTCPVS is pretty out-dated and I was recommended to have a look at Perdition as an example of application level load balancing. I wonder if it requires LVS or can be used "standalone"(looks like the latter). What I really like about LVS is the ability to use Direct Routing scheme(linuxvirtualserver.org/VS-DRouting.html), I wonder if it can be used on a non-LVS server...Romance
I voted this response but point b) seems a generalization of drhirsch's response. Anyway, a good response.Wealth
T
2

You haven't specified where the bottleneck is. Network Traffic? Disk IO? CPU Cycles?

Assuming you mean a layer 7 load balancer and don't have enough CPU power, I think LVS ist not the optimal choice. I have done Web Server load balancing with LVS, which works straightforward and isn't exactly complicated.

But I think load balancing an MMORP this way needs considerable amounts of additional code in LVS, it might be easier to do the load balancing with a multithreaded application distributed over some multicore server. But this isn't fully scalable, this only gets you to 16 cores without prohibitve cost increase.

Tungstic answered 12/8, 2009 at 16:19 Comment(3)
The main goals behind load balancing is to reduce CPU usage and add redundancy in order to make the game service more fault tolerant. What I need from LVS is a basic low level infrastructure which I don't want to re-invent(e.g LVS/DR schema). And I hoped I would only need to implement the balancer on top of the LVS in order to make use of it. I'm also thinking of building a custom load balancing app and I'm currently looking at Perdition and HAProxy for the inspiration.Romance
Just a thought: I think that network latencies of the clients will be the main problem after you solve CPU, RAM, redundancy, etc. I really hate lags when I play on-line...Wealth
This partially can be resolved using clever inter/extrapolation tricksRomance
A
0

The biggest issue in something like this is what happens when players are near a boundary. Obviously they need to be able to see and interact with each other, but they're on separate servers. So you need some pretty fancy inter-server communication, sometimes just duplicating messages to both servers. It can get even more complicated when someone is near a "corner", and then you have to deal with 4 servers!

The book Massively Multiplayer Game Development has a chapter on "The Pitfalls of Shared Server Boundaries" which covers this issue in detail.

I haven't heard of Linux Virtual Server before now, so I don't understand how it fits. I think your actual server application needs to support this game-specific load balancing, rather than trying to run a cluster and assuming that it will automatically know how to split up your application (which it won't). If I were you, I would write the server program to handle its own piece of land, and it should connect to the pieces of land around it, and then design a server-to-server protocol for the passing of these messages ("here comes a player, I'm going to start telling you about him!" "make sure to tell me about messages near our boundary", "okay the player is out of my territory and into yours, here's his detailed data", etc). I think it's a bit more complicated than just running a different flavor of Linux and assuming you'll get automatic load balancing.

Arrow answered 10/8, 2009 at 14:7 Comment(2)
a. the world of my game is not seamless, so I don't need to handle complex situations as describe(and yes, I have read MMGD 1 and 2) b. please read my first post once again, I'm not assuming LVS will do some magic for me. What I really need is working example of Level 7 balancing working under LVS. Perdition(www.vergenet.net/linux/perdition) seems one of such examples.Romance
Well sorry, I guess I missed the point then... I can't find much info about "level 7 balancing" on Google, and I don't fully understand how applications are written to take advantage of KTCPVS or Linux Virtual Server. I guess I'd have to read more into it. Sorry I couldn't help!Arrow
G
-1

Why are you moving the distribution logic to the loadbalancer? It's a component that's not free and can break. It seems your clients are quite aware of which zone they're in. It seems they could very well connect to zone<n>.example.com. You'd then handle loadbalancing at DNS level.

Gluey answered 12/8, 2009 at 13:32 Comment(1)
Thanks for the idea but I don't think it's going to work for me. Firstly I really don't wan't to hardcode this logic into a client. Ideally I'd like to make zone switching completely transparent for the client. Secondly in case of the failure of one of the zone servers I'd like the balancer to redirect users to one of the least loaded zone servers which will play the role of the missing zone server for some time. Again this logic shouldn't be in the client. As for the loadbalancer being the point of possible failure, I think it can be fixed using heartbeat paired with the second loadbalancer.Romance

© 2022 - 2024 — McMap. All rights reserved.