Building a fault-tolerant soft real-time web application with Erlang/OTP

Question

I would like to build a fault-tolerant soft real-time web application for a pizza delivery shop. It should help the pizza shop to accept phone calls from customers, put them as orders into the system (via a CRM web client) and help the dispatchers to assign delivery drivers to the orders.

These goals are nothing unusual, but I would like to make the service available 24/7, i.e. to make it fault-tolerant. Moreover, I would like to make it work very fast and to be very responsive.

Below is a very simple architecture view for such an application.

The problem is that I do not know how to use all of the Erlang/OTP goodness to make the application very responsive and fault tolerant.

Here are my questions:

1. Which system elements should be replicated in order to provide fault-tolerance and how should I do that? I know that I can store the status of each vehicle (coordinates, assigned orders, etc.) in a replicated Mnesia database. Is that a right way to go?
2. Which data-storage services should be conventional SQL-based (e.g. based on boss_db) and which should be done on Mnesia to provide for very fast response? Is it ok to use a conventional SQL database to store customer records and history in a such fault-tolerant and highly-responsive application?
3. Should I try to store all the data for all the services (customers, vehicles status, etc.) in RAM in order to make the application highly-responsive?
4. Should I store the persistent vehicle data (id, capacity, etc.) in a conventional SQL database and store the real-time data (coordinates, assigned orders, orders in the trunk, etc.) in a Mnesia database to make the application more real-time responsive?

Sontich

If you don't replicate all your application logic and data, how would you make the service 24/7 in case of e.g. a database server failure ;) – Indiana

Could you provide a reference on how to replicate data in a Erlang/OTP system which has 2+ nodes dealing with logic? – Sontich

process group, send to all, conflict resolution or simply use riak/couchdb. Or you look at mnesia. But doing this by your own will be a really really hard task. – Indiana

Answer 1

First of all, this is a big question, but i will try to break it down. Lets first look at the facts. Its a web service. Which means we have these layers: Web Server , Middle ware application and then Data Storage. In most highly available applications, the Data Storage layer must have redundancy through replication and load managed through Distribution. In most real-world applications, you will not need to store anything in RAM, unless the application is really real-time in nature, such as a Multi-player Game Server or A telecom Switch. So, your kind of application, in this case really, no need for RAM storage (maybe some kind of caching here and there, as we are going to see.)

Now, this kind of application, involves different kind of data,information that cannot have the same form at any one time, hence, using an RDMS will force you to arrange everything the same way. My suggestion is that you learn to use any document oriented database, a NoSQL DB or key-value system because they are well modelled for real-world complexities. More information about any kind of storage is found in this pdf. I suggest that you use Couch base server whereby your data will simply be JSON documents, schemaless and can be evolved as your application grows. It comes with distribution and replication, just the way any application ever needed it. You can add servers or remove servers,at run-time and the entire system just keeps re-balancing itself. It also comes with memcached built in for caching, so for the IN-Memory part you were talking about, caching will do everything for you.

After the Storage, lets talk about the middle ware. i want to talk about the web server as being part of the middle ware. You will need a very stable Web server, depending on the load, and being that you want to use Erlang, i suggest yaws web server and learn to do RESTFUL services with it using appmods. Using Proxies sunch as Nginx infront of a cluster of web servers may help in load management. Atleast there are several ways of load-balancing infront of web servers. After this, you will need an OTP application. An OTP application doesnot necessary have to have gen_servers. But as you will learn, you will discover, really, where you need parallelisation or where you need sequential code. Its however, worrying that you want to use something which you have not yet mastered. please follow this web book and this Orielly book to help you master everything about Erlang. You could find it useful to try out Chicago Boss and Mochiweb or Misultin Http server libraries.

The other thing i should mention about doing this in Erlang is that, you need to master your Data Structures and an effcient way to work with them. Poor choice of data structures may cause problems. Test and Test everything at each step. Use records everywhere if possible and check memory consumption at each stage. There is just lots to say about this question , but hopefully, others are also going to post their thinking.

Topminnow

The oreilly book is linked to learnyousomeerlang. Would have edited it, but i don't know which book you mean, there are at least 2 oreilly books ;) – Indiana

i have fixed the link. Sorry about that – Topminnow

@MuzaayaJoshua What is the best approach/tool to make Data Storage store everything in RAM in case of Multi-player game server? – Sontich

Maybe this will help you: jeena.net/t/GGS.pdf But you should first design your architecture. Storing everything on every node will not scale. And your main problem of the logic will be the distribution of players and inter-node communication. You should do extensive research in game server design. Have a look how others solved their problem. Your scaling will only work really great if you can minimize inter-node communication. In best case two erland nodes do not really have to communicate with each other. Every node has only to do messageing between players within it's node. – Indiana

look at memcached as well – Topminnow

Answer 2

hack this game: https://github.com/synrc/games all real-time, low latency, pub/sub, database, architecture questions are there, written as a state-of-the-art software. I suggest to use gen_fsm to control states in your app as done in 'okay' supervisors. riak is integrated with kvs lib, that has a good support for social updates also. n2o choosed cowboy server, in my view, the best server around. http://www.ostinelli.net/a-comparison-between-misultin-mochiweb-cowboy-nodejs-and-tornadoweb/

Machzor