The actor model: Why is Erlang/OTP special? Could you use another language?

Question

I've been looking into learning Erlang/OTP, and as a result, have been reading (okay, skimming) about the actor model.

From what I understand, the actor model is simply a set of functions (run within lightweight threads called "processes" in Erlang/OTP), which communicate with each other only via message passing.

This seems fairly trivial to implement in C++, or any other language:

class BaseActor {
    std::queue<BaseMessage*> messages;
    CriticalSection messagecs;
    BaseMessage* Pop();
public:
    void Push(BaseMessage* message)
    {
        auto scopedlock = messagecs.AquireScopedLock();
        messagecs.push(message);
    }
    virtual void ActorFn() = 0;
    virtual ~BaseActor() {} = 0;
}

With each of your processes being an instance of a derived BaseActor. Actors communicate with each other only via message-passing. (namely, pushing). Actors register themselves with a central map on initialization which allows other actors to find them, and allows a central function to run through them.

Now, I understand I'm missing, or rather, glossing over one important issue here, namely: lack of yielding means a single Actor can unfairly consume excessive time. But are cross-platform coroutines the primary thing that makes this hard in C++? (Windows for instance has fibers.)

Is there anything else I'm missing, though, or is the model really this obvious?

Tetradymite

The purpose of a programming language is to aid in the expression of an idea or specification. The actor model is implicit in Erlang, so while you can express your ideas in the model in either language, it'll be much better in Erlang because the boiler-plate is done for you. – Hypnosis

@GMan once the boiler plate is done (it would be a one-time think, I'd think) what is the advantage? – Ashleeashleigh

@SethCarnegie: That is indeed the gist of my question. – Tetradymite

erlang processes can reside on the same machine or on different physical machines (and the actual code that you write to do this is more or less identical), so your example seems to be a gross simplification. And, what about hot swapping code, can c++ do that easily too? Are your c++ actors memory sandboxed? – Nazareth

by the way scala has an actor model as well (Akka) so actors are definitely not limited to erlang. But I don't think it's quite as easy to do in c++ as you think (at least not without big limitations). – Nazareth

@Kevin: Same machine/different machine: Not impossible. Harder than a code snippet, though. But again: one time implementation cost. Hot-swapping code: Unrelated to the actor model, and thus, out of the scope of this question. Memory sandboxing: No. It is also probably impossible to do this(in C++), nor do I see it as a requirement. C++ is not a nanny language. – Tetradymite

isn't everything a one time implementation cost? Seriously though, I think the lack of a sandboxed environment is a serious drawback, since one actor can bring the whole system down. – Nazareth

@Kevin: True enough. The point I was trying to make though, is once the relatively small effort of implementation is done, you continue to use the same language. Which means you don't need to learn another language's idiosyncrasies, nor are you saddled with its performance. Understanding a language, and performance are both critical to well designed distributed systems. Why would you switch languages to write the hardest parts of your code, if the cost to do so, both in the short run, and the long run is worse. My question was: is the actor model really this simple? If not, what am I missing? – Tetradymite

I don't know all the answers but you might start by reading about the Akka project which is actors for java. It does have some limitations relative to erlang actors so that might point you to what is easy and what is hard. – Nazareth

If you can implement safe, reliable, concurrent and maintainable code in C++ with as little effort as people do in erlang, then go right ahead. There's tons missing from this snippet, though. The core of erlang is reliability. If a process is incapable of doing its task, it fails and that failure message is propagated through the system allowing complex graphs of dependency to reorganize themselves on various types of outages (or bugs). You can do it, but you should be asking why nobody does. That's what leads to new languages. – Alforja

@Dustin: That's exactly what I'm asking. Re: What am I missing. The snippet was written in a minute, with about as much thought put in it, so obviously it's not "complete". – Tetradymite

@Seth: "it would be a one-time think, I'd think" Yeah...no. I don't know of any person that writes something perfect the first time. You can't think of any way this code can be improved? Or the existing Erlang implementation? – Hypnosis

@GMan I didn't mean you'd write one thing one time, I meant that you'd not have to write it for every program. – Ashleeashleigh

I'm surprised that hasn't been closed by the SO gestapo. But you're right that actor modeling can be done in C++ and even C. Obviously it requires a great deal more of effort and can become syntactically messy. – Endodontist

See letitcrash.com/post/20964174345/… – Outburst

Answer 1

The C++ code does not deal with fairness, isolation, fault detection or distribution which are all things which Erlang brings as part of its actor model.

• No actor is allowed to starve any other actor (fairness)
• If one actor crashes, it should only affect that actor (isolation)
• If one actor crashes, other actors should be able to detect and react to that crash (fault detection)
• Actors should be able to communicate over a network as if they were on the same machine (distribution)

Also the beam SMP emulator brings JIT scheduling of the actors, moving them to the core which is at the moment the one with least utilization and also hibernates the threads on certain cores if they are no longer needed.

In addition all the libraries and tools written in Erlang can assume that this is the way the world works and be designed accordingly.

These things are not impossible to do in C++, but they get increasingly hard if you add the fact that Erlang works on almost all of the major hw and os configurations.

edit: Just found a description by Ulf Wiger about what he sees erlang style concurrency as.

Plerre

I would definitely include process isolation and error handling in the erlang concurrency model, otherwise what Ulf writes is very good. – Lennox

All of the properties you listed are provided by the operating system to processes. C++ programs can easily make use of them, as can any other program. I think the key to Erlang is that its actors are far cheaper than OS processes for providing those properties. As a result, actors can be used more freely. – Thoroughwort

@Thoroughwort Yes, Erlang processes are very cheap because/so that concurrency is the basic abstraction of structuring applications. We prefer to call them processes not actors, we hadn't heard of actors when we designed Erlang. :-) – Lennox

Answer 2

I don't like to quote myself, but from Virding's First Rule of Programming

Any sufficiently complicated concurrent program in another language contains an ad hoc informally-specified bug-ridden slow implementation of half of Erlang.

With respect to Greenspun. Joe (Armstrong) has a similar rule.

The problem is not to implement actors, that's not that difficult. The problem is to get everything working together: processes, communication, garbage collection, language primitives, error handling, etc ... For example using OS threads scales badly so you need to do it yourself. It would be like trying to "sell" an OO language where you can only have 1k objects and they are heavy to create and use. From our point of view concurrency is the basic abstraction for structuring applications.

Getting carried away so I will stop here.

Lennox

"so I will stop here": more would have been interesting – Venom

Answer 3

This is actually an excellent question, and has received excellent answers that perhaps are yet unconvincing.

To add shade and emphasis to the other great answers already here, consider what Erlang takes away (compared to traditional general purpose languages such as C/C++) in order to achieve fault-tolerance and uptime.

First, it takes away locks. Joe Armstrong's book lays out this thought experiment: suppose your process acquires a lock and then immediately crashes (a memory glitch causes the process to crash, or the power fails to part of the system). The next time a process waits for that same lock, the system has just deadlocked. This could be an obvious lock, as in the AquireScopedLock() call in the sample code; or it could be an implicit lock acquired on your behalf by a memory manager, say when calling malloc() or free().

In any case, your process crash has now halted the entire system from making progress. Fini. End of story. Your system is dead. Unless you can guarantee that every library you use in C/C++ never calls malloc and never acquires a lock, your system is not fault tolerant. Erlang systems can and do kill processes at will when under heavy load in order make progress, so at scale your Erlang processes must be killable (at any single point of execution) in order to maintain throughput.

There is a partial workaround: using leases everywhere instead of locks, but you have no guarantee that all the libraries you utilize also do this. And the logic and reasoning about correctness gets really hairy quickly. Moreover leases recover slowly (after the timeout expires), so your entire system just got really slow in the face of failure.

Second, Erlang takes away static typing, which in turn enables hot code swapping and running two versions of the same code simultaneously. This means you can upgrade your code at runtime without stopping the system. This is how systems stay up for nine 9's or 32 msec of downtime/year. They are simply upgraded in place. Your C++ functions will have to be manually re-linked in order to be upgraded, and running two versions at the same time is not supported. Code upgrades require system downtime, and if you have a large cluster that cannot run more than one version of code at once, you'll need to take the entire cluster down at once. Ouch. And in the telecom world, not tolerable.

In addition Erlang takes away shared memory and shared shared garbage collection; each light weight process is garbage collected independently. This is a simple extension of the first point, but emphasizes that for true fault tolerance you need processes that are not interlocked in terms of dependencies. It means your GC pauses compared to java are tolerable (small instead of pausing a half-hour for a 8GB GC to complete) for big systems.

Interminable

First, you can use lock_guard, which will release your lock in case the program crashes. Second, you can implement hot swap system in C++, but it's a pain in the... The problem with concurrency is that synchronization primitives, even atomics, introduce memory fences and barriers, and slow down. The more threads you have, the more you will slow down. Erlang, as clojure or haskell, don't use mutexes or atomics, which forces the developer to lay down the problem in a different way. This is a very efficient way of solving concurrency problems – Pullulate

Sounds valid, but that's only C++ comparison and C++ is always an easy target to blame. Isn't it possible to have this in Java (or Clojure), for example? Locks in Java are safe, and there are ways to compile/load code at runtime (in Clojure this is also very easy). – Thundery

Answer 4

There are actual actor libraries for C++:

• http://actor-framework.org/

• http://www.theron-library.com/

And a list of some libraries for other languages.

Despotic

libcppa has been renamed to C++ Actor Framework (CAF) recently. The new URL is: github.com/actor-framework/actor-framework – Gianna

Answer 5

It is a lot less about the actor model and a lot more about how hard it is to properly write something analogous to OTP in C++. Also, different operating systems provide radically different debugging and system tooling, and Erlang's VM and several language constructs support a uniform way of figuring out just what all those processes are up to which would be very hard to do in a uniform way (or maybe do at all) across several platforms. (It is important to remember that Erlang/OTP predates the current buzz over the term "actor model", so in some cases these sort of discussions are comparing apples and pterodactyls; great ideas are prone to independent invention.)

All this means that while you certainly can write an "actor model" suite of programs in another language (I know, I have done this for a long time in Python, C and Guile without realizing it before I encountered Erlang, including a form of monitors and links, and before I'd ever heard the term "actor model"), understanding how the processes your code actually spawns and what is happening amongst them is extremely difficult. Erlang enforces rules that an OS simply can't without major kernel overhauls -- kernel overhauls that would probably not be beneficial overall. These rules manifest themselves as both general restrictions on the programmer (which can always be gotten around if you really need to) and basic promises the system guarantees for the programmer (which can be deliberately broken if you really need to also).

For example, it enforces that two processes cannot share state to protect you from side effects. This does not mean that every function must be "pure" in the sense that everything is referentially transparent (obviously not, though making as much of your program referentially transparent as practical is a clear design goal of most Erlang projects), but rather that two processes aren't constantly creating race conditions related to shared state or contention. (This is more what "side effects" means in the context of Erlang, by the way; knowing that may help you decipher some of the discussion questioning whether Erlang is "really functional or not" when compared with Haskell or toy "pure" languages.)

On the other hand, the Erlang runtime guarantees delivery of messages. This is something sorely missed in an environment where you must communicate purely over unmanaged ports, pipes, shared memory and common files which the OS kernel is the only one managing (and OS kernel management of these resources is necessarily extremely minimal compared to what the Erlang runtime provides). This doesn't meant that Erlang guarantees RPC (anyway, message passing is not RPC, nor is it method invocation!), it doesn't promise that your message is addressed correctly, and it doesn't promise that a process you're trying to send a message to exists or is alive, either. It just guarantees delivery if the thing your sending to happens to be valid at that moment.

Built on this promise is the promise that monitors and links are accurate. And based on that the Erlang runtime makes the entire concept of "network cluster" sort of melt away once you grasp what is going on with the system (and how to use erl_connect...). This permits you to hop over a set of tricky concurrency cases already, which gives one a big head start on coding for the successful case instead of getting mired in the swamp of defensive techniques required for naked concurrent programming.

So its not really about needing Erlang, the language, its about the runtime and OTP already existing, being expressed in a rather clean way, and implementing anything close to it in another language being extremely hard. OTP is just a hard act to follow. In the same vein, we don't really need C++, either, we could just stick to raw binary input, Brainfuck and consider Assembler our high level language. We also don't need trains or ships, as we all know how to walk and swim.

All that said, the VM's bytecode is well documented, and a number of alternative languages have emerged that compile to it or work with the Erlang runtime. If we break the question into a language/syntax part ("Do I have to understand Moon Runes to do concurrency?") and a platform part ("Is OTP the most mature way to do concurrency, and will it guide me around the trickiest, most common pitfalls to be found in a concurrent, distributed environment?") then the answer is ("no", "yes").

Shonna

Answer 6

Casablanca is another new kid on the actor model block. A typical asynchronous accept looks like this:

PID replyTo;
NameQuery request;
accept_request().then([=](std::tuple<NameQuery,PID> request)
{
   if (std::get<0>(request) == FirstName)
       std::get<1>(request).send("Niklas");
   else
       std::get<1>(request).send("Gustafsson");
}

(Personally, I find that CAF does a better job at hiding the pattern matching behind a nice interface.)

Gianna