How does Erlang pass messages between processes on the same node?

Question

Between nodes, message are (must be) passed over TCP/IP. However, by what mechanism are they passed between processes running on the same node? Is TCP/IP used in this case as well? Unix domain sockets? What is the difference in performance between "within node" and "between node" message passing?

Nakitanalani

Answer 1

by what mechanism are they passed between processes running on the same node?

Because Erlang processes on the same node are all running within a single native process — the BEAM emulator — message structures are simply copied into the receiver's message queue. The message structure is copied, rather than simply referenced, for all the standard no-side-effects functional programming reasons.

See erts_send_message() in erts/emulator/beam/erl_message.c in the Erlang sources for more detail. In R15B01, the bits most relevant to your question start at line 980 or so, with the call to erts_queue_message().

If you did choose to run multiple BEAM emulators on a single physical machine, I would guess messages get sent between them the same way as between different physical machines. There's probably no good reason to do that now that BEAM has good SMP support, though.

What is the difference in performance between "within node" and "between node" message passing?

A simple benchmark on your actual hardware would be more useful to you than anecdotal evidence from others.

If you want generalities, however, observe that memory bandwidths are around 20 GByte/sec these days, and that you're unlikely to have a network link faster than 10 Gbit/sec between nodes. That means that while there may be many differences between your actual application and any simple benchmark you perform or find, these differences probably cannot swamp an order of magnitude difference in transfer rate.

If you "only" have a 1 Gbit/sec end-to-end network link between nodes, intranode transfers will probably be over two orders of magnitude faster than internode transfers.

Destitution

Answer 2

"All data in messages between Erlang processes is copied, with the exception of refc binaries on the same Erlang node.":

http://erlang.org/doc/efficiency_guide/processes.html#id2265332

Monohydroxy

If TCP/IP were used as the transport the message would certainly be "copied", but they could also be copied used if some other mechanism were used, so I think that this resolves the question. – Nakitanalani

All data in messages between Erlang processes on the same Erlang node is copied in the memory. Moreover refc binaries are shared between Erlang processes on the same Erlang node. So it's much much faster than TCP/IP. – Monohydroxy

Hmm... Did you mean Erlang processes and Erlang node in the original question? – Monohydroxy