Comparing Unix/Linux IPC

Question

Lots of IPCs are offered by Unix/Linux: pipes, sockets, shared memory, dbus, message-queues...

What are the most suitable applications for each, and how do they perform?

Gentle

dbus is implemented on top of the other IPC types: unix domain sockets, TCP/IP and pipes do far... – Buckskins

See also: Which Linux IPC technique to use? – Dinosaurian

Answer 1

Unix IPC

Here are the big seven:

1. Pipe

   Useful only among processes related as parent/child. Call pipe(2) and fork(2). Unidirectional.

2. FIFO, or named pipe

   Two unrelated processes can use FIFO unlike plain pipe. Call mkfifo(3). Unidirectional.

3. Socket and Unix Domain Socket

   Bidirectional. Meant for network communication, but can be used locally too. Can be used for different protocol. There's no message boundary for TCP. Call socket(2).

4. Message Queue

   OS maintains discrete message. See sys/msg.h.

5. Signal

   Signal sends an integer to another process. Doesn't mesh well with multi-threads. Call kill(2).

6. Semaphore

   A synchronization mechanism for multi processes or threads, similar to a queue of people waiting for bathroom. See sys/sem.h.

7. Shared memory

   Do your own concurrency control. Call shmget(2).

Message Boundary issue

One determining factor when choosing one method over the other is the message boundary issue. You may expect "messages" to be discrete from each other, but it's not for byte streams like TCP or Pipe.

Consider a pair of echo client and server. The client sends string, the server receives it and sends it right back. Suppose the client sends "Hello", "Hello", and "How about an answer?".

With byte stream protocols, the server can receive as "Hell", "oHelloHow", and " about an answer?"; or more realistically "HelloHelloHow about an answer?". The server has no clue where the message boundary is.

An age old trick is to limit the message length to CHAR_MAX or UINT_MAX and agree to send the message length first in char or uint. So, if you are at the receiving side, you have to read the message length first. This also implies that only one thread should be doing the message reading at a time.

With discrete protocols like UDP or message queues, you don't have to worry about this issue, but programmatically byte streams are easier to deal with because they behave like files and stdin/out.

Stonedeaf

I guess you could include semaphores in there, but I see it more as concurrency tool than interprocess communication tool. – Stonedeaf

btw, you can send file descriptors over a Unix Domain Socket [linux.die.net/man/7/unix] – Gelsenkirchen

Don't forget about popen (3) to simplify your life when using pipes. – Deafen

One minor nit: pipe(2) can be used in sibling processes as well -- for instance, the shell is parent to all processes in the pipeline. – Otte

Note that you can have message-oriented unix domain sockets. Unlike internet ones, they're reliable. – Puddle

Is there a benchmark or a qualitative performance comparison of these approaches? – Isotope

this answer is so not useful. you just listed them, didn't compare their performance – Stereoscope

But using a named pipe can be very dangerous when the byte stream is really fast. Some data may be lost if the input data is coming in too fast. I have experienced this. – Sevigny

Answer 2

Shared memory can be the most efficient since you build your own communication scheme on top of it, but it requires a lot of care and synchronization. Solutions are available for distributing shared memory to other machines too.

Sockets are the most portable these days, but require more overhead than pipes. The ability to transparently use sockets locally or over a network is a great bonus.

Message queues and signals can be great for hard real-time applications, but they are not as flexible.

These methods were naturally created for communication between processes, and using multiple threads within a process can complicate things -- especially with signals.

Buckskins

In my experience, named pipes can be as fast, and are safer than almost any other method. – Russi

"Message queues and signals can be great for hard real-time applications"? Could you please explain that in more detail for me? Why message queues are great for hard real-time applications – Academia

Answer 3

Here is a webpage with a simple benchmark: https://sites.google.com/site/rikkus/sysv-ipc-vs-unix-pipes-vs-unix-sockets

As far as I can tell, each has their advantages:

• Pipe I/O is the fastest but needs a parent/child relationship to work.
• Sysv IPC has a defined message boundary and can connect disparate processes locally.
• UNIX sockets can connect disparate processes locally and has higher bandwidth but no inherent message boundaries.
• TCP/IP sockets can connect any processes, even over the network but has higher overhead and no inherent message boundaries.

Bobbi

may be this. sites.google.com/site/rikkus/… – Flashlight

how compare dbus to other? – Stereoscope

DBUS uses one or several of these mechanisms. There is some long standing work on their own IPC mechanism called DBUS1 (or KDBUS...) but it still has not been merged into the mainline kernel. – Bobbi

Link sites.google.com/site/rikkus/… is dead – Kennedy

Answer 4

It's worth noting that lots of libraries implement one type of thing on top of another.

Shared memory doesn't need to use the horrible sysv shared memory functions - it's much more elegant to use mmap() (mmap a file in on a tmpfs /dev/shm if you want it named; mmap /dev/zero if you want forked not exec'd processes to inherit it anonymously). Having said that, it still leaves your processes with some need for synchronisation to avoid problems - typically by using some of the other IPC mechanisms to do synchronisation of access to a shared memory area.

Advocaat

I'd never heard of mmaping /dev/zero before. Ingenious! You mention that it can only be shared with children - but can you send the file descriptor you're using to an unrelated process using cmsg/SCM_RIGHTS over a unix-domain socket, and end up with a shared mapping there? Or is it the mapping you inherit, not the file descriptor? Even if it does work, you still need the socket somewhere in the filesystem to do it, so even if the mapping is anonymous, the socket used to set it up isn't. Guh. IPC is hard. Let's go shopping! – Puddle

mmaping /dev/zero is actually used by some kinds of memory allocation. But the advantage is that if you use MAP_SHARED, it gets shared with your fork()ed child processes (normaly memory is logically copied). Can you share it with an unrelated process? I don't think so. I suspect the mmap() call needs to be shared, not the file descriptor. – Advocaat