Example of using named pipes in Linux shell (Bash)

Question

Can someone post a simple example of using named pipes in Bash on Linux?

Catechism

Answer 1

One of the best examples of a practical use of a named pipe...

From http://en.wikipedia.org/wiki/Netcat:

Another useful behavior is using netcat as a proxy. Both ports and hosts can be redirected. Look at this example:

nc -l 12345 | nc www.google.com 80

Port 12345 represents the request.

This starts a nc server on port 12345 and all the connections get redirected to google.com:80. If a web browser makes a request to nc, the request will be sent to google but the response will not be sent to the web browser. That is because pipes are unidirectional. This can be worked around with a named pipe to redirect the input and output.

mkfifo backpipe
nc -l 12345  0<backpipe | nc www.google.com 80 1>backpipe

Gouda

@hft How about mkfifo backpipe; nc -l 12345 0<backpipe | nc www.google.com 80 1>backpipe? – Furbish

While this serves as a useful example, if you're building proxies of this nature you'd generally be better off using socat, which is a Unix command-line tool that essentially implements a very fully featured domain-specific language for building proxies and (to a more limited degree) servers. This particular example would be done more reliably and efficiently with socat STDIO TCP:www.google.com:80. – Quinte

Am I the only one who thinks that this example is a bit overboard and not simple at all? I never used named pipes and wanted a really basic usage example, in that regard the other answers are better I think. – Lenssen

Answer 2

Here are the commands:

mkfifo named_pipe
echo "Hi" > named_pipe &
cat named_pipe

The first command creates the pipe.

The second command writes to the pipe (blocking). The & puts this into the background so you can continue to type commands in the same shell. It will exit when the FIFO is emptied by the next command.

The last command reads from the pipe.

Rainstorm

I would change the # to $ so its not all commented (and not run as root!) – Inrush

It's customary for "#" to refer to a root prompt (ie, a prompt in a root shell). There's nothing here that would require running in a root shell. – Huppah

The echo will block so this won't run if executed in the same shell unless second line is place in the background with an ending &. – Ruthieruthless

Answer 3

Open two different shells, and leave them side by side. In both, go to the /tmp/ directory:

cd /tmp/

In the first one type:

mkfifo myPipe
echo "IPC_example_between_two_shells">myPipe

In the second one, type:

while read line; do echo "What has been passed through the pipe is ${line}"; done<myPipe

First shell won't give you any prompt back until you execute the second part of the code in the second shell. It's because the fifo read and write is blocking.

You can also have a look at the FIFO type by doing a ls -al myPipe and see the details of this specific type of file.

Next step would be to embark the code in a script!

Perceptible

Is it posible to make non blocking writes to the fifo? – Uncomfortable

Answer 4

Creating a named pipe

$ mkfifo pipe_name

On Unix-likes named pipe (FIFO) is a special type of file with no content. The mkfifo command creates the pipe on a file system (assigns a name to it), but doesn't open it. You need to open and close it separately like any other file.

Using a named pipe

Named pipes are useful when you need to pipe from/to multiple processes or if you can't connect two processes with an anonymous pipe. They can be used in multiple ways:

• In parallel with another process:

  $ echo 'Hello pipe!' > pipe_name &       # runs writer in a background
  $ cat pipe_name
  Hello pipe!
  

  Here writer runs along the reader allowing real-time communication between processes.

• Sequentially with file descriptors:

  $ # open the pipe on auxiliary FD #5 in both ways (otherwise it will block),
  $ # then open descriptors for writing and reading and close the auxiliary FD
  $ exec 5<>pipe_name 3>pipe_name 4<pipe_name 5>&-
  $
  $ echo 'Hello pipe!' >&3                 # write into the pipe through FD #3
    ...
  $ exec 3>&-                              # close the FD when you're done
  $                                        # (otherwise reading will block)
  $ cat <&4
  Hello pipe!
  ...
  $ exec 4<&-
  

  In fact, communication through a pipe can be sequential, but it's limited to 64 KB (buffer size).
  It's preferable to use descriptors to transfer multiple blocks of data in order to reduce overhead.

• Conditionally with signals:

  $ handler() {
  >     cat <&3
  >
  >     exec 3<&-
  >     trap - USR1                        # unregister signal handler (see below)
  >     unset -f handler writer            # undefine the functions
  > }
  $
  $ exec 4<>pipe_name 3<pipe_name 4>&-
  $ trap handler USR1                      # register handler for signal USR1
  $
  $ writer() {
  >     if <condition>; then
  >         kill -USR1 $PPID               # send the signal USR1 to a specified process
  >         echo 'Hello pipe!' > pipe_name
  >     fi
  > }
  $ export -f writer                       # pass the function to child shells
  $
  $ bash -c writer &                       # can actually be run sequentially as well
  $
  Hello pipe!
  

  FD allows data transfer to start before the shell is ready to receive it. Required when used sequentially.
  The signal should be sent before data to prevent a deadlock if pipe buffer will fill up.

Destroying a named pipe

The pipe itself (and its content) gets destroyed when all descriptors to it are closed. What's left is just a name.
To make the pipe anonymous and unavailable under the given name (can be done when the pipe is still open) you could use the rm con­sole com­mand (it's the opposite of mkfifo command):

$ rm pipe_name

Scenery

There was no complete(-ish) example, so I wrote one. – Scenery

Answer 5

Terminal 1:

$ mknod new_named_pipe p
$ echo 123 > new_named_pipe

• Terminal 1 created a named pipe.
• It wrote data in it using echo.
• It is blocked as there is no receiving end (as pipes both named and unnamed need receiving and writing ends to it)

Terminal 2:

$ cat new_named_pipe
$ 123
$ 

• From Terminal 2, a receiving end for the data is added.
• It read the data in it using cat.
• Since both receiving and writing ends are there for the new_named_pipe it displays the information and blocking stops

Named pipes are used everywhere in Linux, most of the char and block files we see during ls -l command are char and block pipes (All of these reside at /dev). These pipes can be blocking and non-blocking, and the main advantage is these provides the simplest way for IPC.

Malapert

Answer 6

I want to say that you can create background process for pipe operation instead of main cmd. It will help you a lot with custom log processing.

Let we have the following multiline command.

make target1 \
  --opt 1 \
  --opt 2

We need to grep something from make on the fly. It is hard to replace multine command with ${cmd} >log & (manually or using tools like sed), so you can replace it with >log ${cmd} instead:

mkfifo log
grep "something" log &
>log make target1 \
  --opt 1 \
  --opt 2

Thank you.

Yukoyukon