I often use pipes in Bash, e.g.:
dmesg | less
Although I know what this outputs, it takes dmesg and lets me scroll through it with less, I do not understand what the | is doing. Is it simply the opposite of >?
| does?A Unix pipe connects the STDOUT (standard output) file descriptor of the first process to the STDIN (standard input) of the second. What happens then is that when the first process writes to its STDOUT, that output can be immediately read (from STDIN) by the second process.
Using multiple pipes is no different than using a single pipe. Each pipe is independent, and simply links the STDOUT and STDIN of the adjacent processes.
Your third question is a little bit ambiguous. Yes, pipes, as such, are consistent everywhere in a bash script. However, the pipe character | can represent different things. Double pipe (||), represents the "or" operator, for example.
man 2 pipe for the C library function that is used to create them. They are basically a buffer managed by the kernel for you. –
Affine In Linux (and Unix in general) each process has three default file descriptors:
Normally, when you run a simple program these file descriptors by default are configured as following:
Bash provides several operators to change this behavior (take a look to the >, >> and < operators for example). Thus, you can redirect the output to something other than the standard output or read your input from other stream different than the keyboard. Specially interesting the case when two programs are collaborating in such way that one uses the output of the other as its input. To make this collaboration easy Bash provides the pipe operator |. Please note the usage of collaboration instead of chaining. I avoided the usage of this term since in fact a pipe is not sequential. A normal command line with pipes has the following aspect:
> program_1 | program_2 | ... | program_n
The above command line is a little bit misleading: user could think that program_2 gets its input once the program_1 has finished its execution, which is not correct. In fact, what bash does is to launch ALL the programs in parallel and it configures the inputs outputs accordingly so every program gets its input from the previous one and delivers its output to the next one (in the command line established order).
Following is a simple example from Creating pipe in C of creating a pipe between a parent and child process. The important part is the call to the pipe() and how the parent closes fd1 (writing side) and how the child closes fd1 (writing side). Please, note that the pipe is a unidirectional communication channel. Thus, data can only flow in one direction: fd1 towards fd[0]. For more information take a look to the manual page of pipe().
#include <stdio.h>
#include <unistd.h>
#include <sys/types.h>
int main(void)
{
int fd[2], nbytes;
pid_t childpid;
char string[] = "Hello, world!\n";
char readbuffer[80];
pipe(fd);
if((childpid = fork()) == -1)
{
perror("fork");
exit(1);
}
if(childpid == 0)
{
/* Child process closes up input side of pipe */
close(fd[0]);
/* Send "string" through the output side of pipe */
write(fd[1], string, (strlen(string)+1));
exit(0);
}
else
{
/* Parent process closes up output side of pipe */
close(fd[1]);
/* Read in a string from the pipe */
nbytes = read(fd[0], readbuffer, sizeof(readbuffer));
printf("Received string: %s", readbuffer);
}
return(0);
}
Last but not least, when you have a command line in the form:
> program_1 | program_2 | program_3
The return code of the whole line is set to the last command. In this case program_3. If you would like to get an intermediate return code you have to set the pipefail or get it from the PIPESTATUS.
In short, as described, there are three key 'special' file descriptors to be aware of. The shell by default send the keyboard to stdin and sends stdout and stderr to the screen:
A pipeline is just a shell convenience which attaches the stdout of one process directly to the stdin of the next:
There are a lot of subtleties to how this works, for example, the stderr stream might not be piped as you would expect, as shown below:
I have spent quite some time trying to write a detailed but beginner friendly explanation of pipelines in Bash. The full content is at:
https://effective-shell.com/part-2-core-skills/thinking-in-pipelines/
Every standard process in Unix has at least three file descriptors, which are sort of like interfaces:
The pipe connects the standard output of the process to the left to the standard input of the process of the right. You can think of it as a dedicated program that takes care of copying everything that one program prints, and feeding it to the next program (the one after the pipe symbol). It's not exactly that, but it's an adequate enough analogy.
Each pipe operates on exactly two things: the standard output coming from its left and the input stream expected at its right. Each of those could be attached to a single process or another bit of the pipeline, which is the case in a multi-pipe command line. But that's not relevant to the actual operation of the pipe; each pipe does its own.
The redirection operator (>) does something related, but simpler: by default it sends the standard output of a process directly to a file. As you can see it's not the opposite of a pipe, but actually complementary. The opposite of > is unsurprisingly <, which takes the content of a file and sends it to the standard input of a process (think of it as a program that reads a file byte by byte and types it in a process for you).
A pipe takes the output of a process, by output I mean the standard output (stdout on UNIX) and passes it on the standard input (stdin) of another process. It is not the opposite of the simple right redirection > which purpose is to redirect an output to another output.
For example, take the echo command on Linux which is simply printing a string passed in parameter on the standard output. If you use a simple redirect like :
echo "Hello world" > helloworld.txt
the shell will redirect the normal output initially intended to be on stdout and print it directly into the file helloworld.txt.
Now, take this example which involves the pipe :
ls -l | grep helloworld.txt
The standard output of the ls command will be outputed at the entry of grep, so how does this work?
Programs such as grep when they're being used without any arguments are simply reading and waiting for something to be passed on their standard input (stdin). When they catch something, like the ouput of the ls command, grep acts normally by finding an occurence of what you're searching for.
Pipes are very simple like this.
You have the output of one command. You can provide this output as the input into another command using pipe. You can pipe as many commands as you want.
ex: ls | grep my | grep files
This first lists the files in the working directory. This output is checked by the grep command for the word "my". The output of this is now into the second grep command which finally searches for the word "files". Thats it.
The pipe operator takes the output of the first command, and 'pipes' it to the second one by connecting stdin and stdout. In your example, instead of the output of dmesg command going to stdout (and throwing it out on the console), it is going right into your next command.
yes | rm -r * as an alternative to rm -rf * work evn though yes never finishes executing –
Shankle | puts the STDOUT of the command at left side to the STDIN of the command of right side.
If you use multiple pipes, it's just a chain of pipes. First commands output is set to second commands input. Second commands output is set to next commands input. An so on.
It's available in all Linux/widows based command interpreter.
All of these answere are great. Something that I would just like to mention, is that a pipe in bash (which has the same concept as a unix/linux, or windows named pipe) is just like a pipe in real life. If you think of the program before the pipe as a source of water, the pipe as a water pipe, and the program after the pipe as something that uses the water (with the program output as water), then you pretty much understand how pipes work. And remember that all apps in a pipeline run in parallel.
Regarding the efficiency issue of pipe:
If you treat each unix command as a standalone module,
but you need them to talk to each other using text as a consistent interface,
how can it be done?
cmd input output
echo "foobar" string "foobar"
cat "somefile.txt" file *string inside the file*
grep "pattern" "a.txt" pattern, input file *matched string*
You can say | is a metaphor for passing the baton in a relay marathon.
Its even shaped like one!
cat -> echo -> less -> awk -> perl is analogous to cat | echo | less | awk | perl.
cat "somefile.txt" | echo
cat pass its output for echo to use.
What happens when there is more than one input?
cat "somefile.txt" | grep "pattern"
There is an implicit rule that says "pass it as input file rather than pattern" for grep.
You will slowly develop the eye for knowing which parameter is which by experience.
Future readers: I've tried answering each of OP's questions, and whenever I've deemed it necessary, I've divided the answer to a question into a few headers to make it a more structured and pleasant reading experience. Note that the answers to OP's questions require some foundational knowledge, and in cases where I think a full answer would deviate from the question at hand I've linked to other relevant SO questions/answers.
| is doing. Is it simply the opposite of >?No, | isn't the opposite of >. A pipeline, which is what | represents, is defined as a set of commands,
which are set up to redirect their input/output into each other. For example, in the pipeline
processA | processB | processC
processA is set up to redirect its output into processB's input, and processB to redirect its output into processC's input, which ultimately sends its output to the console.
In other words, a pipeline redirects the input and/or output of processes. As a convenience, the shell represents a pipeline with the metacharacter |, however you can implement a pipeline yourself (which we do below). Similar to pipelining, the shell conveniently provides further I/O functionalities via a few more metacharacters. For example:
<), where a process's standard input is set up to take input from a file instead of the default input, which is usually the keyboard, i.e., processA < file.>), where a process's standard output is set up to send output into a file instead of the default output, which is usually the console, i.e., processA > file.| does?Yes, there is and it comes from the person who first conceived the concept of a pipe! American Computer Scientist Doug McIlroy likened it to a "garden hose" in a note (emphasis mine):
- We should have some ways of coupling programs like garden hose--screw in another segment when it becomes when it becomes necessary to massage data in another way. This is the way of IO also.
It's worth mentioning this was some years before Unix was invented, and therefore the concept predates Unix.
Now on with the metaphor... Conventionally, a Unix pipe has been considered a half-duplex or unidirectional pipe, meaning that the data flows in one direction only. Thus, if you can imagine a unidirectional garden hose, with an end where water pushes through (let's call this the write end) and an end where water comes through (let's call this the read end), we can have this non-award-winning drawing (Drawn with tldraw):
In the drawing, the water can only be sent to the garden hose's write end and the water comes from what we call the water source. The data makes it to the garden hose's read end, from which it can be read by what we call a water sink. With dmesg | less, dmesg's output is the data source which would be connected to the pipe's write end; and less's input is the data sink which would be connected to the pipe's read end. After setting the pipe between dmesg and less, the data that dmesg produces flows through the pipe and into less.
In order to create a pipe, we use the system call pipe(). To summarize the manpage (although you're encouraged to read and understand it), pipe() takes an array of two integers, and it places a file descriptor in each slot. File descriptor at index 0 is the pipe's read end and file descriptor at index 1 is the pipe's write end. To illustrate this, we've the following trivial and mostly useless program in order to show that the data flows from one end of the pipe to the other end:
// useless-program.c
#include <stdio.h>
#include <unistd.h>
int main(int argc, char *argv[]) {
// we set up a pipe!
int pipefd[2];
pipe(pipefd);
// now pipefd[0] contains a file descriptor representing the pipe's read end
// and pipefd[1] contains a file descriptor representing the pipe's write end.
// we write a value on the pipe's write end, and display it on the console.
int sent_value = 100;
write(pipefd[1], &sent_value, sizeof(sent_value));
printf("Sent value: %d\n", sent_value);
// we read a value from the pipe's read end, and display it on the console.
int received_value;
read(pipefd[0], &received_value, sizeof(received_value));
printf("Received value: %d\n", received_value);
return 0;
}
Compiling and running it:
$ gcc useless-program.c && ./a.out
Sent value: 100
Received value: 100
Now you might argue this is quite the roundabout way to use a value within program and you'd be right. In fact Stevens agrees with you in Advanced Programming in the Unix Environment by stating that "a pipe in a single process is next to useless". After all, pipes are a mechanism for inter-process communication, i.e., you need at least two processes.
The short answer is that a pipeline with several pipes isn't any different from a pipeline with a single pipe so let's start with that but first let's talk about processes.
Abstractly, a process is a container for a running program. Thus, when you run the commands dmesg and less, two processes are spawned/forked to run these programs.
As a container, a process contains information/data such as the program/text, a program counter (PC), opened files, etc. to enable the process to carry out its job. We're specifically interested on the opened files. When you launch a command in the terminal, the shell creates a process to run the command and one of the things the shell does is to open three files for that specific process:
stdin, which stands for standard input and where the process can read data from.stdout, which stands for standard output and where the process can send normal data to.stderr, which stands for standard error and where the process can send error data to.A process doesn't deal with the opened files directly though, instead it deals with what's known as a file descriptor, which is a non-negative integer that represents the opened file. When the process wants to read from or write from a specific file, it uses this file descriptor to do so. I won't go into details about file descriptors (see here) but what you should know is:
stdin, stdout, and stderr by default. Unless you close these file descriptors, the next file descriptor within a given process will be 3, next one will be 4, and so on.Takeaway: By default, any newly-created process has three opened files available to it, namely stdin, stdout, and stderr. Collectively, they're known as a process's standard streams. A process can have more open files during its lifetime, but only these first three have a special name and preassigned roles.
Now that we have some idea of a pipeline, are familiar with the pipe() system call, and know about the standard streams in a newly-created process, we can implement a pipeline. However before that, you should be comfortable with the following system calls:
fork() manpage, What is the purpose of fork()?wait() manpage, Why do we need a wait() system call?dup2() manpage, explanation of the dup() system callexeclp()) manpage, What is the difference between the functions of the exec family of system calls like exec and execve?I've linked to both the manpages and relevant SO questions with some good answers. If you aren't familiar with these system calls, you're encouraged to familiarize yourself with them before moving forward. Alternatively, you can read onward, fill the gaps by reading the above links, and then come back for a second pass.
NOTE: On my machine, I ran
dmesgand got:dmesg: read kernel buffer failed: Operation not permittedThus moving forward, I'll use a more down to earth command that most Unix users are familiar with, i.e.,
ls. Instead ofdmesg | less, I'll usels | less. It's the same principle, just a different command.
A lot goes on in the shell but in summary the shell works as follows:
ls | less.|, which tells it the set of commands is a pipeline.fork() to create a new child process to run the command. In our case, it creates two child processes: one for ls and one for less.| metacharacter and it determines that the command to the left (i.e., ls) of | needs its output to be directed to the input of the command to the right (i.e., less). To make this communication possible, it uses a pipe created with pipe().exec() to run the command. Here, we'll be calling execlp to execute both ls and less.wait(). When the child completes, the shell returns from wait() and prints out a prompt again, ready for your next command. Here it'd wait for both ls and shell.We'll implement what happens at step 3 and onward. In the following program,
// ls-to-less-pipe.c
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/wait.h>
int main(int argc, char *argv[]) {
// the parent is forking two processes, and we store their process ID
// in these variables.
pid_t ls_pid, less_pid;
// we set up a pipe that will enable communication between the processes
// for `ls` and `less`.
int pipefd[2];
pipe(pipefd);
// the parent process creates the first child process for the `ls` command.
ls_pid = fork();
if (ls_pid == 0) {
// we assign this process's output to the pipe's write end, i.e.,
// instead of sending output to the screen, it sends it to the pipe's
// write end.
dup2(pipefd[1], STDOUT_FILENO);
// now this process's stdout refers to the pipe's write end too so we
// can close this descriptor.
close(pipefd[1]);
// this process doesn't use the pipe's read end, and thus we close this
// file descriptor.
close(pipefd[0]);
// replace process's current image with this new process image, i.e.,
// the ls command.
if (execlp("ls", "ls", (char *) NULL) < 0) {
fprintf(stderr, "failed trying to execute the ls command");
exit(0);
};
}
else if (ls_pid < 0) {
fprintf(stderr, "failed forking ls process");
}
// the parent process creates the first child process for the `less` command.
less_pid = fork();
if (less_pid == 0) {
// we assign this process's input to the pipe's read end, i.e., instead
// of taking input from the keyboard, it takes it from the pipe's read end.
dup2(pipefd[0], STDIN_FILENO);
// now this process's stdin refers to the pipe's read end too so we
// can close this descriptor.
close(pipefd[0]);
// this process doesn't use the pipe's write end, and thus we close this
// file descriptor.
close(pipefd[1]);
// replace process's current image with this new process image, i.e., the less command.
if (execlp("less", "less", (char *) NULL) < 0) {
fprintf(stderr, "failed trying to execute the less command");
exit(0);
};
}
else if (less_pid < 0) {
fprintf(stderr, "failed forking less process");
exit(0);
}
// the parent process doesn't use the pipe so we close both ends. Also
// needed to send EOF so the children can continue (children blocks until
// all input has been processed).
close(pipefd[0]);
close(pipefd[1]);
// the parent process waits for both child processes to finish their execution.
int ls_status, less_status;
pid_t ls_wpid = waitpid(ls_pid, &ls_status, 0);
pid_t less_wpid = waitpid(less_pid, &less_status, 0);
return 0;
}
Compiling and running it:
$ gcc ls-to-less-pipe.c && ./a.out
file1.txt
file2.txt
file3.txt
:
Therefore we've indeed set up a pipe between ls and less, allowing ls to send its output to less, which is akin, albeit not in the same way, to what the shell does when you run ls | less.
Let's say we've a pipeline for printing the top three authors based on number of commits in a git repo:
git log --format='%an' | sort | uniq -c | sort -nr | head -n 3
This can be implemented as follows:
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/wait.h>
int main(int argc, char *argv[]) {
pid_t git_pid, sort_pid1, sort_pid2, uniq_pid, head_pid;
int pipefd1[2];
int pipefd2[2];
int pipefd3[2];
int pipefd4[2];
pipe(pipefd1);
pipe(pipefd2);
pipe(pipefd3);
pipe(pipefd4);
if ((git_pid = fork()) == 0) {
dup2(pipefd1[1], STDOUT_FILENO);
close(pipefd1[1]);
close(pipefd1[0]);
if (execlp("git", "git", "log", "--format=%an", (char *) NULL) < 0) {
fprintf(stderr, "failed trying to execute the git command");
exit(0);
};
}
else if (git_pid < 0) {
fprintf(stderr, "failed forking git process");
}
if ((sort_pid1 = fork()) == 0) {
dup2(pipefd1[0], STDIN_FILENO);
dup2(pipefd2[1], STDOUT_FILENO);
close(pipefd1[0]);
close(pipefd1[1]);
close(pipefd2[1]);
close(pipefd2[0]);
if (execlp("sort", "sort", (char *) NULL) < 0) {
fprintf(stderr, "failed trying to execute the sort command");
exit(0);
};
}
else if (sort_pid1 < 0) {
fprintf(stderr, "failed forking sort process");
exit(0);
}
if ((uniq_pid = fork()) == 0) {
dup2(pipefd2[0], STDIN_FILENO);
dup2(pipefd3[1], STDOUT_FILENO);
close(pipefd1[0]);
close(pipefd1[1]);
close(pipefd2[0]);
close(pipefd2[1]);
close(pipefd3[0]);
close(pipefd3[1]);
if (execlp("uniq", "uniq", "-c", (char *) NULL) < 0) {
fprintf(stderr, "failed trying to execute the uniq command");
exit(0);
};
}
else if (sort_pid1 < 0) {
fprintf(stderr, "failed forking uniq process");
exit(0);
}
if ((sort_pid2 = fork()) == 0) {
dup2(pipefd3[0], STDIN_FILENO);
dup2(pipefd4[1], STDOUT_FILENO);
close(pipefd1[0]);
close(pipefd1[1]);
close(pipefd2[0]);
close(pipefd2[1]);
close(pipefd3[0]);
close(pipefd3[1]);
close(pipefd4[0]);
close(pipefd4[1]);
if (execlp("sort", "sort", "-nr", (char *) NULL) < 0) {
fprintf(stderr, "failed trying to execute the sort command");
exit(0);
};
}
else if (sort_pid2 < 0) {
fprintf(stderr, "failed forking sort process");
exit(0);
}
if ((head_pid = fork()) == 0) {
dup2(pipefd4[0], STDIN_FILENO);
close(pipefd1[0]);
close(pipefd1[1]);
close(pipefd2[0]);
close(pipefd2[1]);
close(pipefd3[0]);
close(pipefd3[1]);
close(pipefd4[0]);
close(pipefd4[1]);
if (execlp("head", "head", "-n", "3", (char *) NULL) < 0) {
fprintf(stderr, "failed trying to execute the head command");
exit(0);
};
}
else if (head_pid < 0) {
fprintf(stderr, "failed forking head process");
exit(0);
}
close(pipefd1[0]);
close(pipefd1[1]);
close(pipefd2[0]);
close(pipefd2[1]);
close(pipefd3[0]);
close(pipefd3[1]);
close(pipefd4[0]);
close(pipefd4[1]);
int git_status, sort1_status, sort2_status, uniq_status, head_status;
pid_t git_wpid = waitpid(git_pid, &git_status, 0);
pid_t sort1_wpid = waitpid(sort_pid1, &sort1_status, 0);
pid_t sort2_wpid = waitpid(sort_pid2, &sort2_status, 0);
pid_t uniq_wpid = waitpid(uniq_pid, &uniq_status, 0);
pid_t head_wpid = waitpid(head_pid, &head_status, 0);
return 0;
}
Pictorially this looks as follows:
If by this question, you mean that "a pipeline is a sequence of one or more commands separated by one of the control operators | or |&. The output of each command in the pipeline is connected via a pipe to the input of the next command. That is, each command reads the previous command’s output.", then yes.
© 2022 - 2024 — McMap. All rights reserved.
ps | cat. – Incommodious