Confused about stdin, stdout and stderr?

Question

I am rather confused with the purpose of these three files. If my understanding is correct, stdin is the file in which a program writes into its requests to run a task in the process, stdout is the file into which the kernel writes its output and the process requesting it accesses the information from, and stderr is the file into which all the exceptions are entered. On opening these files to check whether these actually do occur, I found nothing seem to suggest so!

What I would want to know is what exactly is the purpose of these files, absolutely dumbed down answer with very little tech jargon!

Bey

Answer 1

Standard input|STDIN - this is the file handle that your process reads to get information from you.

Standard output|STDOUT - your process writes conventional output to this file handle.

Standard error|STDERR - your process writes diagnostic output to this file handle.

That's about as dumbed-down as I can make it :-)

Of course, that's mostly by convention. There's nothing stopping you from writing your diagnostic information to standard output if you wish. You can even close the three file handles totally and open your own files for I/O.

When your process starts, it should already have these handles open and it can just read from and/or write to them.

By default, they're probably connected to your terminal device (e.g., /dev/tty) but shells will allow you to set up connections between these handles and specific files and/or devices (or even pipelines to other processes) before your process starts (some of the manipulations possible are rather clever).

An example being:

my_prog <inputfile 2>errorfile | grep XYZ

which will:

• create a process for my_prog.
• open inputfile as your standard input (file handle 0).
• open errorfile as your standard error (file handle 2).
• create another process for grep.
• attach the standard output of my_prog to the standard input of grep.

---

Re your comment:

When I open these files in /dev folder, how come I never get to see the output of a process running?

It's because they're not normal files. While UNIX presents everything as a file in a file system somewhere, that doesn't make it so at the lowest levels. Most files in the /dev hierarchy are either character or block devices, effectively a device driver. They don't have a size but they do have a major and minor device number.

When you open them, you're connected to the device driver rather than a physical file, and the device driver is smart enough to know that separate processes should be handled separately.

The same is true for the Linux /proc filesystem. Those aren't real files, just tightly controlled gateways to kernel information.

Kibitka

Thats for your response. While I can understand the purpose of the files from what you describe, I would like to move a level more. when I open these files in /dev folder, how come I never get to see the output of a process running. Say I execute top on the terminal, is it not supposed to output its results onto the stdout file periodically, hence when it is being updated I should be able to see an instance of the output being printed onto this file. But this is not so.. So are these files not the same (the ones in the /dev directory). – Bey

Because those aren't technically files. They're device nodes, indicating a specific device to write to. UNIX may present everything to you as a file abstraction but that doesn't make it so at the deepest levels. – Kibitka

Oh okay, that makes a lot more sense now! So incase I want to access the data stream that a process is writing onto these "files" which I am not able to access. How would it be possible for me to intercept it and output it to a file of my own? Is it possible theoretically to log whatever occurs on each one of these files onto a single for a given process? – Bey

Use the shell redirection capability. xyz >xyz.out will write your standard output to a physical file which can be read by other processes. xyz | grep something will connect xyz stdout to grep stdin more directly. If you want unfettered access to a proces you don't control that way, you'll need to look into something like /proc or write code to filter the output by hooking into the kernel somehow. There may be other solutions but they're all probably as dangerous as each other :-) – Kibitka

So essentially what you are telling me is that every process created its own? Thanks that helps a lot. Hope its not getting annoying. :-) – Bey

@Shouvik, note that /dev/stdin is a symlink to /proc/self/fd/0 -- the first file descriptor that the currently running program has open. So, what is pointed to by /dev/stdin will change from program to program, because /proc/self/ always points to the 'currently running program'. (Whichever program is doing the open call.) /dev/stdin and friends were put there to make setuid shell scripts safer, and let you pass the filename /dev/stdin to programs that only work with files, but you want to control more interactively. (Someday this will be a useful trick for you to know. :) – Pork

@Pork @oax thanks for all the help, I guess most of my inquisitiveness is satisfied for now! As an when I run into trouble, I know where to find the standard stream gurus now! :) – Bey

About the original answer. What's the difference between file and file handle? – Nougat

@CarlosW.Mercado, a file is a physical manifestation of the data. For example, the bits stored on the hard disk. A file handle is (usually) a small token used to refer to that file, once you have opened it. – Kibitka

More dumbed down: stdout gets >>redirected into file. stderr prints in the command line. – Wilhoit

I always thought these file descriptors were opened by stblib.h – Pirri

Note that stderr is intended to be used for diagnostic messages in general. Actually, only the normal output should be written to stdout. I sometimes see people splitting the log in stdout and stderr. This is not the intended way of useage. The whole log should go into stderr altogether. – Sitting

The comment of @Sitting is very important and highlights that this answer is actually partly wrong. stderr is clearly for diagnostic messages and not errors : jstorimer.com/blogs/workingwithcode/…. Please fix the answer. – Hoekstra

@Oswin, have changed it to use the terms (conventional and diagnostic) from the ISO standard, since that's the definitive source. – Kibitka

This doesnt work it won't take input from inputfile – Excite

@Phillip, suggest you post a question with more detail so we can figure out what's wrong, since it very much does get input from the file. – Kibitka

Answer 2

It would be more correct to say that stdin, stdout, and stderr are "I/O streams" rather than files. As you've noticed, these entities do not live in the filesystem. But the Unix philosophy, as far as I/O is concerned, is "everything is a file". In practice, that really means that you can use the same library functions and interfaces (printf, scanf, read, write, select, etc.) without worrying about whether the I/O stream is connected to a keyboard, a disk file, a socket, a pipe, or some other I/O abstraction.

Most programs need to read input, write output, and log errors, so stdin, stdout, and stderr are predefined for you, as a programming convenience. This is only a convention, and is not enforced by the operating system.

Fulks

Thanks for your inputs. Would you happen to know how I could intercept the output data stream of a process and output it into a file of my own? – Bey

@Bey you can use a "pipe" for that. If we take the program echo as an example: echo 'foo' > target.txt. Normally echo sends its output to stdout, here we've redirected that output to target.txt. – Ectomy

Is correct the stream term and not file, it according with - en.wikipedia.org/wiki/Standard_streams – Musa

Answer 3

As a complement of the answers above, here is a sum up about Redirections:

EDIT: This graphic is not entirely correct.

The first example does not use stdin at all, it's passing "hello" as an argument to the echo command.

The graphic also says 2>&1 has the same effect as &> however

ls Documents ABC > dirlist 2>&1
#does not give the same output as 
ls Documents ABC > dirlist &>

This is because &> requires a file to redirect to, and 2>&1 is simply sending stderr into stdout

Binocular

Your comment combining with the accepted answer makes perfect sense and clearly explains things! Thanks! – Whoreson

A picture is worth a thousand words ! – Selfdiscipline

&> requires file to redirect in to, but 2>&1 doesn't &> is for logging, but 2>&1 can be used for logging and terminal STDOUT STDERR view at the same time 2>&1 can be used just to view the STDOUT and STDERR (in the terminal's command_prompt) of your program (depending on your program's ability to handle errors) – Devotee

Actually, that first one is just plain wrong. In no way does "hello" interact with the standard input of echo. In fact, if you replaced echo with a program that read standard input, it would just sit there waiting for your terminal input. In this case, hello is provided as an argument (through argc/argv). And the comment about 2>&1 and &> having the same effect is accurate if you see the effect as is stated: "merges stderr with stdout". They both do that but in a slightly different way. The equivalence would be > somefile 2>&1 and &> somefile. – Kibitka

You would be better off rewriting the graphic as text so you could remove that first one easily :-) – Kibitka

Answer 4

I'm afraid your understanding is completely backwards. :)

Think of "standard in", "standard out", and "standard error" from the program's perspective, not from the kernel's perspective.

When a program needs to print output, it normally prints to "standard out". A program typically prints output to standard out with printf, which prints ONLY to standard out.

When a program needs to print error information (not necessarily exceptions, those are a programming-language construct, imposed at a much higher level), it normally prints to "standard error". It normally does so with fprintf, which accepts a file stream to use when printing. The file stream could be any file opened for writing: standard out, standard error, or any other file that has been opened with fopen or fdopen.

"standard in" is used when the file needs to read input, using fread or fgets, or getchar.

Any of these files can be easily redirected from the shell, like this:

cat /etc/passwd > /tmp/out     # redirect cat's standard out to /tmp/foo
cat /nonexistant 2> /tmp/err   # redirect cat's standard error to /tmp/error
cat < /etc/passwd              # redirect cat's standard input to /etc/passwd

Or, the whole enchilada:

cat < /etc/passwd > /tmp/out 2> /tmp/err

There are two important caveats: First, "standard in", "standard out", and "standard error" are just a convention. They are a very strong convention, but it's all just an agreement that it is very nice to be able to run programs like this: grep echo /etc/services | awk '{print $2;}' | sort and have the standard outputs of each program hooked into the standard input of the next program in the pipeline.

Second, I've given the standard ISO C functions for working with file streams (FILE * objects) -- at the kernel level, it is all file descriptors (int references to the file table) and much lower-level operations like read and write, which do not do the happy buffering of the ISO C functions. I figured to keep it simple and use the easier functions, but I thought all the same you should know the alternatives. :)

Pork

So is it when the process is being executed that it writes errors onto this stderr file or when the program is being compiled from its source. Also when we talk about these files from the perspective of compiler is it different than when it is compared with say a program? – Bey

@Shouvik, the compiler is just another program, with its own stdin, stdout, and stderr. When the compiler needs to write warnings or errors, it will write them to stderr. When the compiler front-end outputs intermediate code for the assembler, it might write the intermediate code on stdout and the assembler might accept its input on stdin, but all that would be behind the scenes from your perspective as a user.) Once you have a compiled program, that program can write errors to its standard error as well, but it has nothing to do with being compiled. – Pork

Thanks for that token of information. I guess it's pretty stupid of me not to see it in that perspective anyway... :P – Bey

So you are saying that standard helps us to print the program – Dockery

Answer 5

I think people saying stderr should be used only for error messages is misleading.

It should also be used for informative messages that are meant for the user running the command and not for any potential downstream consumers of the data (i.e. if you run a shell pipe chaining several commands you do not want informative messages like "getting item 30 of 42424" to appear on stdout as they will confuse the consumer, but you might still want the user to see them.

See this for historical rationale:

"All programs placed diagnostics on the standard output. This had always caused trouble when the output was redirected into a file, but became intolerable when the output was sent to an unsuspecting process. Nevertheless, unwilling to violate the simplicity of the standard-input-standard-output model, people tolerated this state of affairs through v6. Shortly thereafter Dennis Ritchie cut the Gordian knot by introducing the standard error file. That was not quite enough. With pipelines diagnostics could come from any of several programs running simultaneously. Diagnostics needed to identify themselves."

Gyrostabilizer

The historical rationale link is broken - the domain is gone! – Microwatt

Replaced with archive.org link. – Gyrostabilizer

Answer 6

stdin

Reads input through the console (e.g. Keyboard input). Used in C with scanf

scanf(<formatstring>,<pointer to storage> ...);

stdout

Produces output to the console. Used in C with printf

printf(<string>, <values to print> ...);

stderr

Produces 'error' output to the console. Used in C with fprintf

fprintf(stderr, <string>, <values to print> ...);

---

Redirection

The source for stdin can be redirected. For example, instead of coming from keyboard input, it can come from a file (echo < file.txt ), or another program ( ps | grep <userid>).

The destinations for stdout, stderr can also be redirected. For example stdout can be redirected to a file: ls . > ls-output.txt, in this case the output is written to the file ls-output.txt. Stderr can be redirected with 2>.

Continuative

just mentioning that echo < file.txt does not work, as mentioned above, echo accepts the strings as argument, does not read stdin – Merits

Answer 7

Using ps -aux reveals current processes, all of which are listed in /proc/ as /proc/(pid)/, by calling cat /proc/(pid)/fd/0 it prints anything that is found in the standard output of that process I think. So perhaps,

/proc/(pid)/fd/0 - Standard Output File
/proc/(pid)/fd/1 - Standard Input File
/proc/(pid)/fd/2 - Standard Error File

for example

But only worked this well for /bin/bash other processes generally had nothing in 0 but many had errors written in 2

Eductive

Answer 8

For authoritative information about these files, check out the man pages, run the command on your terminal.

$ man stdout 

But for a simple answer, each file is for:

stdout for a stream out

stdin for a stream input

stderr for printing errors or log messages.

Each unix program has each one of those streams.

Oddson

Answer 9

stderr will not do IO Cache buffering so if our application need to print critical message info (some errors ,exceptions) to console or to file use it where as use stdout to print general log info as it use IO Cache buffering there is a chance that before writing our messages to file application may close ,leaving debugging complex

Kurr

Answer 10

Here is a lengthy article on stdin, stdout and stderr:

• What Are stdin, stdout, and stderr on Linux?

To summarize:

Streams Are Handled Like Files

Streams in Linux—like almost everything else—are treated as though they were files. You can read text from a file, and you can write text into a file. Both of these actions involve a stream of data. So the concept of handling a stream of data as a file isn’t that much of a stretch.

Each file associated with a process is allocated a unique number to identify it. This is known as the file descriptor. Whenever an action is required to be performed on a file, the file descriptor is used to identify the file.

These values are always used for stdin, stdout, and stderr:

0: stdin
1: stdout
2: stderr

Ironically I found this question on stack overflow and the article above because I was searching for information on abnormal / non-standard streams. So my search continues.

Cestus

Answer 11

A file with associated buffering is called a stream and is declared to be a pointer to a defined type FILE. The fopen() function creates certain descriptive data for a stream and returns a pointer to designate the stream in all further transactions. Normally there are three open streams with constant pointers declared in the header and associated with the standard open files. At program startup three streams are predefined and need not be opened explicitly: standard input (for reading conventional input), standard output (for writing conventional output), and standard error (for writing diagnostic output). When opened the standard error stream is not fully buffered; the standard input and standard output streams are fully buffered if and only if the stream can be determined not to refer to an interactive device

https://www.mkssoftware.com/docs/man5/stdio.5.asp

Disastrous