Piping in Windows cmd.exe doesn't forward standard output until the process completes?

Question

Considering pipes in Windows command shell cmd.exe:

C:\>feed | filter

The standard output from the feeding process doesn't seem to reach the standard input of the filtering process until AFTER the feeding process runs to completion.

This type of 'buffering' can cause annoying delays in output messages for long running feeding processes (where you might want to hit 'ctrl-c' to interrupt it on early failure).

Is there a way to avoid this so that standard output from the feeding process reaches standard input on the filtering process as soon as data is available? (no buffering)

For example, the following simplified example:

feed.bat:

@echo off
echo something
sleep 3
echo something else

filter.bat:

@echo off
for /F "tokens=*" %%a in ('more') do (
    echo _%%a
)

The below command doesn't display anything until after 3 seconds (when the sleep completes):

C:\>feed | filter
_something
_something else

The desired behavior would be that '_something' is printed, followed by a 3 second delay, followed by '_something else' being printed.

Premer

What version of Windows has the SLEEP command? I use TIMEOUT (or the PING hack) – Ohg

It doesn't have anything to do with the way the command processor works, everything to do with the program itself. Which automatically switches output to buffered mode, it makes piping a lot more efficient. The underlying CRT functions are _isatty() and setvbuf(). But with the new behavior that output doesn't leave the buffer until it fills to capacity. Or the program itself flushes the buffer. Which is what is missing in that program. It is very fixable, as long as you have the source of the program. – Semiskilled

Answer 1

Pipes are asynchronous in Windows cmd.exe. They do not wait for the left side to complete before passing the info to the right. But your program does not demonstrate that for two reasons.

1) The FOR /F command does not begin iterating any rows until the command within the IN() clause completes. This is true for all FOR /F variants. The entire result of the IN() clause is buffered before any rows are iterated.

So your filter.bat couldn't possibly demonstrate the asynchronous nature of pipes.

2) The MORE command will not write partial lines - it waits until it receives a newline character before printing to stdout. (unless it reaches the end of file).

If you want to truly see the asynchronous nature of pipes, it is better to use a program that reads each character from stdin and immediately writes it back out to stdout.

---

Here is my version of FEED.BAT - it writes multiple lines with multiple pauses. It also writes three characters without linefeed with a pause after each one.

@echo off
echo something
timeout /nobreak 3 >nul
echo something else
timeout /nobreak 3 >nul
for /l %%N in (1 1 3) do (
  <nul set /p "=%%N"
  timeout /nobreak 3 >nul
)
echo(
echo Done

Here is my version of FILTER.JS - it reads one character from stdin and writes it out to stdout until it reaches the end of file.

while (!WScript.StdIn.AtEndOfStream) WScript.Stdout.Write(WScript.StdIn.Read(1));

And here is the command to test the behavior

feed | cscript //nologo filter.js

And here is the output, with <pause> inserted whenever there is a pause before more output.

something
<pause>something else
1<pause>2<pause>3<pause>
Done

---

My test above demonstrates that a pipe will immediately send any information it receives (assuming the filter is ready to receive it).

Design of the feeder and/or filter may mask the free flowing behavior. Your original test had a bottleneck in the filter in that it waited for all input before proceeding. It is also possible for the feeder to hold things up. Some programs have buffered output. The feeder may not send data until the buffer is full, or the buffer is flushed, or the stream closed.

There are a number of peculiar behaviors associated with Windows pipes. I recommend reading all the answers to Why does delayed expansion fail when inside a piped block of code? for a good overview of many non-intuitive issues.

Ohg