CreateProcess with new console window, but override some std i/o handles

Question

If you use CreateProcess with the flag CREATE_NEW_CONSOLE, the new process has its standard input, output, and error handles directed to the new console window. If you want to override the I/O streams, you can do so by setting the handles in STARTUPINFO fields hStdOutput, hStdInput, and hStdError and setting the flag STARTF_USESTDHANDLES.

But what if you want to override only one of the handles? For example, I might want to redirect stderr to a file while leaving the stdout and stdin connected to the new console window.

The STARTF_USESTDHANDLES flag tells CreateProcess to replace all of the handles instead of connecting them to the ones for the new console window. So it seems we must provide all three handles. Obviously I can set hStdError to the handle of the log file, but what values should be used for hStdInput and hStdOutput?

I tried using NULL, which seems to work on Windows 8.1, but it doesn't work on Windows 7.

I also thought about creating a console window first, and then calling CreateProcess with handles to the new console window's buffers (and omitting the CREATE_NEW_CONSOLE flag). Unfortunately, the parent process is also a console application, and it seems a console application cannot create a second console window.

Dinorahdinosaur

One workaround, if it turns out that the posted answers don't work: instead of launching program.exe launch cmd.exe /c program.exe >CON <CON – Hoxie

Answer 1

According to this MSDN Support article:

How to spawn console processes with redirected standard handles

If the parent process only wishes to redirect one or two standard handles, specifying GetStdHandle() for the specific handles causes the child to create the standard handle as it normally would without redirection. For example, if the parent process only needs to redirect the standard output and error of the child process, then the hStdInput member of the STARTUPINFO structure is filled as follows:

hStdInput = GetStdHandle(STD_INPUT_HANDLE);

According to the GetStdHandle() documentation:

STD_INPUT_HANDLE
(DWORD)-10
The standard input device. Initially, this is the console input buffer, CONIN$.

STD_OUTPUT_HANDLE
(DWORD)-11
The standard output device. Initially, this is the active console screen buffer, CONOUT$.

STD_ERROR_HANDLE
(DWORD)-12
The standard error device. Initially, this is the active console screen buffer, CONOUT$.

...

The standard handles of a process may be redirected by a call to SetStdHandle, in which case GetStdHandle returns the redirected handle. If the standard handles have been redirected, you can specify the CONIN$ value in a call to the CreateFile function to get a handle to a console's input buffer. Similarly, you can specify the CONOUT$ value to get a handle to a console's active screen buffer.

Attach/detach behavior

When attaching to a new console, standard handles are always replaced with console handles unless STARTF_USESTDHANDLES was specified during process creation.

If the existing value of the standard handle is NULL, or the existing value of the standard handle looks like a console pseudohandle, the handle is replaced with a console handle.

When a parent uses both CREATE_NEW_CONSOLE and STARTF_USESTDHANDLES to create a console process, standard handles will not be replaced unless the existing value of the standard handle is NULL or a console pseudohandle.

So, if the parent process's STDIN has NOT been redirected, GetStdHandle(STD_INPUT_HANDLE) will return either NULL or a pseudo-handle that refers to CONIN$. When that value is passed to the child process via STARTUPINFO, the child process will receive a console handle for the STDIN of whichever console it happens to be running in. On the other hand, if the parent process's STDIN has been redirected, GetStdHandle(STD_INPUT_HANDLE) will return an actual handle to a real file/pipe/etc, which the child will inherit and access.

The same applies to STDOUT and STDERR.

So, if you want to redirect the child's STDIN/OUT/ERR handles, you have to set hStdInput/Output/Error to your own handles. If you want the child to receive default handles, use GetStdHandle() and let CreateProcess() decide what kind of handles the child receives based on whether the parent is itself being redirected or not.

Shred

If you create a new console for the child, the child will use the input/output of that console by default. If the parent wants the child to inherit the parent's input, the parent has to redirect the child's input. The parent can detect if its own input was redirected and then pass that along to the child if desired. The child cannot decide that. – Shred

GetStdHandle(STD_INPUT_HANDLE) returns a handle to the calling process's current input. If the input has NOT been redirected, the handle will refer to CONIN$ - the standard input of the current console. If a child process receives that handle for its input, it will read from the standard input of whichever console it happens to be running in. On the other hand, if the parent's input HAS been redirected, the handle will be an actual handle to a real file/pipe/etc which the child can inherit and read from. Same goes for output (the default output handle referring to CONOUT$) – Shred

This is covered in more detail in the Remarks section of the GetStdHandle() documentation. – Shred

"explicitly opening CONIN$ works if and only if your standard input is redirected" - that is not true. I just did a test where a console process without redirected input was able to open CONIN$ without error. And I canot find anything on MSDN that says being able to open CONIN$ is conditional upon STDIN being redirected first. – Shred

I think you should add that bit from the Remarks section of GetStdHandle() to your answer as it more directly addresses the question. – Parrakeet

Let's think about this. If the parent's STDIN is not redirected and you open the parent's CONIN$ and pass it to the child, then you have two processes reading from the same input of the same console. How would that work if you use CREATE_NEW_CONSOLE? When the user types into the parent console, who gets the input? The parent? The child? Both? When using CREATE_NEW_CONSOLE, I think it makes sense that the child should only be able to use CONIN$ of its own console, or redirected input from a file/pipe/etc. – Shred

Comments deleted. See my answer; the observed behaviour is rather odd. – Hoxie

Answer 2

On Windows 7, at least, the documentation here (as quoted by the existing answers) is misleading. The recommended approach only works if the parent process does not have redirected input.

This is the actual behaviour, as observed on my machine:

• When a process opens a handle to CONOUT$ (for example) the returned handle always has the same numeric value (on my machine, the handle for CONOUT$ is always 7) unless that handle already exists.

• So if you are a console process, and you were not launched with redirected output, your standard output handle is 7. If you open another handle to CONOUT$, it will have a different value. If you close handle 7, and then open a handle to CONOUT$, you'll get 7 again.

• When a console process is launched, a handle to CONOUT$ with the magic value of 7 is usually present in the child process (regardless of redirection). If you have used STARTF_USESTDHANDLES, then the child's standard output will go to the console if and only if you specify 7 as the standard output handle. If you specify a handle to CONOUT$ that has any other value, it won't work.

• Sometimes when a console process is launched with standard output redirected, there is no handle to CONOUT$ present in the child process. In particular, this happens when cmd.exe launches a console process with standard output redirected. So far, I have been unable to figure out what combination of parameters to use with CreateProcess to make this happen.

• The behaviour of CONIN$ is analogous; on my machine, the magic value for CONIN$ is 3. (I haven't investigated how standard error works.)

The upshot is that I do not believe it is safe to make use of this behaviour unless you have complete control over the way the parent process was launched, because unless the standard handles already point to the console there is no reliable way of obtaining a handle to the console with the correct magic value.

Instead, launch a proxy process with CREATE_NEW_CONSOLE and without STARTF_USESTDHANDLES. From that process, since you know how it was launched, you know the standard handles will have the correct magic values, so it is safe to specify them as the handles for a child process.

Hoxie

On my Win7 machine, calling CreateFile("CONOUT$", ...) in a console process without redirected output does not return 7. As a test, I called CreateFile() twice and always got 0x00000017 and 0x0x0000001B, respectively. GetStdHandle(STD_OUTPUT_HANDLE) returns 0x00000007 without redirection. That matches with Microsoft's recommendation in the "How to spawn console processes with redirected standard handles" article, and with your comment that passing 7 with STARTF_USESTDHANDLES makes the child process use its own console's output. – Shred

Prior to Windows 8, console buffer handles are created by the console host process, conhost.exe (or csrss.exe prior to Win7), which maintains a per-process handle table. (The lower 2 bits are always set, which tags them for the base API to redirect calls to corresponding LPC-enabled functions such as DuplicateConsoleHandle.) Since it's a new console window, it should be safe to assume conhost.exe will initialize the handle table to 3 for the input buffer and 7 and 11 for the output and error. – Shipowner

Prior to Windows 8, opening the pseudo devices CON, CONIN$, and CONOUT$ redirects to the undocumented function OpenConsoleW. This uses LPC to call conhost!SrvOpenConsole in the console host process, which calls conhost!AllocateIoHandle to allocate a new handle for a console buffer. – Shipowner

Windows 8 uses the console device, \Device\ConDrv. This provides virtual files such as Connect, Input, Output, and ScreenBuffer. Additionly the Win32 device namespace has symbolic links such as \Global??\CONIN$ => \Device\ConDrv\CurrentIn. Handles for console files are created by the object manager, like any other kernel objects. This allows the console API to be implemented in terms of the I/O system services NtReadFile, NtWriteFile, and NtDeviceIoControlFile instead of using LPC. – Shipowner

For creating a new console with partially redirected standard handles, it works to use the Win7-style handles 3, 7, and 11 in Windows 8. Actually, using any handle value with the lower 2 bits set (i.e. the old console handle tag) or NULL (as noted by the OP and in the docs) also works in Windows 8. Just to be clear though, console buffer handles in Windows 8 do not have the lower 2 bits set, and they're present in the process handle table. They get created by the object manager like any other handle, unlike in Windows 7 for which they're created by the console host process. – Shipowner

@Remy: yes, that matches my observations. I think you'll find that if you close the standard output handle first, then opening CONOUT$ will return 7. – Hoxie

@eryksun: well, that explains why it all works so much better on Windows 8! What a mess. Oh, well, I guess it's not all that long until Windows 7 drops out of support ... :-) – Hoxie

Here's another Windows 7 console gripe. The Windows 8 device driver allows writing any sized buffer to the console, whereas in Windows 7 the LPC-based implementation limits the buffer size to be no bigger than the largest block in the heap of shared memory (64K at best). LPC uses shared memory to pass large messages between processes. This system wasn't designed with I/O in mind. It was designed to efficiently communicate realtively small messages over LPC ports (not shared memory) to and from user-mode subsystems such as csrss.exe and lsass.exe. – Shipowner

@ErykSun on my windows 7, redirection to a file only works if cmd /c doesn't open a new conhost.exe, so using cmd /c consoleapp.exe > out.txt or start /b cmd /c consoleapp.exe > out.txt works but start cmd /c consoleapp.exe > out.txt or admin /c consoleapp.exe > out.txt does not. – Trabzon

Answer 3

If i interpreted this documentation correctly, you should use GetStdHandle(STD_INPUT_HANDLE) and GetStdHandle(STD_OUTPUT_HANDLE) for the other two handles.

Rajah