FreeConsole behaviour on Windows 8

Question

On Windows 8, we have an issue with FreeConsole. It seems to close the stdio handles, without shutting the file streams.

This may be a Windows 8 problem, or it could be that I simply don't understand the (totally absurd) way the Windows console/GUI app subsystem does things.

What's going on?

Minimal example below. Tested with compilers: VS2005, VS2013, VS2017, using statically linked CRT.

#include <windows.h>
#include <io.h>
#include <stdio.h>

static void testHandle(FILE* file) {
  HANDLE h = (HANDLE)_get_osfhandle(fileno(file));
  DWORD flags;
  if (!GetHandleInformation(h, &flags)) {
    MessageBoxA(0, "Bogus handle!!", "TITLE", MB_OK);
  }
}

int main(int argc, char** argv)
{
  freopen("NUL", "wb", stdout); // Demonstrate the issue with NUL
  // Leave stderr as it is, to demonstrate the issue with handles
  // to the console device.

  FreeConsole();

  testHandle(stdout);
  testHandle(stderr);
}

Pola

Note: If you run this code on Windows 7, there's no MessageBox. Run it on Windows 8, there's a message box. – Pola

I reported this to Microsoft given its security implications. It nearly caused an extremely dangerous bug in our application. I'd still like to know if there's any explanation or comment SO has. – Pola

your link reporting this to Microsoft is broken now - did you receive any response from Microsoft? – Brim

Good question, I can't find it anywhere on their website now. Not only is the link broken, it's no longer listed in my Microsoft Connect dashboard under "feedback you've submitted". Did they just delete my bug report!? I think their feedback was "invalid, you're relying on undefined/undocumented behaviour". To which my response was, "you're joking, how can it be that the interaction between FreeConsole and stdout isn't documented". – Pola

Answer 1

After disassembling the code for FreeConsole on different Windows versions, I worked out the cause of the problem.

FreeConsole is a remarkably unsubtle function! I really does close a load of handles for you, even if it doesn't "own" those handles (eg HANDLEs owned by stdio functions).

And, the behaviour is different in Windows 7 and 8, and changed again in 10.

Here's the dilemma when coming up with a fix:

• Once stdio has a HANDLE to the console device, there is no documented way to get it to give up that handle, without a call to CloseHandle. You can call close(1) or freopen(stdout) or whatever you like, but if there is an open file descriptor that refers to the console, CloseHandle will be called on it, if you want to switch stdout to a new NUL handle after FreeConsole.
• On the other hand, since Windows 10 there's also no way to avoid FreeConsole calling CloseHandle as well.
• Visual Studio's debugger and the Application Verifier flag the app for calling CloseHandle on an invalid HANDLE. And, they're right, it's really not good.
• So, if you try and "fix up" stdio before the call to FreeConsole then FreeConsole will do an invalid CloseHandle (using its cached handle, and there's no way whatsoever to tell it that handle's gone - FreeConsole no longer checks GetStdHandle(STD_OUTPUT_HANDLE)). And, if you call FreeConsole first, there's no way to fix up the stdio objects without causing them to do an invalid call to CloseHandle.

By elimination, I conclude that the only solution is to use an undocumented function, if the public ones just won't work.

// The undocumented bit!
extern "C" int __cdecl _free_osfhnd(int const fh);
static HANDLE closeFdButNotHandle(int fd) {
  HANDLE h = (HANDLE)_get_osfhandle(fd);
  _free_osfhnd(fd); // Prevent CloseHandle happening in close()
  close(fd);
  return h;
}

static bool valid(HANDLE h) {
  SetLastError(0);
  return GetFileType(h) != FILE_TYPE_UNKNOWN || GetLastError() == 0;
}

static void openNull(int fd, DWORD flags) {
  int newFd;
  // Yet another Microsoft bug! (I've reported four in this code...)
  // They have confirmed a bug in dup2 in Visual Studio 2013, fixed
  // in Visual Studio 2017.  If dup2 is called with fd == newFd, the
  // CRT lock is corrupted, hence the check here before calling dup2.
  if (!_tsopen_s(&newFd, _T("NUL"), flags, _SH_DENYNO, 0) &&
      fd != newFd)
    dup2(newFd, fd);
  if (fd != newFd) close(newFd);
}

void doFreeConsole() {
  // stderr, stdin are similar - left to the reader.  You probably
  // also want to add code (as we have) to detect when the handle
  // is FILE_TYPE_DISK/FILE_TYPE_PIPE and leave the stdio FILE
  // alone if it's actually pointing to disk/pipe.
  HANDLE stdoutHandle = closeFdButNotHandle(fileno(stdout)); 

  FreeConsole(); // error checking left to the reader

  // If FreeConsole *didn't* close the handle then do so now.
  // Has a race condition, but all of this code does so hey.
  if (valid(stdoutHandle)) CloseHandle(stdoutHandle);

  openNull(stdoutRestore, _O_BINARY | _O_RDONLY);
}

Pola

I wonder if you do freopen first (which as you note calls ::CloseHandle(), leaving dangling handles) whether it isn't sufficient to call ::SetStdHandle() to stop ::FreeConsole() from trying to close the dangling (and possibly reused) HANDLE? – Batho

Oh I see you covered that in your final bullet point. – Batho

Altogether it seems to be a bug in the CRT stdio implementation, which should not borrow the standard handle (which could be closed at any time by OS APIs) but should call DuplicateHandle to get its own handles to store in CRT FILE objects, and for which the CRT has total control over the lifetime. – Batho

Answer 2

issue caused by fact that prior Windows 8 standard (not redirected) console handles (that returned by GetStdHandle) where actually pseudohandles, wich values doesnt instersect with other kernel object handles, so writing to that pseudohandle after it being 'closed' by FreeConsole always fails. In Win8 MS changed something inside, so GetStdHandle returns normal kernel object handle that refers to console subsystem driver object (actually that driver also appeared only in Win8). So FreeConsole closes that handle. The most funny thing that CRT does GetStdHandle on startup and saves returned value somewhere inside and uses wherever use called C functions that access std::in/out/err. Since FreeConsole closed that handle, and its not a special pseudohandle value anymore - same handle value can be reused by any other opened kernel object handle, and you will be lucky if it will not be file, pipe, or socket cause in this case all your debug outpout will go there:)

Saltpeter

Indeed, the really awful thing is that FreeConsole closes handles to NUL, which used to be OK. That is, it used to be the case that FreeConsole would leave stdio in an unrecoverable state unless you sent it NUL, but now even that doesn't work. I hope from a security POV they fix it! – Pola