How many memory pages do C compilers on desktop OSes use to detect stack overflows?

Question

This question is related to but different from this one about variable length arrays in C99.

The answers point out that one danger with allocating variable length arrays (or just large arrays of a fixed size) in the stack is that the allocation may fail silently, as opposed to, say, calling malloc, which explicitly tells the caller whether allocation succeeded.

Modern non-embedded compilation platforms use an invalid memory zone to detect some stack overflows at no additional cost (the checks are only the checks already made for free by the MMU). This doesn't protect at 100% from the above problem because a very large local array may cause the stack pointer to jump over the invalid area.

Does any one know how many pages are typically allocated for this detection? I guess it would be at least 4KiB, but it could be more. Is that a choice made by the compiler or the OS, and in either case, is there a way to change it?

Damalis

Good question, but one should remark that malloc does not explict tell on all systems whether allocation succeded. It can still fail afterwards on first access (at least on some systems - see optimistic allocation or memory overcommitment). – Matteson

@Matteson I almost remarked about that, but then I decided to omit it because memory overcommit is really an OS issue, not a language one. As such it doesn't cause undefined behavior, it causes the allocating process, or another process, to be killed cleanly. – Damalis

Also see Qualys Security Advisory - The Stack Clash on OSS-Security mailing list. Its shows off some neat tricks, and its pretty damning of the guard page. Its amazing how many OS'es they took down with it. – Linda

Answer 1

On Windows, it's one 4KB page (at least on x86): See Description of the stack checking for Windows NT-based applications.

This automatic growth method uses a guard page, a reserved, uncommitted, memory page that is contiguous with the committed portion of memory. When the application touches the guard page, the operating system commits that page and the next uncommitted page becomes the new guard page. Automatic stack growth works only for the guard page and stack memory must grow in 4K, or one page, increments. If the application touches another reserved but uncommitted page of stack memory before it touches the guard page, a normal page fault exception occurs and unpredictable behavior can result.

...

To prevent the fault, the compiler calls the __chkstk() function each time the local allocation exceeds 4K. The Windows NT __chkstk() function does not explicitly check for stack overflow as the MS-DOS version does. It simply touches memory addresses every 4K from the current stack pointer location to the requested allocation. This triggers the guard pages in the proper sequence and commits additional memory to the stack as required.

For GCC, GCC Stack checking

I'm not sure how/if C99's VLA's would change the WinNT behaviour.

Manageable

Answer 2

I'm pretty sure the most common practice is using just one page, usually 4k. A good compiler, however, will sequentially attempt to access each page of a stack frame larger than the page size on function entry (or on VLA/alloca allocation) to ensure that a guard page is hit. GCC can optionally do this; see: http://gcc.gnu.org/onlinedocs/gcc/Code-Gen-Options.html#Code-Gen-Options and the -fstack-check option.

Bagger