The "cookie" is basically nothing more than an arbitrary value.
So, the basic idea is that you write the chosen value on the stack before calling a function. Although it's probably not a very good value, let's arbitrarily chose 0x12345678 as the value.
Then it calls the function.
When the function returns, it goes back to the correct spot on the stack, and compares that value to 0x12345678. If the value has changed, this indicates that the function that was called wrote outside the area of the stack where it was allowed to write, so it (and that process in general) are deemed untrustworthy, and shut down.
In this case, instead of choosing 0x12345678, the system chooses a different value on a regular basis, such as every time the system is started. This means it's less likely to hit the correct value by accident -- it might happen to do so once, but if it's writing a specific value there, when the correct/chosen value changes, it'll end up writing the wrong value, and the problem will be detected.
It's probably also worth noting that this basic idea isn't particularly new. Just for example, back in the MS-DOS days, both Borland's and Microsoft's compilers would write some known value at the very bottom of the stack before calling main
in your program. After main
returned, they'd re-check that value. It would then print out an error message (right as the program exited) if the value didn't match what was expected.