https://www.hellboundhackers.org/articles/read-article.php?article_id=729
Just as a quick summary, I wrote this article for two purposes. First, It is interesting, and more knowledge of how your computer works is always helpful. Secondly, there are always programs where flags will be manipulated directly, and it is helpful to know the effects they will have on jumps. For example, something as simple as
CMP eax,ebx
JC somewhere
might confuse most beginning reversers, but hopefully not after this article. Enjoy :)
[Important Note: I will use 8 bit integers for my examples when I write out binary numbers. Just remember that, although 8 bit integers are not commonly used in programming, the same rules that I discuss apply to integers with more bits]
The CMP Instruction:
The CMP instruction operates by performing an implied subtraction of the two operands. This means that the result is not stored in memory. After subtracting them, it does a few quick tests, updating the Z,O,C,S, and P flags. The P, or parity, flag is rarely used, so we'll ignore it in this article for the purpose of brevity.
Binary subtraction is performed by adding the negated version of the second operand from the first. This is just like what you learned in middle school, about how 4+3 = 4 - (-3), and visa versa.
At the end of the article I will explain how this is done, but I'll move onto the more important matters for now since that knowledge is not really needed for cracking or coding.
Sign and Zero Flag:
The four flags that the CMP instruction can set - Z,O,C, and S, are known as the zero, overflow, carry, and sign flags respectively. The zero flag is set whenever the result of the subtraction is equal to zero. This, of course, only occurs when the operands are equal. The sign flag is set when the result of the subtraction is negative. Although we are inclined to think that this means the sign flag in combination with the zero flag are enough to test all > >= < and <=, this is not true, because the result can be negative even if the first number is greater than the second. This is because of overflow.
Overflow Flag:
Signed integers are represented in binary with the same amount of bits as unsigned integers. This means, of course, that the sign must be set in one of the bits of the integer. Signed integers store the sign in the MSB (most significant bit). This means that, while 00000001 converts to 1 in decimal, 10000001 converts to -127. I will discuss why it is -127 and not -1 or -2 later in the article.
When the processor performs subtraction, It wraps around if the subtraction goes below 00000000 or above 11111111. Therefore, if you subtract a negative number from a positive one, or subtract a positive number from a negative one, there is the possibility that the answer will overflow over the boundary. For example, 100 - (-100) is equal to 200, but the highest value an 8 bit signed integer can be is 127, so 200 will wrap through the upper boundary and end up as a negative number, even though it should be positive. The same problem occurs with -100 - 100; It wraps through the low end and ends up positive when it should be negative, causing an underflow. Note that an underflow also sets the overflow flag, and overflow will refer to both overflows and underflows further in the article. The CPU checks for this, and sets the overflow flag if it occurs.
Carry Flag:
The carry flag is set when, if both operands are interpreted as unsigned integers, the first one is greater. This is easy to determine because it occurs whenever the subtraction passes through 00000000 into the higher range (11111111).
For example, 00000001 - 00000010 = 11111111, so carry is set. However, 00000010 - 00000001 = 00000001, so carry is not set.
