how do procedure calls work in assembler?

Question

I just started tinkering with ASM and I'm not sure if my understanding of procedure calls is correct.

say at some point in the code there is a procedure call

call dword ptr[123]

and the procedure consists of only one command, ret:

ret 0004

what would be the effect of this procedure call, and where would the return value be stored? I read somewhere that a return value of 2 bytes would be stored in AX, but when I replace the procedure call by

mov AX, 0004

(together with the necessary NOPs) the program crashes.

Riggins

Answer 1

in x86 assembler the parameter to the ret instruction means:

RET immediate

Return to calling procedure and pop immediate bytes from the stack.

(quoting from Intel® 64 and IA-32 Architectures Software Developer's Manuals Vol 2B)

So when you type:

ret 0004

You're telling the CPU to return to the instruction immediately after the call, and to pop 4 bytes off the stack. This is great if you pushed 4 bytes onto the stack before the call.

push eax
call dword ptr[123]

Note that this has nothing to do with the return value. In fact, a procedure in Assembly has no way of specifying that a value is a return value. This is all done by convention. Most compilers of which I am aware will use EAX to hold the return value, but this is true only because the calling function will expect the result there.

So your calling code would be:

call dword ptr [123]
mov dword ptr [result], eax

and your function that returns the value 4 would be:

mov eax, 4
ret

Emmaline

thanks! most of the tutorials I googled merely used ret, and didn't explain the ret <i>immediate</i> operation. I got further confused when the LLVM assembler apparently had a "ret <type> <value>" command. – Riggins

is there any reason why RET may return to some place other than the calling? I just created a question and found this in related questions – Mesdemoiselles

yes, if you manipulated the return address that CALL pushed onto the stack. – Emmaline

Answer 2

It all depends on the calling convention being used. I won't repeat the Wikipedia article here, just read the definition.

In the C calling convention, for example, the return value would be in EAX/AX/AL. Your single-instruction does not have one: It is a void function taking around 4 bytes of parameters (possibly a single int) that does nothing. As it is the callee's duty to clean up the stack in this calling convention, ignoring to do that and replacing the call with a 'mov ax' does not work.

Also I suspect you may be tinkering with 32-bit assembly while reading a 16-bit document. It's not a big problem, but you should be aware of the differences.

Diopside

Answer 3

// possibly there are arguments pushed here
...
call dword ptr[123] // push next OP code offset in the stack and jump to procedure

// procedure
...
ret 0004 // pop offset, set EIP to that offset and decrease ESP by 4

we additionally decrease the ESP if we had pushed arguments in the stack prior calling the procedure.

If there are pushed arguments, your program crashes because you don't pop them. The return offset for the current procedure will be wrong since it will get a value from one of the pushed arguments as the offset.

Wreak

Answer 4

I don't think the return value is stored in the register AX

Receiptor

@int3: It depends on the calling convention. There's nothing automatic here. The calling function and the called function have to agree on where the return value is. – Emmaline