Define byte appearing in debug after a manually encoded far call

Question

I couldn't get MASM to accept a far call instruction written as call 0f000h:1260h, probably because of the issues brought up in this question.

Instead of messing around with cryptic MASM directives, I decided to manually encode it into my program using DB like so:

pushf                      ;IRET will be executed, push flags.
db 9ah,60h,12h,0,0f0h      ;call location f000:1260.
                           ;Location pointed to by int 1c (System timer tick)
                           ;BIOS defaults it to a dummy IRET

While tracing through the program with DEBUG.COM, I noticed that "DB FE" appears after the execution of the call instruction. However, this doesn't occur when executing int 1ch. What's the difference between these two methods of jumping to location f000:1260?

I assumed that DEBUG wasn't recognizing 0xfe (along with the following bytes) as a valid opcode. I dumped location f000:1260 in order to see which bytes were there.

Byte 0xfe is indeed present, along with some other bytes. I know that 0xcf by itself is the opcode for IRET (which is all that I was expecting to find), so what are these other bytes?

Here's the IVT entry for int 1ch at location 0000:0070.

UPDATE

As Michael Petch stated in his answer, the strange bytes make up a callback mechanism into DOSBox. I was curious to see what would happen if I tried to execute this callback in my main program.

Executing:

xor ah, ah                 ;select set video mode function
mov al, 13h                ;320x200 256 colors
db 0feh,38h,18h,00h        ;set video mode DOSBox callback.
                           ;Nothing pushed to stack.

Seems to be exactly the same as executing:

xor ah, ah                 ;select set video mode function
mov al, 13h                ;320x200 256 colors
int 10h                    ;set video mode.
                           ;Three registers pushed, FLAGS altered (by INT)
                           ;callback occurs, registers popped (by IRET)

The only difference being that int pushes FLAGS, CS, and IP, as well as clears IF and TF. The program is returned to an IRET (at location f000:1264) which undoes all of that.

"DB FE" still shows up in DEBUG. I think the callback is only triggered by the combination of 0xfe and 0x38. It first tries to execute 0xfe, which in this case is not part of a valid opcode and does nothing (0xfe is part of inc opcode when followed by valid bytes), then upon encountering the following 0x38 the callback occurs.

Heavensent

can you make sure the interrupt vector table contains F000:1260 target? Tracing into int calls with SW bugger like debug.com may have its own pitfalls. Also is this clean DOS on real HW, or some virtual machine? From the screenshot I guess this is some VM, or simulator. Then the code you see at that address may be completely bogus, just some "shadow" over what the simulator is truly doing in background with the int call... But so far your explanation in question sounds lot more reasonable than these ideas, I'm taking it a bit too far probably, so check first IVT. – Patriapatriarch

It's DOSBox. So you're saying that executing int 1ch doesn't actually cause a jump to this location? I added an image of the the IVT entry for int 1ch, location 0000:0070. – Heavensent

So, in the end it's about dosbox being just simulator with focus on performance and run-ability of applications, not with focus on fully faithful machine simulation, thus it is taking shortcuts in some places with extra internal implementation of stuff... Interesting, makes sense to me immediately, but I never thought about it before, nor hit such issue myself. – Patriapatriarch

No, 0xfe won't trigger it by itself. 0xfe is also used for encoding INC r/m8.0xfe 0x38 though is not a valid instruction (it doesn't encode a valid inc instruction either). So the emulator will have to check at least the next byte after 0xfe to determine what to do. – Scottscotti

Answer 1

0xFE 0x38 is not a defined prefix and/or instruction on a real Intel x86 processor. In DOSBox the sequence 0xFE 0x38 is a special 4 byte instruction.The remaining two byte make up a 16-bit value that act as a DOSBox callback index number. In this case the index is 0x0013.

What this effectively does is calls across into DOSBox itself to perform the requested task. DOSBox will do whatever processing it needs, sets the registers inside the emulator and then returns. The next instruction is the IRET (0xCF). That will end the interrupt and continue processing instructions prior to the interrupt call.

I discovered this while looking at the DOSBox code. In particular the function CALLBACK_SetupExtra:

case CB_IRET:
    if (use_cb) {
        phys_writeb(physAddress+0x00,(Bit8u)0xFE);  //GRP 4
        phys_writeb(physAddress+0x01,(Bit8u)0x38);  //Extra Callback instruction
        phys_writew(physAddress+0x02,(Bit16u)callback);     //The immediate word
        physAddress+=4;
    }
    phys_writeb(physAddress+0x00,(Bit8u)0xCF);      //An IRET Instruction

This code sets up DOSBox callbacks where an IRET is required after the callback.

Scottscotti

well, technically FE is inc [mem] variant, it's the second 38 which makes it undefined, while for example fe 05 11 00 ... encodes inc byte [0x11]. So FE is instruction INC on x86. – Patriapatriarch

@Patriapatriarch Only fe /0 is inc on x86 though. fe /1 is dec r/m8 and the others are undefined. You can't just ignore the extended opcode. – Gamber

@Gamber ok, that's more accurate... either way, after just FE you can't yet tell if the opcode is invalid (what was the original wording of answer before Michael edited it). Anyway, it's just minor detail, the answer was good enough already before the edit. – Patriapatriarch