Disassembling self-modifying code

Question

I've been just wondering - how does one disassemble self-modifying binary? I guess you can't use olly or IDA because they are static disassemblers, correct? What happens with disassembler if you jump into middle of an instruction? And how does one analyze metamorphic engines?

Limiter

One does not wish to criticize your question too harshly. Maybe I just don't understand it! However, it may not be the sort of question one can answer in a few paragraphs. – Postimpressionism

Answer 1

Both OllyDbg and IDA are not only static analyzers, they can both also run the code. IDA can also run your code remotely and as far as I know, it can even do embedded debugging. Of course if you "break" the program execution at some point, and take a look at the disassembly, it will (in both programs) reflect the current state of the program, including any modifications it has made on itself.

What happens with disassembler if you jump into middle of an instruction?

According to my experience, both the mentioned disassembler can handle this situation. For example, see here, how OllyDbg manages it, here is a screenshot when EIP is 00892C0E:

And when I make EIP = 00892C0F, which is in a middle of an instruction:

as you see, it simply re-disassembles the instruction, making it a different (but still valid) opcode.

And how does one analyze metamorphic engines?

Just as any other code. The tricks you are mentioning (jumping into a middle of a instruction, modifying itself) were mainly popular a while ago, when the disassemblers and debugger weren't as smart as they are now.

Of course static analysis can be very hard, but you can of course still analyze binaries completely offline and decipher the "morphing" (in your mind) to get the idea of what the code will do. But when you can use a debugger live, then you simply see what the code is up to.

Of course, all of this is an endless race between people writing the code and people analyzing it. Who wins depends on who gives up sooner.

Tightwad

Just wanted to quote this, because this is so true :) "Of course, all of this is an endless race between people writing the code and people analyzing it. Who wins depends on who gives up sooner." – Turgite

Answer 2

Ollydbg will handle such situations.

The functionality you are looking for is 'Analyze'. It will re-disassemble the new instructions at the location and refresh the CPU window with updated code.

Assume that instruction at address X will make some self-modifying changes. After you execute that instruction, you can run the 'Analyze' command. This is re-disassemble code around that basicblock and refresh the CPU with the updated instructions.

To Analyze, right click at the desired location in CPU Window and click on Analysis> Analyze Code. The shortcut is Ctrl-A.

Inartificial