Ollydbg 1.10 "Back to user mode" doesn't work

Question

I tried to learn "Lena's reversing for newbies", when some trouble arise. I start Pixtopian Book with ollyDbg, then try to have MessageBox with message about uregistered version. Then i switch to OllyDbg, stop program executing and press "Alt+F9" for "Back to user mode" which stop the program after it exit from DLL.

But after this program does not work, it's frozen and does not respond to my actions. If i turn off "Back to user mode" program normally work.

What's the problem? Can i try to use "Back to user mode" in IDA (uses WinDbg) or some other debugger and How i can do this? Can i repair it function in OllyDbg?

P.S. It's like the program stopped and didn't run after use "ALT+F9".

/Sorry for my English, i'm just learning ;-)/

Perianth

Go to the modules window and mark the appropriate dlls as system dll. – Outshout

@ExtremeCoders, sorry i didn't understand. Please give me link or more information – Perianth

See this question – Outshout

@ExtremeCoders Thanks a lot! I was able to localize my problem, but it has some different type. My threads wasn't suspend. They are just paused and i can't resume this! Later i opened ProcessExplorer and found out, that all threads, except Main program in paused mode. Main window was in WaitUserRequest mode. I try to start other threads and program unfreezed. But OllyDbg couldn't set breakpoint and caught caller. Whats next? – Perianth

Ok, let me try out the tutorial and I will inform you of the proceedings. – Outshout

I just tried the tutorial but in my case Execute till User Code was working fine. Let me see in which cases this cannot work. – Outshout

You may try the newest version of Ollydbg and see if the problem persists and I hope that you are running this on a 32 bit platform – Outshout

@ExtremeCoders i had tried OllyDbg 1.10 and 2.3. Similar situations. But i noticed, that some programs can run, after Back to user mode, and some program can't. I don't know why. But it is very interesting trouble for me. I want to resolve it, of course with your help. – Perianth

You may try Immunity Debugger – Outshout

@ExtremeCoders i can't. It didn't run. – Perianth

Immunity Debugger requires python to run. It is listed in the system requirements. – Outshout

Another question, can you specify the platform where you are debugging, i.e. whether it is a Virtual Machine or a real machine. I am asking this because I found one VM in which you can experience this problem. – Outshout

@ExtremeCoders, just now I tried all this things on VirtualBox Windows XP SP3, and it's normally worked. On prime sistem - Windows 7 x64 Ultimate it's doesn't work. I think, that it may be related with Windows 7 threads management, but i don't know how. P.S. I have python on my Windows 7. I will try Immunity one more time. – Perianth

Ollydbg does not supports x64 at this moment. There will be errors if you try to run even with compatibility mode. For the time being you can either continue in VirtualBox with a 32 bit OS installed or use Windbg in a x64 system. However I use VMWare for debugging as last time I found out the VirtualBox does not supports hardware breakpoints. The situation is same with Parallels Workstation, there you would get a BSOD if you try to single step through FPU instructions. – Outshout

And BTW all of Lena 151 tutorials are on the 32 bit platform. So it's better if you continue working in a x32 environment. – Outshout

@ExtremeCoders, Thanks! It's wonderful! All of your comments were very useful for me! Can you create ANSWER for this question, so that i could take up your rating and note, that my question was completely answered. – Perianth

Glad to know your problem was solved :) – Outshout

Answer 1

First of all, Ollydbg is meant for 32 bit platform. It means that it will run only on a 32 bit OS and can only debug 32 bit apps.

In x64 Operating Systems (specifically Windows) there is a feature called compatibility mode that lets you run 32 bit apps. The 32 bit code is either run by emulation or natively (when the hardware itself implements the instruction set and then it is called x86-64).

So, when you try to run Ollydbg in a x64 environment it will run, but you will experience problems like the one you are facing. It occurs because Ollydbg is run in an emulation mode. Being a debugger it needs access to the registers and other system structures, which it is denied. What it can see is a virtual image of the system.

So the solution to the problem is using a Virtual Machine.

You would install a 32 bit OS in it and debug the app using Ollydbg. As far as Virtual Machines are concerned, I would recommend VMWare. You can use either the workstation or player version. The latter is free but does not support snapshots.

Other solutions are Virtual Box, Parallels Workstation and Microsoft Virtual PC. The disadvantages of them are that Virtual Box does not support hardware breakpoints, Parallels Workstation is no longer supported as of 2014 and moreover there you would get a BSOD if you try to single step through FPU instructions. I have not tested Virtual PC though.

Note : Ollydbg does not supports x64 but its author is working on a x64 version.

Outshout

Answer 2

I just did a thorough learning of the pixtopian file. When I downloaded that tutorial I get the file pixtopian107.exe only. Since it didn't agree with the tutorial I investigated further. That file is an installation file. It produces a regular pixtopian.exe file which is the file you wish to play with. I also noticed that in running the file it never enters the main module. this is because of the TLS callback. right now I am trying to learn how to overcome this which is how i came to this site in the first place. I am using ollydbg vs2.01e very effectively.

Another thing, in vs 2.01e I am still trying to save changed data. Until I succeed I am recording the changes in the exe files using hex editor.

Festoonery

Answer 3

I just learned how to update changes to the exe file for ollydby v 2.01e. suppose I wished to change a jl command to a jmp; do this by clicking the executable modules button, choose the file and right click to view the file. then record the changes and save file. The saved file also has a backup in case something goes wrong.

Festoonery

Answer 4

From my experience I can tell that this functionality won't work on Windows 7.

On Windows 7 64bit --> Won't work at all. On Windows 7 32bit --> Will work partially, but only when using option "File>Attach" in OllyDbg.

For me, best solution was to use Windows XP 32bit, then it worked fine.

Sadonia