MASM Fixing 64 bit Truncation in a DLL

Question

I am working with the Adobe Flash ocx by loading it into my C++ program. The ocx is supposed to be 64 bit but for some reason it has issues when I compile with the x64 platform. I have read up on this and found that it is likely that some function receives DWORD userData instead of void* userData through some structure and then casts it to an object pointer. This works ok in a 32-bit environment, but crashes in 64-bit.

The disassembly of the function calls inside the ocx that cause the crash are the following lines:

mov         ecx,r8d 

The first operation copies only low 32-bits from R8D to ECX (ECX is 32-bit).

cmp         dword ptr [rcx+11BCh],0 

The second operation accesses 64-bit register, where low 32-bits contains correct address and high 32-bits contains some junk. Leading to a crash, of course.

Solution

I have read that one possible solution is to do the following:

1. Create an asm file containing the following code:

   nop 
   nop 
   nop 
   
   mov ecx,r8d 
   cmp dword ptr [rcx+11BCh],0 
   
   nop 
   nop 
   nop 
   
   mov rcx,r8d   // I've replaced ecx with rcx here 
   cmp dword ptr [rcx+11BCh],0 
   

2. Build an obj file using this asm file and MASM.exe

3. Open the obj file with a hex editor and locate the 90's that represent the nop
4. In the Flash ocx locate the first string of bytes between the nops and replace it with the new string of bytes that comes after the nops. This will change it from 32 bit to 64 bit function calls.

Problem

I have attempted this by making the following asm file and building it with ml64.exe (I do not have masm.exe but I think that ml.exe is the new 32 bit version of it, and this code would only build with the ml64.exe, probably because of the 64-bit only operators?):

TITLE: Print String Assembly Program (test.asm)

.Code
main Proc
nop 
nop 
nop 

mov ecx,r8d 
cmp dword ptr [rcx+11BCh],0 

nop 
nop 
nop 

mov rcx,r8   
cmp dword ptr [rcx+11BCh],0 

main ENDP
END

I had trouble getting it to build (I kept getting errors about instruction length matching) until I changed r8d to r8 in the second section.

I got this obj to build, and opened it with a hex editor and was able to locate the two byte strings. But where my problem comes is that when I search for the first byte string that should be in the flash ocx, I cannot find it. It is not there, so I cannot replace it with the second one.

What am I doing wrong?

Thanks!

Scrubland

You might be the smartest person in the Flash community here. heh – Triable

ha I wish.. I actually wish I knew even more about assembly and could get this thing to work. I'm almost positive I know what the problem is, just can't seem to fix it! – Scrubland

Are you saying that you solved your problem by manually editing the Flash OCX file? – Hotpress

Hey, I am on the Flash Player team. And while we do not officially support use of the ocx outside IE this looks like a serious bug and I would really like to know which function you were patching so I can fix it properly. I am a bit confused if this is a general ocx problem or a problem inside flash though. – Engel

I'm using a bunch of callbacks to transfer data to and from the actionscript in the flash. I'll have to do some debugging to see what exact function is the culprit. It was hard for me to see it before because on the call stack was just flash.ocx!some numberes() so I knew the problem was in the 64-bit ocx. I'll let you know once I find out more information (I've got a busy week so it might take a few days). Thanks for looking into it. – Scrubland

@Engel It looks to me from his final answer like he was using the wrong ocx. Your guess is as good as mine on his solution on his "changing a 41 to a 49" though... – Sorority

Definitely wasn't using the wrong ocx. It was the newest version of the flash 64 bit ocx. I had to edit the binary of the ocx itself to change a function within it by changing the binary. Which function exactly that is I am not 100% sure yet since debugging that is quite difficult since I only have access to the binary of the ocx not the actual source. Changing the 41 to a 49 was basically just changing the ecx to rcx to support 64 bit pointers. – Scrubland

Please add the solution as an answer below. That way you can close this question out later. – Tael

Answer 1

1. Create an asm file containing the following code:

   nop 
   nop 
   nop 
   
   mov ecx,r8d 
   cmp dword ptr [rcx+11BCh],0 
   
   nop 
   nop 
   nop 
   
   mov rcx,r8d   // I've replaced ecx with rcx here 
   cmp dword ptr [rcx+11BCh],0 
   

2. Build an obj file using this asm file and MASM.exe

3. Open the obj file with a hex editor and locate the 90's that represent the nop
4. In the Flash ocx locate the first string of bytes between the nops and replace it with the new string of bytes that comes after the nops. This will change it from 32 bit to 64 bit function calls.

I made the following asm file and built it with ml64.exe

TITLE: Print String Assembly Program (test.asm)

.Code
main Proc
nop 
nop 
nop 

mov ecx,r8d 
cmp dword ptr [rcx+11BCh],0 

nop 
nop 
nop 

mov rcx,r8   
cmp dword ptr [rcx+11BCh],0 

main ENDP
END

I got this obj to build, and opened it with a hex editor and was able to locate the two byte strings. I found the first byte string in the Flash OCX and changed it to the second one. (The only actual change was a 41 to a 49 in the strings)

Scrubland