A x64-DLL injected into a x64-process hooking a x86-DLL fails using C++ and EasyHook

Question

A x64-DLL injected into a x64-process hooking a x86-DLL fails using C++ and EasyHook. It works if Loader, InjectionLibrary and InjectionTarget(it's available in both versions and i need both to be hooked) are x86. Getting the address of the exported procedure(GetProcAddress itself) isn't a problem at x64. The InjectionTarget has HookTarget(Kernel32.dll) as a dependency at x64 aswell. LhInstallHook(...) returns STATUS_NOT_SUPPORTED where the source comments say that happens when: "The target entry point contains unsupported instructions."

Due to the fact that the source is fine for x86 builds i've decided to not add it.

I've scratched a little diagram

Kalpa

Can a x86 DLL even be mapped in a x64 process ? This is new to me. – Building

I checked dependencywalker and it's listed, not my territory though. Anyway I need to reach the target of being able to hook GetProcAddress or similiar on x64. – Kalpa

Answer 1

You cannot use a 32-bit DLL in a 64bit process, and indeed, this generalizes- you cannot mix and match x86 and x64 code, a single process is either entirely x64, or entirely x86. That's fundamental to x86-64 and there's nothing you can do about it. In the Windows control console, they make a 64bit process and a 32bit process and use IPC to control the 32bit process to load and deal with all the 32bit shell extensions. You could try something similar, if you're brave and/or desperate.

Edit: Wait a minute, wait a minute. Could you describe this process a little more at basics when everything works fine in x86 mode? Like, X loads a function from Y, I am doing Z, because it appears that I don't understand what you're doing.

You have an injection target, and an injected DLL, from which you're trying to call procedures in the third-party x86-only DLL. So the normal flow of call goes from InjectionTarget -> InjectionLibrary -> HookTarget -> ExportedProcedure. And this isn't working for you because HookTarget is x86 only and you can't change that, so when you recompile InjectionLibrary for x64 for the x64 version of InjectionTarget, it doesn't work anymore because your x64 InjectionLibrary is trying to load an x86 HookTarget.

The only solution to this problem is to create an x86 process and use inter-process communication to get it to call the procedures in HookTarget that you want called. If you can't re-compile HookTarget for x64, then this is the only way to do it.

Emanative

I definitely need to hook exported procedures of a x86-DLL. Could you give me a hint for a step in the right direction? IPC seems too general. – Kalpa

@zEh: Why not just spawn an x86 process? If you're targetting an x64 exe, then it can't load an x86 DLL any more than you can- it'll have to be x86 too. – Emanative

You mean the InjectionLibrary which is injected into the InjectionTarget spawning a x86-process? How would I intercept the GetProcAddress calls of InjectionTarget then? – Kalpa

@zEh: If InjectionTarget is x86, then spawn an x86 process and do a normal hook. If InjectionTarget is x64, then spawn an x64 process and do a normal hook. Either way, all you should have to do is match the InjectionTarget's architecture. If it loads a function from HookTarget.dll, then it has to come pre-matching HookTarget.dll's architecture. – Emanative

@DeadMG: The x64 target is definitely using exports of a bunch of x86 dlls which I have to hook. I still don't see a solution for that. – Kalpa

@zEh: The target cannot use an x86 DLL when running as x64- that's impossible, just like it's impossible for you. – Emanative

@DeadMG: I finally noticed the filesize of the x64-TargetProcess is actually so big that it matches the filesizes of the x86-TargetProcess and the x86-DLLs. Until now I had to believe it's relying on the x86-DLLs. Thanks for your help and patience. – Kalpa