I've recently been working on detouring functions (only in Linux) and so far I've had great success. I was developing my own detouring class until I found this. I modernized the code a bit and converted it to C++ (as a class of course). That code is just like any other detour implementation, it replaces the original function address with a JMP to my own specified 'hook' function. It also creates a 'trampoline' for the original function.
Everything works flawlessly but I'd like to do one simple adjustement. I program in pure C++, I use no global functions and everything is enclosed in classes (just like Java/C#). The problem is that this detouring method breaks my pattern. The 'hook' function needs to be a static/non-class function.
What I want to do is to implement support for _thiscall hooks (which should be pretty simple with the GCC _thiscall convention). I've had no success modifying this code to work with _thiscall hooks. What I want as an end result is something just as simple as this; PatchAddress(void * target, void * hook, void * class);. I'm not asking anyone to do this for me, but I would like to know how to solve/approach my problem?
From what I know, I should only need to increase the 'patch' size (i.e it's now 5 bytes, and I should require an additional 5 bytes?), and then before I use the JMP call (to my hook function), I push my 'this' pointer to the stack (which should be as if I called it as a member function). To illustrate:
push 'my class pointer'
jmp <my hook function>
Instead of just having the 'jmp' call directly/only. Is that the correct approach or is there something else beneath that needs to be taken into account (note: I do not care about support for VC++ _thiscall)?
NOTE: here's is my implementation of the above mentioned code: header : source, uses libudis86