Delegate on instance method passed by P/Invoke

Question

To my surprise I have discovered a powerful feature today. Since it looks too good to be true I want to make sure that it is not just working due to some weird coincidence.

I have always thought that when my p/invoke (to c/c++ library) call expects a (callback) function pointer, I would have to pass a delegate on a static c# function. For example in the following I would always reference a delegate of KINSysFn to a static function of that signature.

[UnmanagedFunctionPointer(CallingConvention.Cdecl)]
public delegate int KINSysFn(IntPtr uu, IntPtr fval, IntPtr user_data );

and call my P/Invoke with this delegate argument:

[DllImport("some.dll", EntryPoint = "KINInit", ExactSpelling = true, CallingConvention = CallingConvention.Cdecl)]
public static extern int KINInit(IntPtr kinmem, KINSysFn func, IntPtr tmpl);

But now I just tried and passed a delegate on an instance method and it worked also! For example:

public class MySystemFunctor
{
    double c = 3.0;
    public int SystemFunction(IntPtr u, IntPtr v, IntPtr userData) {}
}

// ...

var myFunctor = new MySystemFunctor();
KINInit(kinmem, myFunctor.SystemFunction, IntPtr.Zero);

Of course, I understand that inside managed code there is no technical problem at all with packaging a "this" object together with an instance method to form the respective delegate.

But what surprises me about it, is the fact that the "this" object of MySystemFunctor.SystemFunction finds its way also to the native dll, which only accepts a static function and does not incorporate any facility for a "this" object or packaging it together with the function.

Does this mean that any such delegate is translated (marshalled?) individually to a static function where the reference to the respective "this" object is somehow hard coded inside the function definition? How else could be distinguished between the different delegate instances, for example if I have

var myFunctor01 = new MySystemFunctor();
// ...
var myFunctor99 = new MySystemFunctor();

KINInit(kinmem, myFunctor01.SystemFunction, IntPtr.Zero);
// ...
KINInit(kinmem, myFunctor99.SystemFunction, IntPtr.Zero);

These can't all point to the same function. And what if I create an indefinite number of MySystemFunctor objects dynamically? Is every such delegate "unrolled"/compiled to its own static function definition at runtime?

Hochman

I think you know the answer to the question. Your code proves it. – Morrie

@DavidHeffernan: You mean the question about how it is implemented internally? I have just stated a hypothesis and it would be incredibly nice to know if it is true or if there is something I haven't thought about. – Hochman

Answer 1

Does this mean that any such delegate is translated (marshalled?) individually to a static function...

Yes, you guessed at this correctly. Not exactly a "static function", there is a mountain of code inside the CLR that performs this magic. It auto-generates machine code for a thunk that adapts the call from native code to managed code. The native code gets a function pointer to that thunk. The argument values may have to be converted, a standard pinvoke marshaller duty. And always shuffled around to match the call to the managed method. Digging up the stored delegate's Target property to provide this is part of that. And it jiggers the stack frame, tying a link to the previous managed frame, so the GC can see that it again needs to look for object roots.

There is however one nasty little detail that gets just about everybody in trouble. These thunks are automatically cleaned-up again when the callback is not necessary anymore. The CLR gets no help from the native code to determine this, it happens when the delegate object gets garbage-collected. Maybe you smell the rat, what determines in your program when that happens?

 var myFunctor = new MySystemFunctor();

That is a local variable of a method. It is not going to survive for very long, the next collection will destroy it. Bad news if the native code keeps making callbacks through the thunk, it won't be around anymore and that's a hard crash. Not so easy to see when you are experimenting with code since it takes a while.

You have to ensure this can't happen. Storing the delegate objects in your class might work, but then you have to make sure your class object survives long enough. Whatever it takes, no guess from the snippet. It tends to solve itself when you also ensure that you unregister these callbacks again since that requires storing the object reference for use later. You can also store them in a static variable or use GCHandle.Alloc(), but that of course loses the benefit of having an instance callback in a hurry. Feel good about having this done correctly by testing it, call GC.Collect() in the caller.

Worth noting is that you did it right by new-ing the delegate explicitly. C# syntax sugar does not require that, makes it harder to get this right. If the callbacks only occur while you make the pinvoke call into the native code, not uncommon (like EnumWindows), then you don't have to worry about it since the pinvoke marshaller ensures the delegate object stays referenced.

Pignut

In my case the call to KINInit just stores the callback. Afterwards a call to another routine KINSolve is invoked, and guess what, it assumes that the callback still exists. So what I draw from your remarks is, it is all safe as long as I keep a reference to mySystemFunctor between the calls to KINInit and KINSolve, right? – Hochman

Edited to make you feel better. Store them in fields of your class and you'll be okay when KINSolve() is an instance method. Albeit that this does not sound like a cleanup method that ensures there are no further callbacks. Think of a possible KINTerminate() or Dispose() method. – Pignut

But just storing delegate instance in a field might not be enough, because instance can be collected (together with delegate) right during the method which calls KINSolve (assuming it does not use that delegate or other instance fields). – Bid

Is there any guarantee that a storing the delegate in an instance field will help? Even if you keep the containing class alive, theoretically the JIT could detect that the field is not being read from outside the function that stores it, and demote it to a local variable. Which indicates a necessity for a finalizer with GC.KeepAlive in it. I know the current JITter doesn't do this, but it could, right? – Endrin

Answer 2

For the records: I have walked right into the trap, Hans Passant has mentioned. Forced garbage collection has led to a null reference exception because the delegate was transient:

KINInit(kinmem, myFunctor.SystemFunction, IntPtr.Zero);
// BTW: same with:
// KINInit(kinmem, new KINSysFn(myFunctor.SystemFunction), IntPtr.Zero);

GC.Collect();
GC.WaitForPendingFinalizers();

KINSol(/*...*); // BAAM! NullReferenceException

Luckily I had already wrapped the critical two P/Invokes, KINInit (which sets the callback delegate) and KINSolve (which actually uses the callback) into a dedicated managed class. The solution was, as already discussed, to keep the delegate referenced by a class member:

// ksf is a class member of delegate type KINSysFn that keeps ref to delegate instance
ksf = new KINSysFn(myFunctor.SystemFunction); 
KINInit(kinmem, ksf, IntPtr.Zero);

GC.Collect();
GC.WaitForPendingFinalizers();

KINSol(/*...*);

Thanks again, Hans, I'd never have noticed this flaw because it works as long as no GC happens!

Hochman

My guess would be downvote for not answering the question. – Morrie