Keep a TypedReference alive out of method block without returning it
Asked Answered
P

4

11

I want to premise that this question's purpose is checking if there's at least one way, even if through the most unsafe hack, to keep a reference to a non-blittable value type. I am aware that such a design type is comparable to a crime; I wouldn't use it in any practical case, other than learning. So please accept reading heretic unsafe code for now.

We know that a reference to a blittable type can be stored and increased this way:

unsafe class Foo
{
    void* _ptr;

    public void Fix(ref int value)
    {
        fixed (void* ptr = &value) _ptr = ptr;
    }

    public void Increment()
    {
        var pointer = (int*) _ptr;
        (*pointer)++;
    }
}

In terms of safety, the above class is comparable to a jump in the void (no pun intended), but it works, as already mentioned here. If a variable allocated on the stack is passed to it and then the caller method's scope terminates, you're likely to run into a bug or an explicit access violation error. However, if you execute a program like this:

static class Program
{
    static int _fieldValue = 42;

    public static void Main(string[] args)
    {
        var foo = new Foo();
        foo.Fix(ref _fieldValue);
        foo.Increment();
    }
}

The class won't be disposed until the relative application domain is unloaded, and so applies for the field. I honestly don't know if fields in the high-frequency heap can be reallocated but I personally think not. But let's put aside safety even more now (if even possible). After reading this and this questions I was wondering if there was a way to create a similar approach for non-blittable static types, so I made this program, which actually works. Read the comments to see what it does.

static class Program
{
    static Action _event;

    public static void Main(string[] args)
    {
        MakerefTest(ref _event);
        //The invocation list is empty again
        var isEmpty = _event == null;
    }

    static void MakerefTest(ref Action multicast)
    {
        Action handler = () => Console.WriteLine("Hello world.");
        //Assigning a handler to the delegate
        multicast += handler;
        //Executing the delegate's invocation list successfully
        if (multicast != null) multicast();
        //Encapsulating the reference in a TypedReference
        var tr = __makeref(multicast);
        //Removing the handler
        __refvalue(tr, Action) -= handler;
    }
}

The actual problem/opportunity:

We know that the compiler won't let us store a value passed by ref, but the __makeref keyword, as much undocumented and unadvised, offers the possibility of encapsulating and restoring a reference to blittable types. However, the return value of __makeref, TypedReference, is well protected. You can't store it in a field, you can't box it, you can't create an array of it, you can't use it in anonymous methods or lambdas. All that I managed to do was modifying the above code as follows:

static void* _ptr;

static void MakerefTest(ref Action multicast)
{
    Action handler = () => Console.WriteLine("Hello world.");
    multicast += handler;
    if (multicast != null) multicast();
    var tr = __makeref(multicast);
    //Storing the address of the TypedReference (which is on the stack!)
    //inside of _ptr;
    _ptr = (void*) &tr;
    //Getting the TypedReference back from the pointer:
    var restoredTr = *(TypedReference*) _ptr;
    __refvalue(restoredTr, Action) -= handler;
}

The above code works just as well and looks even worse than before but for the sake of knowledge, I wanted to do more with it, so I wrote the following:

unsafe class Horror
{
    void* _ptr;

    static void Handler()
    {
        Console.WriteLine("Hello world.");
    }

    public void Fix(ref Action action)
    {
        action += Handler;
        var tr = __makeref(action);
        _ptr = (void*) &tr;
    }

    public void Clear()
    {
        var tr = *(TypedReference*) _ptr;
        __refvalue(tr, Action) -= Handler;
    }
}

The Horror class is a combination of the Foo class and the above method, but as you'll surely notice, it has one big problem. In the method Fix, the TypedReference tr is declared, its address is copied inside of the generic pointer _ptr, then the method ends and tr no longer exists. When the Clear method is called, the "new" tr is corrupted because _ptr points to an area of the stack which is no longer a TypedReference. So here comes the question:

Is there any way to fool the compiler into keeping a TypedReference instance alive for an undetermined amount of time?

Any way to achieve the desired result will be considered good, even if it involves ugly, unsafe, slow code. A class implementing the following interface would be ideal:

interface IRefStorage<T> : IDisposable
{
    void Store(ref T value);
    //IDisposable.Dispose should release the reference
}

Please don't judge the question as generic discussion because its purpose is to see if, after all, there is a way to store references to blittable types, as wicked as it may be.

One last remark, I'm aware of the possibilities to bind fields through FieldInfo, but it seemed to me that the latter method didn't support types deriving from Delegate very much.

A possible solution (bounty result)

I would've marked AbdElRaheim's answer as chosen as soon as he edited his post to include the solution which he provided in his comment, but I suppose it wasn't very clear. Either way, among the techniques he provided, the one summed up in the following class (which I modified slightly) seemed the most "reliable" (ironic to use that term, since we're talking about exploiting a hack):

unsafe class Horror : IDisposable
{
    void* _ptr;

    static void Handler()
    {
        Console.WriteLine("Hello world.");
    }

    public void Fix(ref Action action)
    {
        action += Handler;
        TypedReference tr = __makeref(action);
        var mem = Marshal.AllocHGlobal(sizeof (TypedReference)); //magic
        var refPtr = (TypedReference*) mem.ToPointer();
        _ptr = refPtr;
        *refPtr = tr;
    }

    public void Dispose()
    {
        var tr = *(TypedReference*)_ptr;
        __refvalue(tr, Action) -= Handler;
        Marshal.FreeHGlobal((IntPtr)_ptr);
    }
}

What Fix does is, starting from the line marked as "magic" in the comment:

  1. Allocates memory in the process -- In the unmanaged part of it.
  2. Declares refPtr as a pointer to a TypedReference and sets it value to the pointer of the memory region allocated above. This is done, instead of using _ptr directly, because a field with type TypedReference* would throw an exception.
  3. Implicitly casts refPtr to void* and assigns the pointer to _ptr.
  4. Sets tr as the value pointed by refPtr and consequently _ptr.

He also offered another solution, the one he originally wrote as an answer, but it seemed less reliable than the one above. On the other hand, there was also another solution provided by Peter Wishart, but it required accurate synchronization and each Horror instance would've "wasted" a thread. I'll take the chance to repeat that the above approach is in no way intended for real world usage, it was just an academic question. I hope it will be helpful for anyone reading this question.

Palmette answered 5/1, 2013 at 2:37 Comment(2)
You are asking for a bug as a feature. Quite infamous in C and C++ programming, languages that very easily create this kind of bug. Google "dangling pointer" to learn more about it.Tesler
even he declared his intention more than twice you couldnt stop writing this, could you?Pulsate
J
1

What are you trying to do exactly? Locals are on stack and arguments are too depending on the calling convention. Storing or returning the address of a local or argument is not good because it will get overriden. There is no way to prevent them from being overriden besides like not calling methods.

If you turn on unmanaged debugging you can use the memory debugger and register window to see what is going on.

Here is simpler to understand C example. Why does print not display the correct value. Because when the print function gets invoked its stack frame overwrites the value.

int* bad(int x, int y)
{
    int sum = x + y;
    return &sum;
};

int* bad2(int x, int y)
{
    x += y;
    return &x;
}

int _tmain(int argc, _TCHAR* argv[])
{
    int* sum1 = bad(10, 10);
    int* sum2 = bad(100, 100);
    printf("%d bad", *sum1);  // prints 200 instead of 20

    sum1 = bad2(10, 10);
    sum2 = bad2(100, 100);
    printf("%d bad", *sum1);  // prints 200 instead of 20

    return 0;
};

Cant get the clr to hold on for it longer. One thing you can do is get the variable on the stack pushed further out. Below is an example. This is all bad though :(

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Linq.Expressions;
using System.Xml.Linq;
using System.Runtime.InteropServices;

namespace Bad
{
    class Program
    {
        static void Main(string[] args)
        {
            Action a = () => Console.WriteLine("test");
            Horror h = new Horror();
            h.Fix(new Big(), ref a, new Big());
            h.Clear();
            Console.WriteLine();
        }
    }
    [StructLayout(LayoutKind.Sequential, Size = 4096)]
    struct Big
    {
    }
    unsafe class Horror
    {
        void* _ptr;

        static void Handler()
        {
            Console.WriteLine("Hello world.");
        }


        public void Fix(Big big, ref Action action, Big big2)
        {
            action += Handler;
            var tr = __makeref(action);
            _ptr = (void*)&tr;
        }

        public void Clear()
        {
            var tr = *(TypedReference*)_ptr;
            __refvalue(tr, Action) -= Handler;
        }
    }
}
Jarredjarrell answered 9/1, 2013 at 4:8 Comment(8)
About why a pointer to a variable allocated on the stack may will be invalid after the call where it's pushed terminates, thanks for your explanation but that was the actual point I've used to ask the question -- Finding a way to allocate it on the heap. The method you suggested, though, is new to me and it works (even if as you said, it's really "evil"). Would you explain its trick? I will accept the answer in case better methods aren't provided. By "better" I mean a systematic way to push a TypedReference on the heap and retrieve it when needed.Palmette
sorry tr is actually a local that is getting created on the stack not an argument. You can use the static ToObject method and store the object as a field in your class. Still confused on purpose though? TypeReferences are mainly for __arglist and hacky stuff like this here #4765073Jarredjarrell
The purpose is finding out if there's a way to bypass the compiler's checks against persisting a TypedReference instance. ToObject lets me store it as an object, but I don't know how to unbox it.Palmette
toObject is going to return the value the typed reference is wrapping. You can just copy the typedreference. It should be like 4-8 bytes. public void Fix(ref Action action) { action += Handler; TypedReference tr = __makeref(action); IntPtr mem = Marshal.AllocHGlobal(sizeof(TypedReference)); // leaked TypedReference* refPtr = (TypedReference*)mem.ToPointer(); _ptr = mem.ToPointer(); *refPtr = tr; }Jarredjarrell
This is exactly what I was looking for, nice workaround. Why did you comment the result of AllocHGlobal as leaked? Is it possible to use FreeHGlobal to deallocate the unmanaged memory, given the pointer? Either way, I'll close the bounty as soon as you edit your post to add the above comment's method. Thank you for your help.Palmette
yeah you can clean up or create your own type instead to represent the typedref. They dont let you have typedreference as field but it is just two intptrs. Make a struct that has two intptrs and copy maybeJarredjarrell
So, will you edit your post or can I do it for you? I just want the information that you wrote in the comment to be easily readable for anyone who will stumble upon this question.Palmette
even it seems to be valid it will point to some random garbage on memory. because as soon as those clever Fix method returns, CLR will leave monitoring it. in short you have to create and keep it on stack. in another words it is designed that way and it needs special treatment.Pulsate
J
3

You can also achieve this without using unmanaged memory, by creating a "fake" type that resembles typed reference in its structure:

unsafe class Horror
{
    FakeTypedRef ftr;

    static void Handler()
    {
        Console.WriteLine("Hello void.");
    }

    public void Fix(ref Action action)
    {
        action += Handler;
        TypedReference tr = __makeref(action);
        ftr = *(FakeTypedRef*)(&tr);
    }

    public void Clear()
    {
        fixed(FakeTypedRef* ptr = &ftr)
        {
            var tr = *(TypedReference*)ptr;
            __refvalue(tr, Action) -= Handler;
        }
    }

    [StructLayout(LayoutKind.Sequential)]
    struct FakeTypedRef
    {
        public IntPtr Value;
        public IntPtr Type;
    }
}

Important Edit: I strongly advise against passing any reference around as a pointer. The GC is allowed to freely move objects on the managed heap as it sees fit, and there is no guarantee that the pointer will stay valid not even after it is returned from the method. You may not see the immediate effect of this due to debug, but you are unleashing all sorts of problems by this.

If you really need to handle it as a pointer (and there may be some legitimate reasons), you need to emit a custom CIL with a pinned reference. It could even be initialized by extracting the pointer from a TypedReference, but it guarantees that the location will not change. Pass it to a lambda method then.

Joell answered 14/11, 2014 at 10:57 Comment(0)
R
2

Yes! You can trick the compiler into creating a field of type TypedReference, by creating an iterator or lambda that uses one (or maybe an async block):

static IEnumerable<object> Sneaky(TypedReference t1)
{
    yield return TypedReference.ToObject(t1);
}

static Func<object> Lambda(TypedReference t1)
{
    return () => TypedReference.ToObject(t1);
}

Unfortunately, the CLR does not let you actually use the class. You'll get a TypeLoadException when you try.

In other words, the TypedReference and ArgIterator types are protected not just by the compiler, but by the runtime. If you want to save a copy of either one, you'll have to do so by doing an unsafe blit onto the heap or Reflection.

Note that I tried this with the .Net 4.0 C# compiler. Other compilers may be smarter.

Reflect answered 7/1, 2013 at 7:19 Comment(1)
I've tried this with .NET 4.0 and 4.5 through VS2012 and I none of the two compile correctly. About using TypedReference.ToObject, I couldn't find a way to unbox or get the address of the TypedReference that the resulting object points to. I've tried GCHandle.Alloc but since the TypedReference points to a non-blittable type (in this case, an Action), the call throws an exception.Palmette
J
1

What are you trying to do exactly? Locals are on stack and arguments are too depending on the calling convention. Storing or returning the address of a local or argument is not good because it will get overriden. There is no way to prevent them from being overriden besides like not calling methods.

If you turn on unmanaged debugging you can use the memory debugger and register window to see what is going on.

Here is simpler to understand C example. Why does print not display the correct value. Because when the print function gets invoked its stack frame overwrites the value.

int* bad(int x, int y)
{
    int sum = x + y;
    return &sum;
};

int* bad2(int x, int y)
{
    x += y;
    return &x;
}

int _tmain(int argc, _TCHAR* argv[])
{
    int* sum1 = bad(10, 10);
    int* sum2 = bad(100, 100);
    printf("%d bad", *sum1);  // prints 200 instead of 20

    sum1 = bad2(10, 10);
    sum2 = bad2(100, 100);
    printf("%d bad", *sum1);  // prints 200 instead of 20

    return 0;
};

Cant get the clr to hold on for it longer. One thing you can do is get the variable on the stack pushed further out. Below is an example. This is all bad though :(

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Linq.Expressions;
using System.Xml.Linq;
using System.Runtime.InteropServices;

namespace Bad
{
    class Program
    {
        static void Main(string[] args)
        {
            Action a = () => Console.WriteLine("test");
            Horror h = new Horror();
            h.Fix(new Big(), ref a, new Big());
            h.Clear();
            Console.WriteLine();
        }
    }
    [StructLayout(LayoutKind.Sequential, Size = 4096)]
    struct Big
    {
    }
    unsafe class Horror
    {
        void* _ptr;

        static void Handler()
        {
            Console.WriteLine("Hello world.");
        }


        public void Fix(Big big, ref Action action, Big big2)
        {
            action += Handler;
            var tr = __makeref(action);
            _ptr = (void*)&tr;
        }

        public void Clear()
        {
            var tr = *(TypedReference*)_ptr;
            __refvalue(tr, Action) -= Handler;
        }
    }
}
Jarredjarrell answered 9/1, 2013 at 4:8 Comment(8)
About why a pointer to a variable allocated on the stack may will be invalid after the call where it's pushed terminates, thanks for your explanation but that was the actual point I've used to ask the question -- Finding a way to allocate it on the heap. The method you suggested, though, is new to me and it works (even if as you said, it's really "evil"). Would you explain its trick? I will accept the answer in case better methods aren't provided. By "better" I mean a systematic way to push a TypedReference on the heap and retrieve it when needed.Palmette
sorry tr is actually a local that is getting created on the stack not an argument. You can use the static ToObject method and store the object as a field in your class. Still confused on purpose though? TypeReferences are mainly for __arglist and hacky stuff like this here #4765073Jarredjarrell
The purpose is finding out if there's a way to bypass the compiler's checks against persisting a TypedReference instance. ToObject lets me store it as an object, but I don't know how to unbox it.Palmette
toObject is going to return the value the typed reference is wrapping. You can just copy the typedreference. It should be like 4-8 bytes. public void Fix(ref Action action) { action += Handler; TypedReference tr = __makeref(action); IntPtr mem = Marshal.AllocHGlobal(sizeof(TypedReference)); // leaked TypedReference* refPtr = (TypedReference*)mem.ToPointer(); _ptr = mem.ToPointer(); *refPtr = tr; }Jarredjarrell
This is exactly what I was looking for, nice workaround. Why did you comment the result of AllocHGlobal as leaked? Is it possible to use FreeHGlobal to deallocate the unmanaged memory, given the pointer? Either way, I'll close the bounty as soon as you edit your post to add the above comment's method. Thank you for your help.Palmette
yeah you can clean up or create your own type instead to represent the typedref. They dont let you have typedreference as field but it is just two intptrs. Make a struct that has two intptrs and copy maybeJarredjarrell
So, will you edit your post or can I do it for you? I just want the information that you wrote in the comment to be easily readable for anyone who will stumble upon this question.Palmette
even it seems to be valid it will point to some random garbage on memory. because as soon as those clever Fix method returns, CLR will leave monitoring it. in short you have to create and keep it on stack. in another words it is designed that way and it needs special treatment.Pulsate
P
1

TypedReference seems pretty conclusively locked down.

I guess it was simpler to just lock the type down to keep it safe, rather than allow it to be passed about but only in an unsafe context.

You can hold on to one for while... will cost you a thread though :)

namespace TehHorror
{
    using System;
    using System.Threading;    
    class Horror
    {
        private ManualResetEvent waiter = null;    
        public void Fix(ref Action multicast)
        {
            waiter = new ManualResetEvent(false);
            multicast += HorrorHandler;
            if (multicast != null) multicast();
            var tr = __makeref(multicast);
            waiter.WaitOne();
            __refvalue(tr, Action) -= HorrorHandler;
        }    
        public void Clear() { waiter.Set(); }    
        private static void HorrorHandler()
        {
            Console.WriteLine("Hello from horror handler.");
        }
    }    
    class Program
    {
        static void Main()
        {
            Action a = () => Console.WriteLine("Hello from original delegate");
            var horror = new Horror();
            a.Invoke();
            Action fix = () => horror.Fix(ref a);
            fix.BeginInvoke(fix.EndInvoke, null);
            Thread.Sleep(1000);
            horror.Clear();
            a.Invoke();
        }
    }
}
Pheni answered 11/1, 2013 at 19:38 Comment(1)
This method works as well, however it requires accurate synchronization to produce reliable results. In your instance, that Thread.Sleep is critical for the code to execute correctly. Furthermore, it would require another synchronization after horror.Clear() or there would be no guarantee for the handler to be reset to its original state. +1 either way, it is a possible solution.Palmette

© 2022 - 2024 — McMap. All rights reserved.