Using reflection to determine how a .Net type is layed out in memory

Question

I'm experimenting with optimizing parser combinators in C#. One possible optimization, when the serialized format matches the in-memory format, is to just do an (unsafe) memcpy of the data to be parsed over an instance or even many instances of the type.

I want to write code that determines if the in-memory format matches the serialized format, in order to dynamically determine if the optimization can be applied. (Obviously this is an unsafe optimization and might not work for a whole bunch of subtle reasons. I'm just experimenting, not planning to use this in production code.)

I use the attribute [StructLayout(LayoutKind.Sequential, Pack = 1)] to force no padding and to force the in-memory order to match declaration order. I check for that attribute with reflection, but really all this confirms is "no padding". I also need the order of the fields. (I would strongly prefer to not have to manually specified FieldOffset attributes for every field, since that would be very error prone.)

I assumed I could use the order of fields returned by GetFields, but the documentation explicitly calls out that the order is unspecified.

Given that I am forcing the order of fields with the StructLayout attribute, is there a way to reflect on that ordering?

edit I'm fine with the restriction that all of the fields must be blittable.

Camden

Could you not work it out by reflecting into those attributes? – Gomuti

@newStackExchangeInstance Which attributes? – Camden

LayoutKind.Sequential only controls the managed representation if only blittable types are present in the structure. If there is an unblittable type, the field order is controlled by the runtime anyway. E.g. see https://mcmap.net/q/1158311/-unions-in-c-structure-members-do-not-seem-to-be-aligned/11683. – Bayles

The actual layout of a type in memory seems like it's going to be completely implementation-dependent and therefore your proposed optimization a non-starter. What good is an experiment if it will never be usable in production code? – Hub

@CodyGray I use the StructLayout attribute to force the layout. It shouldn't change between implementations, unless the underlying values are changing in size (e.g. pointers). Sometimes people do things for fun. – Camden

@Bayles It's good to know that there's actually types considered to be blittable. So, assuming I have a struct filled with blittable fields, how do I get the order? – Camden

@CodyGray - we do things like this all the time to squeeze out the last drops of performance for our trading systems – Struck

If you care that much about performance, why write the code in C#, @hoo? – Hub

I write the code in several languages, including C++, CIL, and assembly as required. C# makes a great language for gluing it all together though because it has the facilities to manage precision memory layout. It also serves as a wonderful bridge so that junior devs can work on other parts of the project, like the GUI or business code. – Struck

I did all structs (except nullable github.com/invertedtomato/lightweight-serialization/issues/2, working on nullables). I can do classes if these are blittable, but trying to find way do all classes. – Footcloth

Answer 1

This is unnecessary if using LayoutKind.Sequential with blittable types

You don't need to use reflection or any other mechanism to find out the order of struct fields in memory, as long as all the fields are blittable.

The blittable fields for a struct declared with LayoutKind.Sequential will be in memory in the order in which the fields are declared. That's what LayoutKind.Sequential means!

From this documentation:

For blittable types, LayoutKind.Sequential controls both the layout in managed memory and the layout in unmanaged memory. For non-blittable types, it controls the layout when the class or structure is marshaled to unmanaged code, but does not control the layout in managed memory.

Note that this doesn't tell you how much padding each field is using. To find that out, see below.

To determine the field order when using LayoutKind.Auto, or the field offsets when using any layout

It's fairly easy to find the struct field offsets if you're happy to use unsafe code, and to not use reflection.

You just need to take the address of each field of the struct and calculate its offset from the start of the struct. Knowing the offsets of each field, you can calculate their order (and any padding bytes between them). To calculate the padding bytes used for the last field (if any) you will also need to get the total size of the struct using sizeof(StructType).

The following example works for 32-bit and 64-bit. Note that you don't need to use fixed keyword because the struct is already fixed due to it being on the stack (you'll get a compile error if you try to use fixed with it):

using System;
using System.Runtime.InteropServices;

namespace Demo
{
    [StructLayout(LayoutKind.Auto, Pack = 1)]

    public struct TestStruct
    {
        public int    I;
        public double D;
        public short  S;
        public byte   B;
        public long   L;
    }

    class Program
    {
        void run()
        {
            var t = new TestStruct();

            unsafe
            {
                IntPtr p  = new IntPtr(&t);
                IntPtr pI = new IntPtr(&t.I);
                IntPtr pD = new IntPtr(&t.D);
                IntPtr pS = new IntPtr(&t.S);
                IntPtr pB = new IntPtr(&t.B);
                IntPtr pL = new IntPtr(&t.L);

                Console.WriteLine("I offset = " + ptrDiff(p, pI));
                Console.WriteLine("D offset = " + ptrDiff(p, pD));
                Console.WriteLine("S offset = " + ptrDiff(p, pS));
                Console.WriteLine("B offset = " + ptrDiff(p, pB));
                Console.WriteLine("L offset = " + ptrDiff(p, pL));

                Console.WriteLine("Total struct size = " + sizeof(TestStruct));
            }
        }

        long ptrDiff(IntPtr p1, IntPtr p2)
        {
            return p2.ToInt64() - p1.ToInt64();
        }

        static void Main()
        {
            new Program().run();
        }
    }
}

To determine the field offsets when using LayoutKind.Sequential

If your struct uses LayoutKind.Sequential then you can use Marshal.OffsetOf() to get the offset directly, but this does not work with LayoutKind.Auto:

foreach (var field in typeof(TestStruct).GetFields())
{
    var offset = Marshal.OffsetOf(typeof (TestStruct), field.Name);
    Console.WriteLine("Offset of " + field.Name + " = " + offset);
}

This is clearly a better way to do it if you are using LayoutKind.Sequential since it doesn't require unsafe code, and it's much shorter - and you don't need to know the names of the fields in advance. As I said above, it is not needed to determine the order of the fields in memory - but this might be useful if you need to find out about how much padding is used.

Phonometer

Thanks, using the pointer differences is exactly the sort of thing I needed. As long as .Net disallows any optimizations were fields are ellided or anything like that... – Camden

I get a "Cannot take the address of the given expression" compiler error when I try to apply the & operator to a field like t.I. – Camden

@Strilanc If you copy and paste my code it will work fine, so you must be doing something different. Can you make a new question asking why what you're doing won't work? It's impossible to diagnose in comments here. I know the code I posted works, and it also doesn't contain the code t.l (note the lowercase l) anywhere in it, so I know you must be doing something different. :) – Phonometer

@Strilanc That's interesting - I've never tried to take the address of a readonly field, so I didn't know that! – Phonometer

@Strilanc If you do need to do that, you can do it inside a constructor for that struct (but you would need to use the fixed keyword when taking the address of the fields if you do it from a constructor). – Phonometer

Answer 2

As a reference for those who want to know the order and the kind of layout. For example if a type contains non-blittable types.

var fields = typeof(T).GetFields(BindingFlags.NonPublic | BindingFlags.Public | BindingFlags.Instance);
fields.SortByFieldOffset();

var isExplicit = typeof(T).IsExplicitLayout;
var isSequential = typeof(T).IsLayoutSequential;

It uses an extension method that I wrote:

    public static void SortByFieldOffset(this FieldInfo[] fields) {
        Array.Sort(fields, (a, b) => OffsetOf(a).CompareTo(OffsetOf(b)) );
    }

    private static int OffsetOf(FieldInfo field) {
        return Marshal.OffsetOf(field.DeclaringType, field.Name).ToInt32();
    }

MSDN contains useful info on IsLayoutSequential.

Longstanding

return fields.OrderBy(OffsetOf).ToArray() is a bit more succinct, and immutable to boot. – Camden