Casting a byte array to a managed structure

Question

Update: Answers to this question helped me code the open sourced project AlicanC's Modern Warfare 2 Tool on GitHub. You can see how I am reading these packets in MW2Packets.cs and the extensions I've coded to read big endian data in Extensions.cs.

I am capturing UDP packets of Call of Duty: Modern Warfare 2 using Pcap.Net in my C# application. I receive a byte[] from the library. I tried to parse it like a string, but that didn't work well.

The byte[] I have has a generic packet header, then another header specific to the packet type then info about each player in the lobby.

A helpful person inspected some packets for me and came up with these structures:

// Fields are big endian unless specified otherwise.
struct packet_header
{
    uint16_t magic;
    uint16_t packet_size;
    uint32_t unknown1;
    uint32_t unknown2;
    uint32_t unknown3;
    uint32_t unknown4;
    uint16_t unknown5;
    uint16_t unknown6;
    uint32_t unknown7;
    uint32_t unknown8;
    cstring_t packet_type; // \0 terminated string
};

// Fields are little endian unless specified otherwise.
struct header_partystate //Header for the "partystate" packet type
{
    uint32_t unknown1;
    uint8_t unknown2;
    uint8_t player_entry_count;
    uint32_t unknown4;
    uint32_t unknown5;
    uint32_t unknown6;
    uint32_t unknown7;
    uint8_t unknown8;
    uint32_t unknown9;
    uint16_t unknown10;
    uint8_t unknown11;
    uint8_t unknown12[9];
    uint32_t unknown13;
    uint32_t unknown14;
    uint16_t unknown15;
    uint16_t unknown16;
    uint32_t unknown17[10];
    uint32_t unknown18;
    uint32_t unknown19;
    uint8_t unknown20;
    uint32_t unknown21;
    uint32_t unknown22;
    uint32_t unknown23;
};

// Fields are little endian unless specified otherwise.
struct player_entry
{
    uint8_t player_id;

    // The following fields may not actually exist in the data if it's an empty entry.
    uint8_t unknown1[3];
    cstring_t player_name;
    uint32_t unknown2;
    uint64_t steam_id;
    uint32_t internal_ip;
    uint32_t external_ip;
    uint16_t unknown3;
    uint16_t unknown4;
    uint32_t unknown5;
    uint32_t unknown6;
    uint32_t unknown7;
    uint32_t unknown8;
    uint32_t unknown9;
    uint32_t unknown10;
    uint32_t unknown11;
    uint32_t unknown12;
    uint16_t unknown13;
    uint8_t unknown14[???];     // Appears to be a bit mask, sometimes the length is zero, sometimes it's one. (First entry is always zero?)
    uint8_t unknown15;
    uint32_t unknown16;
    uint16_t unknown17;
    uint8_t unknown18[???];     // Most of the time this is 4 bytes, other times it is 3 bytes.
};

I recreated the packet header structure in my C# application like this:

[StructLayout(LayoutKind.Sequential, Pack=1)]
struct PacketHeader
{
    public UInt16 magic;
    public UInt16 packetSize;
    public UInt32 unknown1;
    public UInt32 unknown2;
    public UInt32 unknown3;
    public UInt32 unknown4;
    public UInt16 unknown5;
    public UInt16 unknown6;
    public UInt32 unknown7;
    public UInt32 unknown8;
    public String packetType;
}

Then I tried to make a structure for the "partystate" header, but I got errors saying fixed keyword is unsafe:

[StructLayout(LayoutKind.Sequential, Pack=1)]
struct PartyStateHeader
{
    UInt32 unknown1;
    Byte unknown2;
    Byte playerEntryCount;
    UInt32 unknown4;
    UInt32 unknown5;
    UInt32 unknown6;
    UInt32 unknown7;
    Byte unknown8;
    UInt32 unknown9;
    UInt16 unknown10;
    Byte unknown11;
    fixed Byte unknown12[9];
    UInt32 unknown13;
    UInt32 unknown14;
    UInt16 unknown15;
    UInt16 unknown16;
    fixed UInt32 unknown17[10];
    UInt32 unknown18;
    UInt32 unknown19;
    Byte unknown20;
    UInt32 unknown21;
    UInt32 unknown22;
    UInt32 unknown23;
}

I couldn't do anything for the player entries because of the varying size of unknown14 and unknown18. (Player entries are the most important.)

Now, somehow, I have to cast the byte[] I have to these PacketHeader structures. Sadly, it's not easy as (PacketHeader)bytes. I tried this method I've found on the internet but it threw an AccessViolationException:

GCHandle handle = GCHandle.Alloc(bytes, GCHandleType.Pinned);
PacketHeader packetHeader = (PacketHeader)Marshal.PtrToStructure(handle.AddrOfPinnedObject(), typeof(PacketHeader));

How can I achieve this?

Hebraism

My answer here doesn't marshal the structure directly, but takes the approach of extracting field values from the byte array you read from the network. You'll probably have to flip the big-endian bits yourself though; the .NET base classes are all little-endian AFAIK. – Prurient

Answer 1

I'd turn the byte array into a memory stream. Then instantiate a binary reader on that stream. And then define helper functions that take a binary reader and parse a single class.

The built in BinaryReader class always uses little endian.

I'd use classes instead of structs here.

class PacketHeader 
{
    uint16_t magic;
    uint16_t packet_size;
    uint32_t unknown1;
    uint32_t unknown2;
    uint32_t unknown3;
    uint32_t unknown4;
    uint16_t unknown5;
    uint16_t unknown6;
    uint32_t unknown7;
    uint32_t unknown8;
    string packet_type; // replaced with a real string
};

PacketHeader ReadPacketHeader(BinaryReader reader)
{
  var result=new PacketHeader();
  result.magic = reader.ReadInt16();
  ...
  result.packet_type=ReadCString();//Some helper function you might need to define yourself
  return result;
}

Milli

This is not an answer to my question but what I actually need. The structs (which are classes now) had dynamic sizes. Player entries don't have their last byte in certain situations and they have strings with dynamic sizes. The best solution was to read them one by one instead of casting them directly. – Hebraism

Answer 2

//I have found this at: http://code.cheesydesign.com/?p=572 (I have not tested yet, but // at first sight it will work well.)

    /// <summary>
    /// Reads in a block from a file and converts it to the struct
    /// type specified by the template parameter
    /// </summary>
    /// <typeparam name="T"></typeparam>
    /// <param name="reader"></param>
    /// <returns></returns>
    private static T FromBinaryReader<T>(BinaryReader reader)
    {

        // Read in a byte array
        byte[] bytes = reader.ReadBytes(Marshal.SizeOf(typeof(T)));

        // Pin the managed memory while, copy it out the data, then unpin it
        GCHandle handle = GCHandle.Alloc(bytes, GCHandleType.Pinned);
        T theStructure = (T)Marshal.PtrToStructure(handle.AddrOfPinnedObject(), typeof(T));
        handle.Free();

        return theStructure;
    }

Tactical

This answer helped me tremendously. It can be used as a reinterpret cast. – Litta

How does this function handle c strings? – Melly

Protip: add a try/finally block to ensure that handle.Free() is always called, even if PtrToStructure throws an exception, otherwise you'll leak memory. – Capsicum

Answer 3

This is how i did:

using System;
using System.Runtime.InteropServices;
public static object GetObjectFromBytes(byte[] buffer, Type objType)
{
    object obj = null;
    if ((buffer != null) && (buffer.Length > 0))
    {
        IntPtr ptrObj = IntPtr.Zero;
        try
        {
            int objSize = Marshal.SizeOf(objType);
            if (objSize > 0)
            {
                if (buffer.Length < objSize)
                    throw new Exception(String.Format("Buffer smaller than needed for creation of object of type {0}", objType));
                ptrObj = Marshal.AllocHGlobal(objSize);
                if (ptrObj != IntPtr.Zero)
                {
                    Marshal.Copy(buffer, 0, ptrObj, objSize);
                    obj = Marshal.PtrToStructure(ptrObj, objType);
                }
                else
                    throw new Exception(String.Format("Couldn't allocate memory to create object of type {0}", objType));
            }
        }
        finally
        {
            if (ptrObj != IntPtr.Zero)
                Marshal.FreeHGlobal(ptrObj);
        }
    }
    return obj;
}

And in the struct definition I didn't use any fixed region, instead I used the MarshalAs attribute if the standard marshalling didn't worked. This is what you will probally need for the string.

You would use this function like this:

PacketHeader ph = (PacketHeader)GetObjectFromBytes(buffer, typeof(PacketHeader));

Edit: I didn´t see your BigEndian "restriction" in the code example. This solution will only work if the bytes are LittleEndian.

Edit 2: In the string of your example you would decorate it with:

[MarshalAs(UnmanagedType.LPStr)]

In the arrays I would go with something like this for a n-sized array:

[MarshalAs(UnmanagedType.ByValArray, SizeConst = n)]

Unreserved

Answer 4

If you want fast code without copy, this is the solution. We are working here on raw byte[], simply casting the pointer in unsafe code, as you would in native C / C++. So there is no overhead calling into expensive framework methods, making copies etc. etc.

Any change to the unmanaged struct will reflect in managed byte[], and vice versa.

//FOR DEBUG/TEST ONLY
using System.Runtime.InteropServices;
namespace ByteStructCast1
{
    class Program
    {
        [StructLayout(LayoutKind.Sequential, Pack = 1)]
        unsafe struct StructTest//4B
        {
            [MarshalAs(UnmanagedType.U2)]
            public ushort item1; //2B
            public fixed byte item2[2]; //2B =2x 1B
        }
        static void Main(string[] args)
        {
            //managed byte array
            byte[] DB1 = new byte[7]; //7B more than we need. byte buffer usually is greater.
            DB1[0] = 2;//test data |> LITTLE ENDIAN
            DB1[1] = 0;//test data |
            DB1[2] = 3;//test data
            DB1[3] = 4;//test data
            unsafe //we'll now pin unmanaged struct over managed byte array
            {
                fixed(byte* db1 = DB1) //db1 is pinned pointer to DB1 byte[] array
                {
                    //StructTest t1 = *(StructTest*)db1;    //does not change DB1/db1
                    //t1.item1 = 11;                        //does not change DB1/db1
                    db1[0] = 22;                            //does CHANGE DB1/db1
                    DB1[0] = 33;                            //does CHANGE DB1/db1
                    StructTest* ptest = (StructTest*)db1;   //does CHANGE DB1/db1
                    ptest->item1 = 44;                      //does CHANGE DB1/db1
                    ptest->item2[0]++;                      //does CHANGE DB1/db1
                    ptest->item2[1]--;                      //does CHANGE DB1/db1
                }
            }
        }
    }
}

This can also be used when you are working with primitive-typed fixed-size buffers, and need to work on elements thereof as structs with members, e.g. ulong to MyStruct, both 64 bits long.

Beverle

Answer 5

For those who have access to C# 7.3 features, I use this piece of unsafe code to "serialize" to bytes:

public static class Serializer
{
    public static unsafe byte[] Serialize<T>(T value) where T : unmanaged
    {
        byte[] buffer = new byte[sizeof(T)];

        fixed (byte* bufferPtr = buffer)
        {
            Buffer.MemoryCopy(&value, bufferPtr, sizeof(T), sizeof(T));
        }

        return buffer;
    }

    public static unsafe T Deserialize<T>(byte[] buffer) where T : unmanaged
    {
        T result = new T();

        fixed (byte* bufferPtr = buffer)
        {
            Buffer.MemoryCopy(bufferPtr, &result, sizeof(T), sizeof(T));
        }

        return result;
    }
}

A unmanaged type can be a struct (simple struct without reference types, those a considered managed structs) or a native type such as int, short, etc.

Forepaw

Answer 6

Well, you have two tasks here really. First is to interpret byte[] as struct essentially and second is to deal with possible different endianness.

So, they are somewhat diverge. AFAIK if you want to use marshaling - it will just interpret bytes as if it were managed structure. So converting from one endian to another is left to you. It is not hard to do but it will not be automatic.

So, to interpret byte[] as struct you have to have something like that:

[StructLayout(LayoutKind.Sequential)]
internal struct X
{
    public int IntValue;
    [MarshalAs(UnmanagedType.ByValArray, SizeConst = 3, ArraySubType = UnmanagedType.U1)] 
    public byte[] Array;
}

static void Main(string[] args)
{
    byte[] data = {1, 0, 0, 0, 9, 8, 7}; // IntValue = 1, Array = {9,8,7}
    IntPtr ptPoit = Marshal.AllocHGlobal(data.Length);
    Marshal.Copy(data, 0, ptPoit, data.Length);
    var x = (X) Marshal.PtrToStructure(ptPoit, typeof (X));
    Marshal.FreeHGlobal(ptPoit);

    Console.WriteLine("x.IntValue = {0}", x.IntValue);
    Console.WriteLine("x.Array = ({0}, {1}, {2})", x.Array[0], x.Array[1], x.Array[2]);
}

So first 4 bytes go to IntValue (1,0,0,0) -> [little endian] -> 1 Next 3 bytes go directly to array.

If you want BigEndian you should do it yourself:

int LittleToBigEndian(int littleEndian)
{
    byte[] buf = BitConverter.GetBytes(littleEndian).Reverse().ToArray();
    return BitConverter.ToInt32(buf, 0);
}

It is somewhat messy like that, so probably for you will be better to stick with your custom-written parser that takes bytes one-by-one from source byte[] and fill your data class without StructLayout and other native interop.

Simulator

it would be great to point out that in case of unmanaged structure packed up to a byte it best to declare the structure as [StructLayout(LayoutKind.Sequential), Pack = 1] – Jacobo

Answer 7

To convert a byte array to a string you do this;

byte [] dBytes = ...
string str;
System.Text.UTF8Encoding enc = new System.Text.UTF8Encoding();
str = enc.GetString(dBytes);

And to convert the string back to a byte array

public static byte[] StrToByteArray(string str)
{
    System.Text.UTF8Encoding  encoding=new System.Text.UTF8Encoding();
    return encoding.GetBytes(str);
}

Now read your string and see what your data is.

Nanette

I could convert it to a string. It didn't work well because it's not the way it should be done. – Hebraism