Is there an alternative for StructLayout "Pack" attribute in Compact Framework?

Question

I would like to do the following:

  [StructLayout(LayoutKind.Sequential, Pack = 1)]
  public struct SomeStruct
  {
     public byte  SomeByte;
     public int   SomeInt;
     public short SomeShort;
     public byte  SomeByte2;
  }

Is there an alternative since Pack is not supported in the compact framework?

Update: Explicitly setting up the structure and giving FieldOffset for each does not work either as it does not affect how the struct is packed

Update2: If you try the following, the CF program wont even run because of how the structure is packed:

[StructLayout(LayoutKind.Explicit, Size=8)]
public struct SomeStruct
{
   [FieldOffset(0)]
   public byte SomeByte;
   [FieldOffset(1)]
   public int SomeInt;
   [FieldOffset(5)]
   public short SomeShort;
   [FieldOffset(7)]
   public byte SomeByte2;
}

I know it seems hard to believe, but if you try it you will see. Add it to a CF project and try to run it and you will get a TypeLoadException. Changing the offsets to 0,4,8,10 respectively and it will work (but the size ends up being 12).

I was hoping maybe someone had a solution using reflection maybe to marshal the size of each of the field types individually (something involving recursion to handle structs within structs or arrays of types).

Higa

You commented "Explicit Layout does not solve the problem either" - what is the actual problem you want to solve? What is currently actually not working. – Sorn

Trying to pack the structure so that the size turns out to be 8 bytes. Without the pack attribute the size is marshaled larger than expected (because of default packing size) – Higa

The solution might involve marshallings individual fields of the struct to determine the size (also, note that I could have arrays inside my structure as well); which introduces more problems as seen with my other post here: #1269398 – Higa

If my answer is not allowed because you require this as a marshalled form of some struct defined elsewhere you have no option but to declare some struct of bytes/ints/whatever with acceptable alignment and do the unpacking yourself respecting endianess etc...) – Sabella

I supplied a work around albeit a painful one given your needs – Sabella

It would be nice if we could have a sort of universal custom marshaler that used reflection. But in doing this you would basically be rewriting and extending the built-in marshaling framework. Unless you put in an incredible amount of work it would be horribly fragile and likely unusable. – Keeley

Answer 1

This probably isn't the type of answer you're looking for, but I'll post it anyway for the hell of it:

public struct SomeStruct
{
    public byte SomeByte;
    public int SomeInt;
    public short SomeShort;
    public byte SomeByte2;


    public byte[] APIStruct
    {
        get
        {
            byte[] output = new byte[8];
            output[0] = this.SomeByte;
            Array.Copy(BitConverter.GetBytes(this.SomeInt), 0,
                output, 1, 4);
            Array.Copy(BitConverter.GetBytes(this.SomeShort), 0,
                output, 5, 2);
            output[7] = this.SomeByte2;
            return output;
        }
        set
        {
            byte[] input = value;
            this.SomeByte = input[0];
            this.SomeInt = BitConverter.ToInt32(input, 1);
            this.SomeShort = BitConverter.ToInt16(input, 5);
            this.SomeByte2 = input[7];
        }
    }
}

Basically it does the packing/unpacking itself in the APIStruct property.

Gaikwar

Yeah, this is one approach that could work, I was hoping to avoid manually have to index each one, but maybe this is the best solution. – Higa

I think it could be modified to iterate through the properties and use SizeOf or whatever to pack it into a byte[], so you wouldn't have to manually index everything. This was just a quick sample. Another benefit is that it would work in CF or the full framework without modification (but I think maybe I'm the only person writing apps that run in either without recompiling). – Gaikwar

I'm trying :P I definitely wanted something that could run on both frameworks. – Higa

Just in case I'm missing something, I'm curious how this is different from the answer I wrote 2 days ago? – Keeley

@Stephen: it's simpler and more readable, IMHO. Other than that, it's the same basic principle. – Gaikwar

Fair enough. I thought I might be misunderstanding something. – Keeley

Answer 2

The simplest method of dealing with this type of problem is along the same lines as you might for a bit field, simply pack your data into a private member (or members if it is large) of the appropriate data type and then present public properties that unpack the data for you. The unpacking operations are extremely fast and will have little impact on performance. For your particular type the following is probably what you want:

public struct SomeStruct
{
    private long data;

    public byte SomeByte { get { return (byte)(data & 0x0FF); } }
    public int SomeInt { get { return (int)((data & 0xFFFFFFFF00) << 8); } }
    public short SomeShort { get { return (short)((data & 0xFFFF0000000000) << 40); } }
    public byte SomeByte2 { get { return (byte)((data & unchecked((long)0xFF00000000000000)) << 56); } }
}

For some structures even this method is not workable due to the unfortunate way a structure has been defined. In those cases you will generally have to use a byte array as a blob of data from which the elements can be unpacked.

EDIT: To expand on what I mean about structs that can't be handled using this simple method. When you can't do simple packing/unpacking like this you need to manually marshal the irregular struct . This can be done using manual methods at the point you call the pInvoked API or by using a custom marshaler. The following is an example of a custom marhsaler that can be easily adapted to on the spot manual marshaling.

using System.Runtime.InteropServices;
using System.Threading;

public class Sample
{
    [DllImport("sample.dll")]
    public static extern void TestDataMethod([MarshalAs(UnmanagedType.CustomMarshaler, MarshalTypeRef = typeof(TestDataMarshaler))] TestDataStruct pData);
}

public class TestDataStruct
{
    public byte data1;
    public int data2;
    public byte[] data3 = new byte[7];
    public long data4;
    public byte data5;
}

public class TestDataMarshaler : ICustomMarshaler
{
    //thread static since this could be called on 
    //multiple threads at the same time.
    [ThreadStatic()]
    private static TestDataStruct m_MarshaledInstance;

    private static ICustomMarshaler m_Instance = new TestDataMarshaler();

    public static ICustomFormatter GetInstance(string cookie)
    {
        return m_Instance;
    }

    #region ICustomMarshaler Members

    public void CleanUpManagedData(object ManagedObj)
    {
        //nothing to do.
    }

    public void CleanUpNativeData(IntPtr pNativeData)
    {
        Marshal.FreeHGlobal(pNativeData);
    }

    public int GetNativeDataSize()
    {
        return 21;
    }

    public IntPtr MarshalManagedToNative(object ManagedObj)
    {
        m_MarshaledInstance = (TestDataStruct)ManagedObj;
        IntPtr nativeData = Marshal.AllocHGlobal(GetNativeDataSize());

        if (m_MarshaledInstance != null)
        {
            unsafe //unsafe is simpler but can easily be done without unsafe if necessary
            {
                byte* pData = (byte*)nativeData;
                *pData = m_MarshaledInstance.data1;
                *(int*)(pData + 1) = m_MarshaledInstance.data2;
                Marshal.Copy(m_MarshaledInstance.data3, 0, (IntPtr)(pData + 5), 7);
                *(long*)(pData + 12) = m_MarshaledInstance.data4;
                *(pData + 20) = m_MarshaledInstance.data5;
            }
        }
        return nativeData;
    }

    public object MarshalNativeToManaged(IntPtr pNativeData)
    {
        TestDataStruct data = m_MarshaledInstance;
        m_MarshaledInstance = null; //clear out TLS for next call.

        if (data == null) data = new TestDataStruct(); //if no in object then return a new one

        unsafe //unsafe is simpler but can easily be done without unsafe if necessary
        {
            byte* pData = (byte*)pNativeData;
            data.data1 = *pData;
            data.data2 = *(int*)(pData + 1);
            Marshal.Copy((IntPtr)(pData + 5), data.data3, 0, 7);
            data.data4 = *(long*)(pData + 12);
            data.data5 = *(pData + 20);
        }
        return data;
    }

    #endregion
}

In the case of arrays of these structs you can't use custom marshaling unless the array size is fixed but it is relatively easy to manually marshal the array data as a whole using the same techniques.

Keeley

Answer 3

Do you absolutely require that specific layout or is it acceptable to simply make the size 8?

I ask this because the lay out as follows

[StructLayout(LayoutKind.Explicit, Size=8)]
public struct SomeStruct
{
   [FieldOffset(0)]
   public byte SomeByte;
   [FieldOffset(1)]
   public int SomeInt;
   [FieldOffset(5)]
   public short SomeShort;
   [FieldOffset(7)]
   public byte SomeByte2;
}

Has non word aligned fields which may be what is causing your problem.

If you can 'rearrange' things then this might work for you:

[StructLayout(LayoutKind.Explicit, Size=8)]
public struct SomeStruct
{
   [FieldOffset(0)]
   public byte SomeByte;
   [FieldOffset(1)]
   public byte SomeByte2;
   [FieldOffset(2)]
   public short SomeShort;
   [FieldOffset(4)]
   public int SomeInt;
}

When I test with this on the emulator it works fine.

Obviously unless you are willing to allow the rearrangement there's nothing you can do.

This answer and this old article would strongly indicate that you must at a minimum align your structs on multiples of their size (I tried with an int aligned on offset 2 and this also triggered the error)

Given your need to interoperate with externally defined data then the following is likely your easiest solution:

[StructLayout(LayoutKind.Explicit, Size=8)]
public struct SomeStruct
{ 
   [FieldOffset(0)] private byte b0;
   [FieldOffset(1)] private byte b1;
   [FieldOffset(2)] private byte b2;
   [FieldOffset(3)] private byte b3;
   [FieldOffset(4)] private byte b4;
   [FieldOffset(5)] private byte b5;
   [FieldOffset(6)] private byte b6;
   [FieldOffset(7)] private byte b7;

   // not thread safe - alter accordingly if that is a requirement
   private readonly static byte[] scratch = new byte[4];       

   public byte SomeByte 
   { 
       get { return b0; }
       set { b0 = value; }
   }

   public int SomeInt
   {
       get 
       { 
           // get the right endianess for your system this is just an example!
           scratch[0] = b1;
           scratch[1] = b2;
           scratch[2] = b3;
           scratch[3] = b4;
           return BitConverter.ToInt32(scratch, 0);
       }
   }

   public short SomeShort
   {
        get 
        { 
            // get the right endianess for your system this is just an example!
            scratch[0] = b5;
            scratch[1] = b6;
            return BitConverter.ToInt16(scratch, 0);
        }
    }

    public byte SomeByte2 
    { 
        get { return b7; }
        set { b7 = value; }
    }
}

Sabella

Thanks for the post Shuggy and giving this a try. I do realize that its an alignment issue, but I need to keep the struct the way it is (specific message format). – Higa

Answer 4

You need to post a more relevant example. Setting packing on that struct would have no effect anyway.

My bet is that you need to use LaoutKind.Explicit and then give the offsets for each member. It's way better than messing with the packing anyway, because it's way more obvious to someone looking at the code that the original developer explicitly meant for things to be unaligned.

Something along these lines:

[StructLayout(LayoutKind.Explicit)]
struct Foo
{
    [FieldOffset(0)]
    byte a;
    [FieldOffset(1)]
    uint b;
}

Victualler

Answer 5

I think one should take Stephen Martin's answer, make it accept a T, and use reflection to generically implement the MarshalManagedToNative and MarshalNativeToManaged methods. Then, you'll have a custom packed struct marshaler that will work for any type of struct.

Here's the code:

using System;
using System.Threading;
using System.Reflection;
using System.Runtime.InteropServices;

namespace System.Runtime.InteropServices
{
    public class PinnedObject : IDisposable
    {
        private GCHandle gcHandle = new GCHandle();
        public PinnedObject(object o)
        {
            gcHandle = GCHandle.Alloc(o, GCHandleType.Pinned);
        }

        public unsafe static implicit operator byte*(PinnedObject po)
        {
            return (byte*)po.gcHandle.AddrOfPinnedObject();
        }

        #region IDisposable Members
        public void Dispose()
        {
            if (gcHandle.IsAllocated)
            {
                gcHandle.Free();
            }
        }
        #endregion
    }

    public class PackedStructMarshaler<T> : ICustomMarshaler where T : struct
    {
        private static ICustomMarshaler m_instance = new PackedStructMarshaler<T>();

        public static ICustomMarshaler GetInstance()
        {
            return m_instance;
        }

        private void ForEachField(Action<FieldInfo> action)
        {
            foreach (var fi in typeof(T).GetFields(BindingFlags.Public | BindingFlags.NonPublic))
            {
                // System.Diagnostics.Debug.Assert(fi.IsValueType);
                action(fi);
            }
        }

        private unsafe void MemCpy(byte* dst, byte* src, int numBytes)
        {
            for (int i = 0; i < numBytes; i++)
            {
                dst[i] = src[i];
            }
        }

        #region ICustomMarshaler Members
        public void CleanUpManagedData(object ManagedObj)
        {
        }

        public void CleanUpNativeData(IntPtr pNativeData)
        {
            Marshal.FreeHGlobal(pNativeData);
        }

        public int GetNativeDataSize()
        {
            unsafe
            {
                int ret = 0;
                ForEachField(
                    (FieldInfo fi) =>
                    {
                        Type ft = fi.FieldType;
                        ret += Marshal.SizeOf(ft);
                    });
                return ret;
            }
        }

        private object m_marshaledObj = null;

        public unsafe IntPtr MarshalManagedToNative(object obj)
        {
            IntPtr nativeData = (IntPtr)0;

            if (obj != null)
            {
                if (m_marshaledObj != null)
                    throw new ApplicationException("This instance has already marshaled a managed type");

                m_marshaledObj = obj;

                nativeData = Marshal.AllocHGlobal(GetNativeDataSize());
                byte* pData = (byte*)nativeData;
                int offset = 0;

                ForEachField(
                    (FieldInfo fi) =>
                    {
                        int size = Marshal.SizeOf(fi.FieldType);
                        using (PinnedObject po = new PinnedObject(fi.GetValue(obj)))
                        {
                            MemCpy(pData + offset, po, size);
                        }
                        offset += size;
                    });
            }

            return nativeData;
        }

        public object MarshalNativeToManaged(IntPtr pNativeData)
        {
            if (m_marshaledObj != null)
                m_marshaledObj = null;

            unsafe
            {
                byte* pData = (byte*)pNativeData;
                int offset = 0;

                object res = new T();

                ForEachField(
                    (FieldInfo fi) =>
                    {
                        int size = Marshal.SizeOf(fi.FieldType);
                        fi.SetValue(res, (object)(*((byte*)(pData + offset))));
                        offset += size;
                    });

                return res;
            }
        }

        #endregion
    }
}

Revue

Answer 6

LayoutKind.Explicit would be your best bet for defining a specific memory layout. However, do not use LayoutKind.Explicit for structures that contain pointer-sized values such as true pointers, operating system handles or IntPtrs; this is just asking for mysterious trouble at runtime on random platforms.

In particular, LayoutKind.Explicit is a poor substitute for anonymous unions. If your target structure contains an anonymous union, convert it to a named union; you can safely represent a named union as a struct with LayoutKind.Explicit where all offsets are 0.

Sapajou

Explicit layout with pointers is actually pretty common practice in the CF where the Marshaller sucks and can't do it for you. – Victualler

Agreed, the CF marshaller is weak. I usually use the second technique though: LayoutKind.Sequential where possible and convert anonymous unions to named ones. – Sapajou

@JeffreyHantin if you can change all of the code involved or at least re-compile it, then yes, using anonymous unions is a poor decision. But often it's a hotfix for framework(s) inconsistency. For example, ecma-334 allows enum to hold any value of the underlying type. But most of the BCL classes completely ignore this "feature", also serializers often force Enum.IsDefined. One way to go is to overlap enum with some public integer. And yes, I know that it's strange to post on topics dated back to '09 =) – Litter

Answer 7

LayoutKind.Explicit and FieldOffsetAttribute will allow you to do anything you could do with the Pack property. These explicit layout attributes allow you to specify the exact byte position of each field in the struct (relative to the beginning of the struct's range of memory). The Pack property is used by the runtime to help determine the exact position of each field when using a sequential layout. The pack property has no other effect, so using explicit layout allows you to emulate the exact same behavior, albeit a bit more verbosely. If you don't think this solves your problem, perhaps you could post a bit more information about what you're trying to do or why you think you need to use the Pack property.

Edit: I just noticed the additional comment about trying to get the entire structure's size to 8 bytes. Have you tried using the StructLayoutAttribute.Size property? Unlike Pack, it is available in the Compact Framework.

Stadler

See my update2. Size property doesnt help (I think it will actually case a TypeLoadException also when the size is set to smaller than the actual struct size using explicit) – Higa