Bit fields in C#

Question

I have a structure which I need to populate and write to disk (several actually).

An example is:

byte-6    
bit0 - original_or_copy  
bit1 - copyright  
bit2 - data_alignment_indicator  
bit3 - PES_priority  
bit4-bit5 - PES_scrambling control.  
bit6-bit7 - reserved  

In C I might do something like the following:

struct PESHeader  {
    unsigned reserved:2;
    unsigned scrambling_control:2;
    unsigned priority:1;
    unsigned data_alignment_indicator:1;
    unsigned copyright:1;
    unsigned original_or_copy:1;
};

Is there any way to do this in C# that would enable me to access the bits using the struct dereferencing dot operator?

For a couple of structures, I can just do bit shifting wrapped in an accessor function.

I have loads of structures to handle in this way, so I'm looking for something that's easier to read and quicker to write.

Egan

Answer 1

I'd probably knock together something using attributes, then a conversion class to convert suitably attributed structures to the bitfield primitives. Something like...

using System;

namespace BitfieldTest
{
    [global::System.AttributeUsage(AttributeTargets.Field, AllowMultiple = false)]
    sealed class BitfieldLengthAttribute : Attribute
    {
        uint length;

        public BitfieldLengthAttribute(uint length)
        {
            this.length = length;
        }

        public uint Length { get { return length; } }
    }

    static class PrimitiveConversion
    {
        public static long ToLong<T>(T t) where T : struct
        {
            long r = 0;
            int offset = 0;

            // For every field suitably attributed with a BitfieldLength
            foreach (System.Reflection.FieldInfo f in t.GetType().GetFields())
            {
                object[] attrs = f.GetCustomAttributes(typeof(BitfieldLengthAttribute), false);
                if (attrs.Length == 1)
                {
                    uint fieldLength  = ((BitfieldLengthAttribute)attrs[0]).Length;

                    // Calculate a bitmask of the desired length
                    long mask = 0;
                    for (int i = 0; i < fieldLength; i++)
                        mask |= 1 << i;

                    r |= ((UInt32)f.GetValue(t) & mask) << offset;

                    offset += (int)fieldLength;
                }
            }

            return r;
        }
    }

    struct PESHeader
    {
        [BitfieldLength(2)]
        public uint reserved;
        [BitfieldLength(2)]
        public uint scrambling_control;
        [BitfieldLength(1)]
        public uint priority;
        [BitfieldLength(1)]
        public uint data_alignment_indicator;
        [BitfieldLength(1)]
        public uint copyright;
        [BitfieldLength(1)]
        public uint original_or_copy;
    };

    public class MainClass
    {
        public static void Main(string[] args)
        {
            PESHeader p = new PESHeader();

            p.reserved = 3;
            p.scrambling_control = 2;
            p.data_alignment_indicator = 1;

            long l = PrimitiveConversion.ToLong(p);


            for (int i = 63; i >= 0; i--)
            {
                Console.Write( ((l & (1l << i)) > 0) ? "1" : "0");
            }

            Console.WriteLine();

            return;
        }
    }
}

Which produces the expected ...000101011. Of course, it needs more error checking and a slightly saner typing, but the concept is (I think) sound, reusable, and lets you knock out easily maintained structures by the dozen.

adamw

Theologize

NOTE: Per MSDN, "The GetFields method does not return fields in a particular order, such as alphabetical or declaration order. Your code must not depend on the order in which fields are returned, because that order varies." Doesn't this cause a problem here? – Phosphorism

If you create a IBitfield 'marker' interface (having no members) you can convert the PrimitiveConversion class to extension methods for any structure that implements IBitfield. For example: public static long ToLong(this IBitfield obj) {}. Then, the ToLong() method will appear in Intellisense for IBitfield objects. – Phosphorism

Can you reverse the process by using 'f.SetValue(t,someValue)'? I'm using this to convert packet class to message buffers for socket transfers. Works great but I can't read data from the stream back to the structure using f.SetValue() for some reason. No errors, just doesn't work. – Anastice

GetFields may suffer from reordering due to reflection cache, but sorting by 'MetadataToken' should overcome that problem (which you can reproduce by mixing get single field and get all fields in some order). – Distillate

Answer 2

By using an enum you can do this, but will look awkward.

[Flags]
public enum PESHeaderFlags
{
    IsCopy = 1, // implied that if not present, then it is an original
    IsCopyrighted = 2,
    IsDataAligned = 4,
    Priority = 8,
    ScramblingControlType1 = 0,
    ScramblingControlType2 = 16,
    ScramblingControlType3 = 32,
    ScramblingControlType4 = 16+32,
    ScramblingControlFlags = ScramblingControlType1 | ScramblingControlType2 | ... ype4
    etc.
}

Hymenium

Answer 3

You want StructLayoutAttribute

[StructLayout(LayoutKind.Explicit, Size=1, CharSet=CharSet.Ansi)]
public struct Foo 
{ [FieldOffset(0)]public byte original_or_copy; 
  [FieldOffset(0)]public byte copyright;
  [FieldOffset(0)]public byte data_alignment_indicator; 
  [FieldOffset(0)]public byte PES_priority; 
  [FieldOffset(0)]public byte PES_scrambling_control; 
  [FieldOffset(0)]public byte reserved; 
}

This is really a union but you can use it as a bitfield--you just have to be conscious of where in the byte the bits for each field are supposed to be. Utility functions and/or constants to AND against can help.

const byte _original_or_copy = 1;
const byte _copyright        = 2;

//bool ooo = foo.original_or_copy();
static bool original_or_copy(this Foo foo) 
{ return  (foo.original_or_copy & _original_or_copy)  == original_or_copy;
}    

There is also LayoutKind.Sequential which will allow you to do it the C way.

Ariew

Answer 4

As Christophe Lambrechts suggested BitVector32 provides a solution. Jitted performance should be adequate, but don't know for sure. Here's the code illustrating this solution:

public struct rcSpan
{
    //C# Spec 10.4.5.1: The static field variable initializers of a class correspond to a sequence of assignments that are executed in the textual order in which they appear in the class declaration.
    internal static readonly BitVector32.Section sminSection = BitVector32.CreateSection(0x1FFF);
    internal static readonly BitVector32.Section smaxSection = BitVector32.CreateSection(0x1FFF, sminSection);
    internal static readonly BitVector32.Section areaSection = BitVector32.CreateSection(0x3F, smaxSection);

    internal BitVector32 data;

    //public uint smin : 13; 
    public uint smin
    {
        get { return (uint)data[sminSection]; }
        set { data[sminSection] = (int)value; }
    }

    //public uint smax : 13; 
    public uint smax
    {
        get { return (uint)data[smaxSection]; }
        set { data[smaxSection] = (int)value; }
    }

    //public uint area : 6; 
    public uint area
    {
        get { return (uint)data[areaSection]; }
        set { data[areaSection] = (int)value; }
    }
}

You can do a lot this way. You can do even better without using BitVector32, by providing handmade accessors for every field:

public struct rcSpan2
{
    internal uint data;

    //public uint smin : 13; 
    public uint smin
    {
        get { return data & 0x1FFF; }
        set { data = (data & ~0x1FFFu ) | (value & 0x1FFF); }
    }

    //public uint smax : 13; 
    public uint smax
    {
        get { return (data >> 13) & 0x1FFF; }
        set { data = (data & ~(0x1FFFu << 13)) | (value & 0x1FFF) << 13; }
    }

    //public uint area : 6; 
    public uint area
    {
        get { return (data >> 26) & 0x3F; }
        set { data = (data & ~(0x3F << 26)) | (value & 0x3F) << 26; }
    }
}

Surprisingly this last, handmade solution seems to be the most convenient, least convoluted, and the shortest one. That's of course only my personal preference.

Hellbent

Answer 5

One more based off of Zbyl's answer. This one is a little easier to change around for me - I just have to adjust the sz0,sz1... and also make sure mask# and loc# are correct in the Set/Get blocks.

Performance wise, it should be the same as they both resolved to 38 MSIL statements. (constants are resolved at compile time)

public struct MyStruct
{
    internal uint raw;

    const int sz0 = 4, loc0 = 0,          mask0 = ((1 << sz0) - 1) << loc0;
    const int sz1 = 4, loc1 = loc0 + sz0, mask1 = ((1 << sz1) - 1) << loc1;
    const int sz2 = 4, loc2 = loc1 + sz1, mask2 = ((1 << sz2) - 1) << loc2;
    const int sz3 = 4, loc3 = loc2 + sz2, mask3 = ((1 << sz3) - 1) << loc3;

    public uint Item0
    {
        get { return (uint)(raw & mask0) >> loc0; }
        set { raw = (uint)(raw & ~mask0 | (value << loc0) & mask0); }
    }

    public uint Item1
    {
        get { return (uint)(raw & mask1) >> loc1; }
        set { raw = (uint)(raw & ~mask1 | (value << loc1) & mask1); }
    }

    public uint Item2
    {
        get { return (uint)(raw & mask2) >> loc2; }
        set { raw = (uint)(raw & ~mask2 | (value << loc2) & mask2); }
    }

    public uint Item3
    {
        get { return (uint)((raw & mask3) >> loc3); }
        set { raw = (uint)(raw & ~mask3 | (value << loc3) & mask3); }
    }
}

Pour

Great setup. Reused with joy ;). I discovered that when the bitfield is "full" (e.g. when setting raw=uint.MaxValue) I did have to change the last item slightly. Or, maybe it regards the last property only. Not sure. So, for your example above, the ItemX property getters look like this: get { return (uint)((Raw & Mask3) >> Loc3); }. The setter look like this: set { Raw = (uint)(Raw & ~Mask3 | (value << Loc3) & Mask3); }` Without that change the casting fails for the last property. – Dobbin

@Spiralis: Thanks for noticing that. I updated it like you said and it now works better. – Pour

Answer 6

You could also use the BitVector32 and especially the Section struct. The example is very good.

Haik

Answer 7

I find myself quite comfortable with these helper functions:

uint SetBits(uint word, uint value, int pos, int size)
{
    uint mask = ((((uint)1) << size) - 1) << pos;
    word &= ~mask; //resettiamo le posizioni
    word |= (value << pos) & mask;
    return word;
}

uint ReadBits(uint word, int pos, int size)
{
    uint mask = ((((uint)1) << size) - 1) << pos;
    return (word & mask) >> pos;
}

then:

uint the_word;

public uint Itemx
{
    get { return ReadBits(the_word, 5, 2); }
    set { the_word = SetBits(the_word, value, 5, 2) }
}

Engulf

Straightforward and short solution ! – Swampland

Answer 8

While it is a class, using BitArray seems like the way to least reinvent the wheel. Unless you're really pressed for performance, this is the simplest option. (Indexes can be referenced with the [] operator.)

Abash

Answer 9

Could an Enum with the Flags Attribute help maybe? See here:

What does the [Flags] Enum Attribute mean in C#?

Harlie

Answer 10

A flags enum can work too, I think, if you make it a byte enum:

[Flags] enum PesHeaders : byte { /* ... */ }

Ariew

Answer 11

I wrote one, share it, may help someone:

[global::System.AttributeUsage(AttributeTargets.Field, AllowMultiple = false)]
public sealed class BitInfoAttribute : Attribute {
    byte length;
    public BitInfoAttribute(byte length) {
        this.length = length;
    }
    public byte Length { get { return length; } }
}

public abstract class BitField {

    public void parse<T>(T[] vals) {
        analysis().parse(this, ArrayConverter.convert<T, uint>(vals));
    }

    public byte[] toArray() {
        return ArrayConverter.convert<uint, byte>(analysis().toArray(this));
    }

    public T[] toArray<T>() {
        return ArrayConverter.convert<uint, T>(analysis().toArray(this));
    }

    static Dictionary<Type, BitTypeInfo> bitInfoMap = new Dictionary<Type, BitTypeInfo>();
    private BitTypeInfo analysis() {
        Type type = this.GetType();
        if (!bitInfoMap.ContainsKey(type)) {
            List<BitInfo> infos = new List<BitInfo>();

            byte dataIdx = 0, offset = 0;
            foreach (System.Reflection.FieldInfo f in type.GetFields()) {
                object[] attrs = f.GetCustomAttributes(typeof(BitInfoAttribute), false);
                if (attrs.Length == 1) {
                    byte bitLen = ((BitInfoAttribute)attrs[0]).Length;
                    if (offset + bitLen > 32) {
                        dataIdx++;
                        offset = 0;
                    }
                    infos.Add(new BitInfo(f, bitLen, dataIdx, offset));
                    offset += bitLen;
                }
            }
            bitInfoMap.Add(type, new BitTypeInfo(dataIdx + 1, infos.ToArray()));
        }
        return bitInfoMap[type];
    }
}

class BitTypeInfo {
    public int dataLen { get; private set; }
    public BitInfo[] bitInfos { get; private set; }

    public BitTypeInfo(int _dataLen, BitInfo[] _bitInfos) {
        dataLen = _dataLen;
        bitInfos = _bitInfos;
    }

    public uint[] toArray<T>(T obj) {
        uint[] datas = new uint[dataLen];
        foreach (BitInfo bif in bitInfos) {
            bif.encode(obj, datas);
        }
        return datas;
    }

    public void parse<T>(T obj, uint[] vals) {
        foreach (BitInfo bif in bitInfos) {
            bif.decode(obj, vals);
        }
    }
}

class BitInfo {

    private System.Reflection.FieldInfo field;
    private uint mask;
    private byte idx, offset, shiftA, shiftB;
    private bool isUnsigned = false;

    public BitInfo(System.Reflection.FieldInfo _field, byte _bitLen, byte _idx, byte _offset) {
        field = _field;
        mask = (uint)(((1 << _bitLen) - 1) << _offset);
        idx = _idx;
        offset = _offset;
        shiftA = (byte)(32 - _offset - _bitLen);
        shiftB = (byte)(32 - _bitLen);

        if (_field.FieldType == typeof(bool)
            || _field.FieldType == typeof(byte)
            || _field.FieldType == typeof(char)
            || _field.FieldType == typeof(uint)
            || _field.FieldType == typeof(ulong)
            || _field.FieldType == typeof(ushort)) {
            isUnsigned = true;
        }
    }

    public void encode(Object obj, uint[] datas) {
        if (isUnsigned) {
            uint val = (uint)Convert.ChangeType(field.GetValue(obj), typeof(uint));
            datas[idx] |= ((uint)(val << offset) & mask);
        } else {
            int val = (int)Convert.ChangeType(field.GetValue(obj), typeof(int));
            datas[idx] |= ((uint)(val << offset) & mask);
        }
    }

    public void decode(Object obj, uint[] datas) {
        if (isUnsigned) {
            field.SetValue(obj, Convert.ChangeType((((uint)(datas[idx] & mask)) << shiftA) >> shiftB, field.FieldType));
        } else {
            field.SetValue(obj, Convert.ChangeType((((int)(datas[idx] & mask)) << shiftA) >> shiftB, field.FieldType));
        }
    }
}

public class ArrayConverter {
    public static T[] convert<T>(uint[] val) {
        return convert<uint, T>(val);
    }

    public static T1[] convert<T0, T1>(T0[] val) {
        T1[] rt = null;
        // type is same or length is same
        // refer to https://mcmap.net/q/243708/-convert-byte-to-sbyte
        if (typeof(T0) == typeof(T1)) { 
            rt = (T1[])(Array)val;
        } else {
            int len = Buffer.ByteLength(val);
            int w = typeWidth<T1>();
            if (w == 1) { // bool
                rt = new T1[len * 8];
            } else if (w == 8) {
                rt = new T1[len];
            } else { // w > 8
                int nn = w / 8;
                int len2 = (len / nn) + ((len % nn) > 0 ? 1 : 0);
                rt = new T1[len2];
            }

            Buffer.BlockCopy(val, 0, rt, 0, len);
        }
        return rt;
    }

    public static string toBinary<T>(T[] vals) {
        StringBuilder sb = new StringBuilder();
        int width = typeWidth<T>();
        int len = Buffer.ByteLength(vals);
        for (int i = len-1; i >=0; i--) {
            sb.Append(Convert.ToString(Buffer.GetByte(vals, i), 2).PadLeft(8, '0')).Append(" ");
        }
        return sb.ToString();
    }

    private static int typeWidth<T>() {
        int rt = 0;
        if (typeof(T) == typeof(bool)) { // x
            rt = 1;
        } else if (typeof(T) == typeof(byte)) { // x
            rt = 8;
        } else if (typeof(T) == typeof(sbyte)) {
            rt = 8;
        } else if (typeof(T) == typeof(ushort)) { // x
            rt = 16;
        } else if (typeof(T) == typeof(short)) {
            rt = 16;
        } else if (typeof(T) == typeof(char)) {
            rt = 16;
        } else if (typeof(T) == typeof(uint)) { // x
            rt = 32;
        } else if (typeof(T) == typeof(int)) {
            rt = 32;
        } else if (typeof(T) == typeof(float)) {
            rt = 32;
        } else if (typeof(T) == typeof(ulong)) { // x
            rt = 64;
        } else if (typeof(T) == typeof(long)) {
            rt = 64;
        } else if (typeof(T) == typeof(double)) {
            rt = 64;
        } else {
            throw new Exception("Unsupport type : " + typeof(T).Name);
        }
        return rt;
    }
}

and the usage:

class MyTest01 : BitField {
    [BitInfo(3)]
    public bool d0;
    [BitInfo(3)]
    public short d1;
    [BitInfo(3)]
    public int d2;
    [BitInfo(3)]
    public int d3;
    [BitInfo(3)]
    public int d4;
    [BitInfo(3)]
    public int d5;

    public MyTest01(bool _d0, short _d1, int _d2, int _d3, int _d4, int _d5) {
        d0 = _d0;
        d1 = _d1;
        d2 = _d2;
        d3 = _d3;
        d4 = _d4;
        d5 = _d5;
    }

    public MyTest01(byte[] datas) {
        parse(datas);
    }

    public new string ToString() {
        return string.Format("d0: {0}, d1: {1}, d2: {2}, d3: {3}, d4: {4}, d5: {5} \r\nbinary => {6}",
            d0, d1, d2, d3, d4, d5, ArrayConverter.toBinary(toArray()));
    }
};

class MyTest02 : BitField {
    [BitInfo(5)]
    public bool val0;
    [BitInfo(5)]
    public byte val1;
    [BitInfo(15)]
    public uint val2;
    [BitInfo(15)]
    public float val3;
    [BitInfo(15)]
    public int val4;
    [BitInfo(15)]
    public int val5;
    [BitInfo(15)]
    public int val6;

    public MyTest02(bool v0, byte v1, uint v2, float v3, int v4, int v5, int v6) {
        val0 = v0;
        val1 = v1;
        val2 = v2;
        val3 = v3;
        val4 = v4;
        val5 = v5;
        val6 = v6;
    }

    public MyTest02(byte[] datas) {
        parse(datas);
    }

    public new string ToString() {
        return string.Format("val0: {0}, val1: {1}, val2: {2}, val3: {3}, val4: {4}, val5: {5}, val6: {6}\r\nbinary => {7}",
            val0, val1, val2, val3, val4, val5, val6, ArrayConverter.toBinary(toArray()));
    }
}

public class MainClass {

    public static void Main(string[] args) {
        MyTest01 p = new MyTest01(false, 1, 2, 3, -1, -2);
        Debug.Log("P:: " + p.ToString());
        MyTest01 p2 = new MyTest01(p.toArray());
        Debug.Log("P2:: " + p2.ToString());

        MyTest02 t = new MyTest02(true, 1, 12, -1.3f, 4, -5, 100);
        Debug.Log("t:: " + t.ToString());
        MyTest02 t2 = new MyTest02(t.toArray());
        Debug.Log("t:: " + t.ToString());

        Console.Read();
        return;
    }
}

Mcbride

Answer 12

I wrote one this morning with T4. :) Same example as Zbyl, though I threw in a bit of uint sizing fun. This is just a first pass, it could obviously use a little error checking. Also the bitFields spec array would be nicer in a separate file, maybe a .ttinclude, or a json/yaml..

=== BitFields.tt ===

<#@ template language="C#" #>
<#@ assembly name="System.Core" #>
<#@ import namespace="System.Linq" #>

<#
var bitFields = new[]
{
    new
    {
        Name = "rcSpan2", Fields = new[] { ("smin", 13), ("smax", 13), ("area", 6) },
    },  
};

foreach (var bitField in bitFields)
{
    static string getType(int size) =>
        size switch
        {
            > 32 => "ulong",
            > 16 => "uint",
            > 8 => "ushort",
            _ => "byte",
        };

    var bitFieldType = getType(bitField.Fields.Sum(f => f.Item2)); 
#>
public struct <#=bitField.Name#>
{
    <#=bitFieldType#> _bitfield;

<#
var offset = 0;
foreach (var (fieldName, fieldSize) in bitField.Fields)
{
    var fieldType = getType(fieldSize);
    var fieldMask = $"0x{((1UL<<fieldSize)-1):X}U";
#>
    public <#=fieldType#> <#=fieldName#> // : <#=fieldSize#>
    {
        get => (<#=fieldType#>)(<#=offset > 0 ? $"(_bitfield >> {offset})" : "_bitfield"#> & <#=fieldMask#>);
        set => _bitfield = (<#=bitFieldType#>)((_bitfield & ~((<#=bitFieldType#>)<#=fieldMask#> << <#=offset#>)) | ((<#=bitFieldType#>)(value & <#=fieldMask#>) << <#=offset#>));
    }  
<#
    offset += fieldSize;
}
#>
}

<#}#>

=== BitFields.cs === (generated)

public struct rcSpan2
{
    uint _bitfield;

    public ushort smin // : 13
    {
        get => (ushort)(_bitfield & 0x1FFFU);
        set => _bitfield = (uint)((_bitfield & ~((uint)0x1FFFU << 0)) | ((uint)(value & 0x1FFFU) << 0));
    }  
    public ushort smax // : 13
    {
        get => (ushort)((_bitfield >> 13) & 0x1FFFU);
        set => _bitfield = (uint)((_bitfield & ~((uint)0x1FFFU << 13)) | ((uint)(value & 0x1FFFU) << 13));
    }  
    public byte area // : 6
    {
        get => (byte)((_bitfield >> 26) & 0x3FU);
        set => _bitfield = (uint)((_bitfield & ~((uint)0x3FU << 26)) | ((uint)(value & 0x3FU) << 26));
    }  
}

Caulicle