C# - Binary reader in Big Endian?

Question

I'm trying to improve my understanding of the STFS file format by using a program to read all the different bits of information. Using a website with a reference of which offsets contain what information, I wrote some code that has a binary reader go through the file and place the values in the correct variables.

The problem is that all the data is SUPPOSED to be Big Endian, and everything the binary reader read is Little Endian. So, what's the best way to go about fixing this?

Can I create a mimic class of Binary reader that returns a reversed array of bytes? Is there something I can change in class instance that will make it read in big endian so I don't have to rewrite everything?

Any help is appreciated.

edit: I tried adding Encoding.BigEndianUnicode as a parameter, but it still reads little endian.

Byars

@HansPassant, Would this be one of those dlls that require me to make my code open source? Why do some dlls require that? – Byars

Walkerneo I deleted my answer because zmbq answered essentially the same thing 3 minutes before me. The concept of endianness does not apply to byte arrays, only to words, dwords, qwords, etc., that is to groups of 2, 4, 8 and so on bytes. I am sorry if it would mean changing a lot of code, but a man has to do what a man has to do. – Consecutive

Skeet sells books, the code has few strings attached. Check the license section on that page. Apache terms are here: apache.org/licenses/LICENSE-2.0.html – Salute

If what you are concerned about is extracting words, dwords, qwords etc. AND converting them to the proper endianness in one step, then this question has been answered elsewhere: #1674660 – Consecutive

@MikeNakis, Oh yeah, you're right about the byte arrays. I'm still learning :D – Byars

@HansPassant, His Binary Reader doesn't have all the methods of the system's binary reader.. – Byars

@HansPassant, Thanks, but I answered my own question :D – Byars

Well, there you go. Good programmers spin miracles in 21 minutes or less :) – Salute

Answer 1

I'm not usually one to answer my own questions, but I've accomplished exactly what I wanted with some simple code:

class BinaryReader2 : BinaryReader { 
    public BinaryReader2(System.IO.Stream stream)  : base(stream) { }

    public override int ReadInt32()
    {
        var data = base.ReadBytes(4);
        Array.Reverse(data);
        return BitConverter.ToInt32(data, 0);
    }

    public Int16 ReadInt16()
    {
        var data = base.ReadBytes(2);
        Array.Reverse(data);
        return BitConverter.ToInt16(data, 0);
    }

    public Int64 ReadInt64()
    {
        var data = base.ReadBytes(8);
        Array.Reverse(data);
        return BitConverter.ToInt64(data, 0);
    }

    public UInt32 ReadUInt32()
    {
        var data = base.ReadBytes(4);
        Array.Reverse(data);
        return BitConverter.ToUInt32(data, 0);
    }

}

I knew that's what I wanted, but I didn't know how to write it. I found this page and it helped: http://www.codekeep.net/snippets/870c4ab3-419b-4dd2-a950-6d45beaf1295.aspx

Byars

Off-topic, but your class's fields (a16 etc) are unnecessary. You assign an array to them during construction, but within each method you replace that array with a new array returned by the Read function. You could just put var a32 = base.ReadBytes... in each method and get rid of the fields. – Bearish

They're not unnecessary, they're harmful. Turning what is (potentially, ignoring the share underlying stream) a thread-safe code into a shared state situation. – Mispronounce

You probably want to check BitConverter.IsLittleEndian before reversing. If it is false you don't need to reverse. – Wonderstricken

@JoãoPortela that depends upon the expect endianess of the source data! :D – Skye

@Skye Yes. You have to know the endianess of the source data and the endianess of BitConverter, if they don't match: reverse it. – Wonderstricken

Allocating an array for each read creates work for the GC. You might just call GetByte as many times as required, then shift/OR the bytes into place. – Mechanician

Answer 2

IMHO a slightly better answer as it doesn't require a different class to be newed-up, makes the big-endian calls obvious and allows big- and little-endian calls to be mixed in the stream.

public static class Helpers
{
  // Note this MODIFIES THE GIVEN ARRAY then returns a reference to the modified array.
  public static byte[] Reverse(this byte[] b)
  {
    Array.Reverse(b);
    return b;
  }

  public static UInt16 ReadUInt16BE(this BinaryReader binRdr)
  {
    return BitConverter.ToUInt16(binRdr.ReadBytesRequired(sizeof(UInt16)).Reverse(), 0);
  }

  public static Int16 ReadInt16BE(this BinaryReader binRdr)
  {
    return BitConverter.ToInt16(binRdr.ReadBytesRequired(sizeof(Int16)).Reverse(), 0);
  }

  public static UInt32 ReadUInt32BE(this BinaryReader binRdr)
  {
    return BitConverter.ToUInt32(binRdr.ReadBytesRequired(sizeof(UInt32)).Reverse(), 0);
  }

  public static Int32 ReadInt32BE(this BinaryReader binRdr)
  {
    return BitConverter.ToInt32(binRdr.ReadBytesRequired(sizeof(Int32)).Reverse(), 0);
  }

  public static byte[] ReadBytesRequired(this BinaryReader binRdr, int byteCount)
  {
    var result = binRdr.ReadBytes(byteCount);

    if (result.Length != byteCount)
      throw new EndOfStreamException(string.Format("{0} bytes required from stream, but only {1} returned.", byteCount, result.Length));

    return result;
  }
}

Dentilingual

Remember to check BitConverter.IsLittleEndian before reversing. – Wonderstricken

looks like you need ".ToArray()" after the Reverse, since Reverse returns IEnumerable<byte> and not byte[] (which is what the BitConverter expects) – Figwort

Since .NET Core, there is also a BinaryPrimitives class making this obsolete: learn.microsoft.com/en-us/dotnet/api/… – Germanium

@JoãoPortela Actually, that's not necessary. BinaryReader always reads in little-endian, as you can see in the source code – it actually just calls BinaryPrimitives.ReadWhateverLittleEndian. Kind of wish they'd include big-endian versions by default… seems like it would be pretty easy. – Slapjack

A much shorter and probably faster version of this code would just use BinaryPrimitives.ReverseEndianness. – Slapjack

@MorganHarris yes. But that check is for the bitconverter. If this code is run in an arquitecture with different endianess, the assumptions of how BitConverter works would no longer hold. – Wonderstricken

@JoãoPortela you know what, I didn't even see the BitConverter there 🤦‍♂️ Yeah, just don't use it IMO. BinaryPrimitives is the correct tool here. – Slapjack

Like @Germanium mentioned in a previous comment, we now have BinaryPrimitives for this. This is a very old answer and it wasn't available at the time. – Wonderstricken

Answer 3

A mostly-complete (for my purposes) drop-in replacement for BinaryReader that handles endianness correctly, unlike most of these answers. By default it works exactly like BinaryReader, but can be constructed to read in the required endianness. Additionally the Read<Primitive> methods are overloaded to allow you to specify the endianness to read a particular value in - useful in the (unlikely) scenario that you're dealing with a stream of mixed LE/BE data.

public class EndiannessAwareBinaryReader : BinaryReader
{
    public enum Endianness
    {
        Little,
        Big,
    }

    private readonly Endianness _endianness = Endianness.Little;

    public EndiannessAwareBinaryReader(Stream input) : base(input)
    {
    }

    public EndiannessAwareBinaryReader(Stream input, Encoding encoding) : base(input, encoding)
    {
    }

    public EndiannessAwareBinaryReader(Stream input, Encoding encoding, bool leaveOpen) : base(input, encoding, leaveOpen)
    {
    }

    public EndiannessAwareBinaryReader(Stream input, Endianness endianness) : base(input)
    {
        _endianness = endianness;
    }

    public EndiannessAwareBinaryReader(Stream input, Encoding encoding, Endianness endianness) : base(input, encoding)
    {
        _endianness = endianness;
    }

    public EndiannessAwareBinaryReader(Stream input, Encoding encoding, bool leaveOpen, Endianness endianness) : base(input, encoding, leaveOpen)
    {
        _endianness = endianness;
    }

    public override short ReadInt16() => ReadInt16(_endianness);

    public override int ReadInt32() => ReadInt32(_endianness);

    public override long ReadInt64() => ReadInt64(_endianness);

    public override ushort ReadUInt16() => ReadUInt16(_endianness);

    public override uint ReadUInt32() => ReadUInt32(_endianness);

    public override ulong ReadUInt64() => ReadUInt64(_endianness);

    public short ReadInt16(Endianness endianness) => BitConverter.ToInt16(ReadForEndianness(sizeof(short), endianness));

    public int ReadInt32(Endianness endianness) => BitConverter.ToInt32(ReadForEndianness(sizeof(int), endianness));

    public long ReadInt64(Endianness endianness) => BitConverter.ToInt64(ReadForEndianness(sizeof(long), endianness));

    public ushort ReadUInt16(Endianness endianness) => BitConverter.ToUInt16(ReadForEndianness(sizeof(ushort), endianness));

    public uint ReadUInt32(Endianness endianness) => BitConverter.ToUInt32(ReadForEndianness(sizeof(uint), endianness));

    public ulong ReadUInt64(Endianness endianness) => BitConverter.ToUInt64(ReadForEndianness(sizeof(ulong), endianness));

    private byte[] ReadForEndianness(int bytesToRead, Endianness endianness)
    {
        var bytesRead = ReadBytes(bytesToRead);

        if ((endianness == Endianness.Little && !BitConverter.IsLittleEndian)
            || (endianness == Endianness.Big && BitConverter.IsLittleEndian))
        {
            Array.Reverse(bytesRead);
        }

        return bytesRead;
    }
}

Mccraw

Best solution, handles host system endianness as well as source data endianness and only reverses data when it has to be reversed. – Educative

Awesome solution, however the BitConverter methods needed an extra parameter of startIndex appended: public short ReadInt16(Endianness endianness) => BitConverter.ToInt16(ReadForEndianness(sizeof(short), endianness), 0); – Lugsail

@PeterWilson Are you trying to use this in .NET Framework? – Mccraw

that clearly works, but not so effective, as CPU do have single instruction for endianness conversion, instead of array manipulation. bswap – Branen

Great solution, in case you are using .NET Core 2.1+, you can use this version that uses BinaryPrimitives, so the reversing is handled by the framework and it is more performant according to Stephen Toub devblogs.microsoft.com/dotnet/…. – Ackerman

Answer 4

I'm not familiar with STFS, but changing endianess is relatively easy. "Network Order" is big endian, so all you need to do is translate from network to host order.

This is easy because there's already code that does that. Look at IPAddress.NetworkToHostOrder, as explained here: ntohs() and ntohl() equivalent?

Shang

Answer 5

In my opinion, you want to be careful doing this. The reason one would want to Convert from BigEndian to LittleEndian is if the bytes being read are in BigEndian and the OS calculating against them is operating in LittleEndian.

C# isn't a window only language anymore. With ports like Mono, and also other Microsoft Platforms like Windows Phone 7/8, Xbox 360/Xbox One, Windwos CE, Windows 8 Mobile, Linux With MONO, Apple with MONO, etc. It is quite possible the operating platform could be in BigEndian, in which case you'd be screwing yourself if you converted the code without doing any checks.

The BitConverter already has a field on it called "IsLittleEndian" you can use this to determine if the operating environment is in LittleEndian or not. Then you can do the reversing conditionally.

As such, I actually just wrote some byte[] extensions instead of making a big class:

    /// <summary>
    /// Get's a byte array from a point in a source byte array and reverses the bytes. Note, if the current platform is not in LittleEndian the input array is assumed to be BigEndian and the bytes are not returned in reverse order
    /// </summary>
    /// <param name="byteArray">The source array to get reversed bytes for</param>
    /// <param name="startIndex">The index in the source array at which to begin the reverse</param>
    /// <param name="count">The number of bytes to reverse</param>
    /// <returns>A new array containing the reversed bytes, or a sub set of the array not reversed.</returns>
    public static byte[] ReverseForBigEndian(this byte[] byteArray, int startIndex, int count)
    {
        if (BitConverter.IsLittleEndian)
            return byteArray.Reverse(startIndex, count);
        else
            return byteArray.SubArray(startIndex, count);

    }

    public static byte[] Reverse(this byte[] byteArray, int startIndex, int count)
    {
        byte[] ret = new byte[count];
        for (int i = startIndex + (count - 1); i >= startIndex; --i)
        {
            byte b = byteArray[i];
            ret[(startIndex + (count - 1)) - i] = b;
        }
        return ret;
    }

    public static byte[] SubArray(this byte[] byteArray, int startIndex, int count)
    {
        byte[] ret = new byte[count];
        for (int i = 0; i < count; ++i)            
            ret[0] = byteArray[i + startIndex];
        return ret;
    }

So imagine this example code:

byte[] fontBytes = byte[240000]; //some data loaded in here, E.G. a TTF TrueTypeCollection font file. (which is in BigEndian)

int _ttcVersionMajor = BitConverter.ToUint16(fontBytes.ReverseForBigEndian(4, 2), 0);

//output
_ttcVersionMajor = 1 //TCCHeader is version 1

Pore

Answer 6

You better to use BinaryPrimitives class

        public override double ReadDouble()
        {
            return BinaryPrimitives.ReadDoubleBigEndian(ReadBytes(8));
        }

        public override short ReadInt16()
        {
            return BinaryPrimitives.ReadInt16BigEndian(ReadBytes(2));
        }

        public override int ReadInt32()
        {
            return BinaryPrimitives.ReadInt32BigEndian(ReadBytes(4));
        }

        public override long ReadInt64()
        {
            return BinaryPrimitives.ReadInt64BigEndian(ReadBytes(8));
        }

        public override float ReadSingle()
        {
            return BinaryPrimitives.ReadSingleBigEndian(ReadBytes(4));
        }

        public override ushort ReadUInt16()
        {
            return BinaryPrimitives.ReadUInt16BigEndian(ReadBytes(2));
        }

        public override uint ReadUInt32()
        {
            return BinaryPrimitives.ReadUInt32BigEndian(ReadBytes(4));
        }

        public override ulong ReadUInt64()
        {
            return BinaryPrimitives.ReadUInt64BigEndian(ReadBytes(8));
        }

Label

Answer 7

I've expanded on Ian Kemp's excellent suggestion, I'm using the new BinaryPrimitives, available in .NET Core 2.1+, they are more performant according to Stephen Toub's post and can handle the endianness and reversal internally.

So if you are running .NET Core 2.1+ you should definitely use this version:

public class EndiannessAwareBinaryReader : BinaryReader
{
    public enum Endianness
    {
        Little,
        Big,
    }

    private readonly Endianness _endianness = Endianness.Little;

    public EndiannessAwareBinaryReader(Stream input) : base(input)
    {
    }

    public EndiannessAwareBinaryReader(Stream input, Encoding encoding) : base(input, encoding)
    {
    }

    public EndiannessAwareBinaryReader(Stream input, Encoding encoding, bool leaveOpen) : base(
        input, encoding, leaveOpen)
    {
    }

    public EndiannessAwareBinaryReader(Stream input, Endianness endianness) : base(input)
    {
        _endianness = endianness;
    }

    public EndiannessAwareBinaryReader(Stream input, Encoding encoding, Endianness endianness) :
        base(input, encoding)
    {
        _endianness = endianness;
    }

    public EndiannessAwareBinaryReader(Stream input, Encoding encoding, bool leaveOpen,
        Endianness endianness) : base(input, encoding, leaveOpen)
    {
        _endianness = endianness;
    }

    public override short ReadInt16() => ReadInt16(_endianness);

    public override int ReadInt32() => ReadInt32(_endianness);

    public override long ReadInt64() => ReadInt64(_endianness);

    public override ushort ReadUInt16() => ReadUInt16(_endianness);

    public override uint ReadUInt32() => ReadUInt32(_endianness);

    public override ulong ReadUInt64() => ReadUInt64(_endianness);

    public short ReadInt16(Endianness endianness) => endianness == Endianness.Little
        ? BinaryPrimitives.ReadInt16LittleEndian(ReadBytes(sizeof(short)))
        : BinaryPrimitives.ReadInt16BigEndian(ReadBytes(sizeof(short)));

    public int ReadInt32(Endianness endianness) => endianness == Endianness.Little
        ? BinaryPrimitives.ReadInt32LittleEndian(ReadBytes(sizeof(int)))
        : BinaryPrimitives.ReadInt32BigEndian(ReadBytes(sizeof(int)));

    public long ReadInt64(Endianness endianness) => endianness == Endianness.Little
        ? BinaryPrimitives.ReadInt64LittleEndian(ReadBytes(sizeof(long)))
        : BinaryPrimitives.ReadInt64BigEndian(ReadBytes(sizeof(long)));

    public ushort ReadUInt16(Endianness endianness) => endianness == Endianness.Little
        ? BinaryPrimitives.ReadUInt16LittleEndian(ReadBytes(sizeof(ushort)))
        : BinaryPrimitives.ReadUInt16BigEndian(ReadBytes(sizeof(ushort)));

    public uint ReadUInt32(Endianness endianness) => endianness == Endianness.Little
        ? BinaryPrimitives.ReadUInt32LittleEndian(ReadBytes(sizeof(uint)))
        : BinaryPrimitives.ReadUInt32BigEndian(ReadBytes(sizeof(uint)));

    public ulong ReadUInt64(Endianness endianness) => endianness == Endianness.Little
        ? BinaryPrimitives.ReadUInt64LittleEndian(ReadBytes(sizeof(ulong)))
        : BinaryPrimitives.ReadUInt64BigEndian(ReadBytes(sizeof(ulong)));
}

Ackerman

Answer 8

You might like this option, inspired by some of the other answers but written generically so as to make for a more concise implementation that allows us to include all the variations without too much extra code.

public static class BinaryReaderEndianExtensions
{
    public static T ReadNativeEndian<T>(this BinaryReader reader, Func<byte[], T> converter) =>
        converter(reader.ReadBytesRequired(Marshal.SizeOf<T>()));
    public static T ReadForeignEndian<T>(this BinaryReader reader, Func<byte[], T> converter) =>
        converter(reader.ReadBytesRequired(Marshal.SizeOf<T>()).Reverse().ToArray());

    public static T ReadBigEndian<T>(this BinaryReader reader, Func<byte[], T> converter) =>
        BitConverter.IsLittleEndian ? reader.ReadForeignEndian(converter) : reader.ReadNativeEndian(converter);
    public static T ReadLittleEndian<T>(this BinaryReader reader, Func<byte[], T> converter) =>
        BitConverter.IsLittleEndian ? reader.ReadNativeEndian(converter) : reader.ReadForeignEndian(converter);

    public static UInt16 ReadUInt16BE(this BinaryReader reader) => reader.ReadBigEndian(bytes => BitConverter.ToUInt16(bytes));
    public static Int16 ReadInt16BE(this BinaryReader reader) => reader.ReadBigEndian(bytes => BitConverter.ToInt16(bytes));
    public static UInt32 ReadUInt32BE(this BinaryReader reader) => reader.ReadBigEndian(bytes => BitConverter.ToUInt32(bytes));
    public static Int32 ReadInt32BE(this BinaryReader reader) => reader.ReadBigEndian(bytes => BitConverter.ToInt32(bytes));
    public static UInt64 ReadUInt64BE(this BinaryReader reader) => reader.ReadBigEndian(bytes => BitConverter.ToUInt64(bytes));
    public static Int64 ReadInt64BE(this BinaryReader reader) => reader.ReadBigEndian(bytes => BitConverter.ToInt64(bytes));
    public static Double ReadDoubleBE(this BinaryReader reader) => reader.ReadBigEndian(bytes => BitConverter.ToDouble(bytes));
    public static Single ReadSingleBE(this BinaryReader reader) => reader.ReadBigEndian(bytes => BitConverter.ToSingle(bytes));
    public static Half ReadHalfBE(this BinaryReader reader) => reader.ReadBigEndian(bytes => BitConverter.ToHalf(bytes));

    public static UInt16 ReadUInt16LE(this BinaryReader reader) => reader.ReadLittleEndian(bytes => BitConverter.ToUInt16(bytes));
    public static Int16 ReadInt16LE(this BinaryReader reader) => reader.ReadLittleEndian(bytes => BitConverter.ToInt16(bytes));
    public static UInt32 ReadUInt32LE(this BinaryReader reader) => reader.ReadLittleEndian(bytes => BitConverter.ToUInt32(bytes));
    public static Int32 ReadInt32LE(this BinaryReader reader) => reader.ReadLittleEndian(bytes => BitConverter.ToInt32(bytes));
    public static UInt64 ReadUInt64LE(this BinaryReader reader) => reader.ReadLittleEndian(bytes => BitConverter.ToUInt64(bytes));
    public static Int64 ReadInt64LE(this BinaryReader reader) => reader.ReadLittleEndian(bytes => BitConverter.ToInt64(bytes));
    public static Double ReadDoubleLE(this BinaryReader reader) => reader.ReadLittleEndian(bytes => BitConverter.ToDouble(bytes));
    public static Single ReadSingleLE(this BinaryReader reader) => reader.ReadLittleEndian(bytes => BitConverter.ToSingle(bytes));
    public static Half ReadHalfLE(this BinaryReader reader) => reader.ReadLittleEndian(bytes => BitConverter.ToHalf(bytes));

    public static byte[] ReadBytesRequired(this BinaryReader reader, int count)
    {
        byte[] bytes = reader.ReadBytes(count);
        return bytes.Length == count ? bytes
            : throw new EndOfStreamException(string.Format("{0} bytes required from stream, but only {1} returned.", count, bytes.Length));
    }
}

Cloven