Encoding.UTF8.GetString doesn't take into account the Preamble/BOM

Question

In .NET, I'm trying to use Encoding.UTF8.GetString method, which takes a byte array and converts it to a string.

It looks like this method ignores the BOM (Byte Order Mark), which might be a part of a legitimate binary representation of a UTF8 string, and takes it as a character.

I know I can use a TextReader to digest the BOM as needed, but I thought that the GetString method should be some kind of a macro that makes our code shorter.

Am I missing something? Is this like so intentionally?

Here's a reproduction code:

static void Main(string[] args)
{
    string s1 = "abc";
    byte[] abcWithBom;
    using (var ms = new MemoryStream())
    using (var sw = new StreamWriter(ms, new UTF8Encoding(true)))
    {
        sw.Write(s1);
        sw.Flush();
        abcWithBom = ms.ToArray();
        Console.WriteLine(FormatArray(abcWithBom)); // ef, bb, bf, 61, 62, 63
    }

    byte[] abcWithoutBom;
    using (var ms = new MemoryStream())
    using (var sw = new StreamWriter(ms, new UTF8Encoding(false)))
    {
        sw.Write(s1);
        sw.Flush();
        abcWithoutBom = ms.ToArray();
        Console.WriteLine(FormatArray(abcWithoutBom)); // 61, 62, 63
    }

    var restore1 = Encoding.UTF8.GetString(abcWithoutBom);
    Console.WriteLine(restore1.Length); // 3
    Console.WriteLine(restore1); // abc

    var restore2 = Encoding.UTF8.GetString(abcWithBom);
    Console.WriteLine(restore2.Length); // 4 (!)
    Console.WriteLine(restore2); // ?abc
}

private static string FormatArray(byte[] bytes1)
{
    return string.Join(", ", from b in bytes1 select b.ToString("x"));
}

Hyperthermia

Answer 1

It looks like this method ignores the BOM (Byte Order Mark), which might be a part of a legitimate binary representation of a UTF8 string, and takes it as a character.

It doesn't look like it "ignores" it at all - it faithfully converts it to the BOM character. That's what it is, after all.

If you want to make your code ignore the BOM in any string it converts, that's up to you to do... or use StreamReader.

Note that if you either use Encoding.GetBytes followed by Encoding.GetString or use StreamWriter followed by StreamReader, both forms will either produce then swallow or not produce the BOM. It's only when you mix using a StreamWriter (which uses Encoding.GetPreamble) with a direct Encoding.GetString call that you end up with the "extra" character.

Schwarz

@RonKlein Additionally, you could say restore2 = restore2.TrimStart('\uFEFF') to remove leading BOM characters. I have also at one time wondered why (new UTF8Encoding(true)).GetBytes("abc") and (new UTF8Encoding(false)).GetBytes("abc") produce the same output, but as you probably know by now, GetBytes does not assume you're in the beginning of a file, so it never uses GetPreamble. You have to GetPreamble explicitly, or skip the preamble explicitly, if you use GetBytes, or GetString. – Whomp

Answer 2

Based on the answer by Jon Skeet (thanks!), this is how I just did it:

var memoryStream = new MemoryStream(byteArray);
var s = new StreamReader(memoryStream).ReadToEnd();

Note that this will probably only work reliably if there is a BOM in the byte array you are reading from. If not, you might want to look into another StreamReader constructor overload which takes an Encoding parameter so you can tell it what the byte array contains.

Algophobia

I think you might want this constructor overload instead which lets you specify whether it should look for a BOM to figure out encoding. – Prescind

Answer 3

for those who do not want to use streams I found a quite simple solution using Linq:

public static string GetStringExcludeBOMPreamble(this Encoding encoding, byte[] bytes)
{
    var preamble = encoding.GetPreamble();
    if (preamble?.Length > 0 && bytes.Length >= preamble.Length && bytes.Take(preamble.Length).SequenceEqual(preamble))
    {
        return encoding.GetString(bytes, preamble.Length, bytes.Length - preamble.Length);
    }
    else
    {
        return encoding.GetString(bytes);
    }
}

Perren

Answer 4

I know I am kind of late to the party but here's the code I am using (feel free to adapt to C#) if you need:

Public Function Serialize(Of YourXMLClass)(ByVal obj As YourXMLClass,
                                                      Optional ByVal omitXMLDeclaration As Boolean = True,
                                                      Optional ByVal omitXMLNamespace As Boolean = True) As String

    Dim serializer As New XmlSerializer(obj.GetType)
    Using memStream As New MemoryStream()
        Dim settings As New XmlWriterSettings() With {
                    .Encoding = Encoding.UTF8,
                    .Indent = True,
                    .omitXMLDeclaration = omitXMLDeclaration}

        Using writer As XmlWriter = XmlWriter.Create(memStream, settings)
            Dim xns As New XmlSerializerNamespaces
            If (omitXMLNamespace) Then xns.Add("", "")
            serializer.Serialize(writer, obj, xns)
        End Using

        Return Encoding.UTF8.GetString(memStream.ToArray())
    End Using
End Function

Public Function Deserialize(Of YourXMLClass)(ByVal obj As YourXMLClass, ByVal xml As String) As YourXMLClass
    Dim result As YourXMLClass
    Dim serializer As New XmlSerializer(GetType(YourXMLClass))

    Using memStream As New MemoryStream()
        Dim bytes As Byte() = Encoding.UTF8.GetBytes(xml.ToArray)
        memStream.Write(bytes, 0, bytes.Count)
        memStream.Seek(0, SeekOrigin.Begin)

        Using reader As XmlReader = XmlReader.Create(memStream)
            result = DirectCast(serializer.Deserialize(reader), YourXMLClass)
        End Using

    End Using
    Return result
End Function

Dodecahedron