Convert Unicode surrogate pair to literal string
Asked Answered
E

2

19

I am trying to read a high Unicode character from one string into another. For brevity, I will simplify my code as shown below:

public static void UnicodeTest()
{
    var highUnicodeChar = "๐€"; //Not the standard A

    var result1 = highUnicodeChar; //this works
    var result2 = highUnicodeChar[0].ToString(); // returns \ud835
}

When I assign highUnicodeChar to result1 directly, it retains its literal value of ๐€. When I try to access it by index, it returns \ud835. As I understand it, this is a surrogate pair of UTF-16 characters used to represent a UTF-32 character. I am pretty sure this problem has to do with trying to implicitly convert a char to a string.

In the end, I want result2 to yield the same value as result1. How can I do this?

Endaendall answered 1/10, 2018 at 3:42 Comment(0)
G
33

In Unicode, you have code points. These are 21 bits long. Your character ๐€, Mathematical Bold Capital A, has a code point of U+1D400.

In Unicode encodings, you have code units. These are the natural unit of the encoding: 8-bit for UTF-8, 16-bit for UTF-16, and so on. One or more code units encode a single code point.

In UTF-16, two code units that form a single code point are called a surrogate pair. Surrogate pairs are used to encode any code point greater than 16 bits, i.e. U+10000 and up.

This gets a little tricky in .NET, as a .NET Char represents a single UTF-16 code unit, and a .NET String is a collection of code units.

So your code point ๐€ (U+1D400) can't fit in 16 bits and needs a surrogate pair, meaning your string has two code units in it:

var highUnicodeChar = "๐€";
char a = highUnicodeChar[0]; // code unit 0xD835
char b = highUnicodeChar[1]; // code unit 0xDC00

Meaning when you index into the string like that, you're actually only getting half of the surrogate pair.

You can use IsSurrogatePair to test for a surrogate pair. For instance:

string GetFullCodePointAtIndex(string s, int idx) =>
    s.Substring(idx, char.IsSurrogatePair(s, idx) ? 2 : 1);

Important to note that the rabbit hole of variable encoding in Unicode doesn't end at the code point. A grapheme cluster is the "visible thing" most people when asked would ultimately call a "character". A grapheme cluster is made from one or more code points: a base character, and zero or more combining characters. An example of a combining character is an umlaut or various other decorations/modifiers you might want to add. See this answer for a horrifying example of what combining characters can do.

To test for a combining character, you can use GetUnicodeCategory to check for an enclosing mark, non-spacing mark, or spacing mark.

Glottis answered 1/10, 2018 at 4:8 Comment(3)
An example of "zero or more combining characters" can be seen on #1732848 and there are tools to generate this, like lingojam.com/GlitchTextGenerator โ€“ Skyscraper
โ€œcode points are 21 bits longโ€ โ€“ though it's true that 21 bits is the amount of data with which you can represent any code point, it's not really practically very meaningful to say that this is the โ€œlength of a code pointโ€. I don't think such a representation is used anywhere important; for direct access of codepoints you'd actually use UTF-32 or perhaps store the code points in 64 or even 128 bits for reasons of memory uniformity. โ€” Also: umlauts are usually not implemented as combining characters, since most combinations already have a single code point allocated to them. โ€“ Countdown
@Countdown when identifying the distinction between Unicode and one of its encoding, I find the concept of "21 bits" to be a good way to break people loose of the sorta-but-not-really correct "UTF32=codepoint" idea. โ€“ Glottis
H
10

It appears that you want to extract the first "atomic" character from the user point of view (i.e. the first Unicode grapheme cluster) from the highUnicodeChar string, where an "atomic" character includes both halves of a surrogate pair.

You can use StringInfo.GetTextElementEnumerator() to do just this, breaking a string down into atomic chunks then taking the first.

First, define the following extension method:

public static class TextExtensions
{
    public static IEnumerable<string> TextElements(this string s)
    {
        // StringInfo.GetTextElementEnumerator is a .Net 1.1 class that doesn't implement IEnumerable<string>, so convert
        if (s == null)
            yield break;
        var enumerator = StringInfo.GetTextElementEnumerator(s);
        while (enumerator.MoveNext())
            yield return enumerator.GetTextElement();
    }
}

Now, you can do:

var result2 = highUnicodeChar.TextElements().FirstOrDefault() ?? "";

Note that StringInfo.GetTextElementEnumerator() will also group Unicode combining characters, so that the first grapheme cluster of the string Hฬ‚=Tฬ‚+Vฬ‚ will be Hฬ‚ not H.

Sample fiddle here.


Update for .NET Core

In .NET 6 and later, you can use StringInfo.GetNextTextElementLength(ReadOnlySpan<Char>) to iterate through the text elements of a string as a sequence of slices like so:

public static class TextExtensions
{
    public static IEnumerable<ReadOnlyMemory<char>> TextElements(this string s) => (s ?? "").AsMemory().TextElements();

    public static IEnumerable<ReadOnlyMemory<char>> TextElements(this ReadOnlyMemory<char> s)
    {
        for (int index = 0, length = StringInfo.GetNextTextElementLength(s.Span); 
             length > 0; 
             index += length, length = StringInfo.GetNextTextElementLength(s.Span.Slice(index)))
            yield return s.Slice(index, length);
    }
}

This avoids allocating a string for each grapheme.

Or, if you just want the first grapheme, you can do:

var first = highUnicodeChar.AsSpan()
    .Slice(0, StringInfo.GetNextTextElementLength(highUnicodeChar));

Demo fiddle #2 here.

And in .NET Core 3 and later, if you really only want to enumerate through the Unicode code points of a string treating surrogate pairs as a single character but ignoring combining characters and other grapheme groupings, you may use String.EnumerateRunes() to enumerate it as a sequence of Rune structs:

var highUnicodeChar = "๐€"; //Not the standard A

foreach (var rune in highUnicodeChar.EnumerateRunes())
{
    Console.WriteLine($"{rune} = {rune.Value:X}"); // Prints ๐€ = 1D400
}

The Rune struct:

represents a Unicode scalar value, which means any code point excluding the surrogate range (U+D800..U+DFFF). The type's constructors and conversion operators validate the input, so consumers can call the APIs assuming that the underlying Rune instance is well formed.

Demo fiddle #3 here.

Hesitate answered 1/10, 2018 at 4:8 Comment(1)
+1 this is the only reasonable approach. Unfortunately the .NET API (same as most other languages) does not exactly encourage it. There should be a linter for .NET that flags very random access of chars inside a string as an error. โ€“ Virtual

© 2022 - 2024 โ€” McMap. All rights reserved.