Can we simplify this string encoding code

Question

Is it possible to simplify this code into a cleaner/faster form?

StringBuilder builder = new StringBuilder();
var encoding = Encoding.GetEncoding(936);

// convert the text into a byte array
byte[] source = Encoding.Unicode.GetBytes(text);

// convert that byte array to the new codepage. 
byte[] converted = Encoding.Convert(Encoding.Unicode, encoding, source);

// take multi-byte characters and encode them as separate ascii characters 
foreach (byte b in converted)
    builder.Append((char)b);

// return the result
string result = builder.ToString();

Simply put, it takes a string with Chinese characters such as 鄆 and converts them to ài.

For example, that Chinese character in decimal is 37126 or 0x9106 in hex.

See http://unicodelookup.com/#0x9106/1

Converted to a byte array, we get [145, 6] (145 * 256 + 6 = 37126). When encoded in CodePage 936 (simplified chinese), we get [224, 105]. If we break this byte array down into individual characters, we 224=e0=à and 105=69=i in unicode.

See http://unicodelookup.com/#0x00e0/1 and http://unicodelookup.com/#0x0069/1

Thus, we're doing an encoding conversion and ensuring that all characters in our output Unicode string can be represented using at most two bytes.

Update: I need this final representation because this is the format my receipt printer is accepting. Took me forever to figure it out! :) Since I'm not an encoding expert, I'm looking for simpler or faster code, but the output must remain the same.

Update (Cleaner version):

return Encoding.GetEncoding("ISO-8859-1").GetString(Encoding.GetEncoding(936).GetBytes(text));

Purgatorial

I'm guessing your receipt printer doesn't accept .NET strings, so what exactly are you sending to the receipt printer? Text blobs? If so, all text is encoded over the wire, so there's a good chance there's some hidden encoding going on later in the process; it might be easier to figure out the "best" solution if it were clear how you're communicating with the printer. – Serpentiform

I'm using POS for .NET ... it accepts strings and it worked fine as long as I remained in CodePage 1252 ... but moving to 936 caused issues, which is due to the way this specific printer recognizes these characters. – Purgatorial

It's often faster to allocate an array of characters, use a for loop to assign to it, and then use string's constructor to make it into a string, rather than appending to a string character by character. – Rumba

Answer 1

Well, for one, you don't need to convert the "built-in" string representation to a byte array before calling Encoding.Convert.

You could just do:

byte[] converted = Encoding.GetEncoding(936).GetBytes(text);

To then reconstruct a string from that byte array whereby the char values directly map to the bytes, you could do...

static string MangleTextForReceiptPrinter(string text) {
    return new string(
        Encoding.GetEncoding(936)
            .GetBytes(text)
            .Select(b => (char) b)
            .ToArray());
}

I wouldn't worry too much about efficiency; how many MB/sec are you going to print on a receipt printer anyhow?

Joe pointed out that there's an encoding that directly maps byte values 0-255 to code points, and it's age-old Latin1, which allows us to shorten the function to...

return Encoding.GetEncoding("Latin1").GetString(
           Encoding.GetEncoding(936).GetBytes(text)
       );

By the way, if this is a buggy windows-only API (which it is, by the looks of it), you might be dealing with codepage 1252 instead (which is almost identical). You might try reflector to see what it's doing with your System.String before it sends it over the wire.

Serpentiform

Your code is good enough for me! Was wondering if there was a bult-in mangling function I was not aware of that would be more efficient than my loop. :) – Purgatorial

Answer 2

Almost anything would be cleaner than this - you're really abusing text here, IMO. You're trying to represent effectively opaque binary data (the encoded text) as text data... so you'll potentially get things like bell characters, escapes etc.

The normal way of encoding opaque binary data in text is base64, so you could use:

return Convert.ToBase64String(Encoding.GetEncoding(936).GetBytes(text));

The resulting text will be entirely ASCII, which is much less likely to cause you hassle.

EDIT: If you need that output, I would strongly recommend that you represent it as a byte array instead of as a string... pass it around as a byte array from that point onwards, so you're not tempted to perform string operations on it.

Educative

+1. I suspect that the OP's approach will not always be reversible. Meaning you'll be able to encode some data but not decode it correctly. – Tavel

The end encoding is required by a receipt printer I am sending data to. – Purgatorial

Answer 3

Does your receipt printer have an API that accepts a byte array rather than a string? If so you may be able to simplify the code to a single conversion, from a Unicode string to a byte array using the encoding used by the receipt printer.

Also, if you want to convert an array of bytes to a string whose character values correspond 1-1 to the values of the bytes, you can use the code page 28591 aka Latin1 aka ISO-8859-1.

I.e., the following

foreach (byte b in converted) 
    builder.Append((char)b); 

string result = builder.ToString(); 

can be replaced by:

// All three of the following are equivalent
// string result = Encoding.GetEncoding(28591).GetString(converted);
// string result = Encoding.GetEncoding("ISO-8859-1").GetString(converted);
string result = Encoding.GetEncoding("Latin1").GetString(converted);

Latin1 is a useful encoding when you want to encode binary data in a string, e.g. to send through a serial port.

Stormystorting

It doesn't, unfortunately. If it had, I wouldn't have spent as much time trying to understand its cryptic encoding scheme! – Purgatorial

Probably internally it is converting the unicode string back into a byte array for transmission to the printer, perhaps by using an encoding such as Latin1. – Stormystorting

Nice! I didn't know that converting to Latin-1 would replace my loop. – Purgatorial