UTF-8, UTF-16, and UTF-32

Question

What are the differences between UTF-8, UTF-16, and UTF-32?

I understand that they will all store Unicode, and that each uses a different number of bytes to represent a character. Is there an advantage to choosing one over the other?

Malvinamalvino

Watch this video if you are interested in how Unicode works youtube.com/watch?v=MijmeoH9LT4 – Malvinamalvino

The video focuses on UTF-8, and yes it explains well how variable length encoding works and is mostly compatible with computers reading or writing only fixed length ASCII. Unicode guys were smart when designing UTF-8 encoding. – Sutphin

UTF-8 is the de-facto standard in most modern software for saved files. More specifically, it's the most widely used encoding for HTML and configuration and translation files (Minecraft, for example, doesn't accept any other encoding for all its text information). UTF-32 is fast for internal memory representation, and UTF-16 is kind of deprecated, currently used only in Win32 for historical reasons (UTF-16 was fixed-length when Windows 95 was a thing) – Coldhearted

@VladislavToncharov UTF-16 was never a fixed length encoding. You're confusing it with UCS-2. – Naseberry

@Coldhearted Javascript still uses UTF-16 for almost everything – Pereyra

@Malvinamalvino - I clicked the link, saw Tom Scott, and automatically upvoted your comment before even watching the video b/c Tom is freaking awesome and has a gift for conveying information. Thank you for the link. – Seroka

Answer 1

UTF-8 has an advantage in the case where ASCII characters represent the majority of characters in a block of text, because UTF-8 encodes these into 8 bits (like ASCII). It is also advantageous in that a UTF-8 file containing only ASCII characters has the same encoding as an ASCII file.

UTF-16 is better where ASCII is not predominant, since it uses 2 bytes per character, primarily. UTF-8 will start to use 3 or more bytes for the higher order characters where UTF-16 remains at just 2 bytes for most characters.

UTF-32 will cover all possible characters in 4 bytes. This makes it pretty bloated. I can't think of any advantage to using it.

Gannie

UTF-32 advantage: you don't need to decode stored data to the 32-bit Unicode code point for e.g. character by character handling. The code point is already available right there in your array/vector/string. – Damaris

@rq: You're quite right and Adam makes the same point. However, most character by character handling I've seen works with 16 bit short ints not with a vector of 32 bit integers. In terms of raw speed some operations will be quicker with 32 bits. – Gannie

It's also easier to parse if (heaven help you) you have to re-implement the wheel. – Ceiba

UTF32 advantage: When transferring over network especially in UDP, it's a good thing to know that 4 bytes is always one character, in scenarios where all characters are needed. – Thermionic

Well, UTF-8 has an advantage in network transfers - no need to worry about endianness since you're transfering data one byte at a time (as opposed to 4). – Kikelia

@Damaris You can't do character-by-character handling in UTF-32, as code point does not always correspond to a character. – Vipul

-1 just because there are a couple of advantages to UTF-32 even if they're not important enough to make many projects want to choose it. The only projects I know of that uses it is the word processor AbiWord. – Altimeter

@Altimeter could you name some advantages of utf-32, which are not already mentioned by other comments? – Libertinage

@naxa: The advantages of UTF-32 are already mentioned by other comments. AnthonyWJones said "I can't think of any advantage to use it." – Altimeter

Personally I use UTF-8 always except with Windows API code where I use UTF-16. Years ago when I was involved with AbiWord they chose to use UTF-32 internally and it is still the only project I know of to do this. I don't know if they stuck with it. Just because some people mix up "character" and "codepoint" doesn't mean that there's no advantages to knowing that codepoints are all a fixed size. – Altimeter

UTF-32 advantage: string manipulation is possibly faster compared to the utf-8 equivalent – Sturgis

@Wes: I doubt it; most of string handling is not done on a per-character basis (things like substring searches work equally well on ASCII, UTF-8, or arbitrary arrays of bytes, regardless of character data they [might] encode). If it is, then UTF-32 does not suffice (you can have multiple units per character even in fully-normalized UTF-32!). If anything, it makes it slower due to (roughly) 4x the data for copying. – Kikelia

@Wes: I found a good source for this: Figure 5 in this (official!) document. Note how even the normalized characters are multiple code points (so, even in UTF-32). – Kikelia

@TimČas Talking of code points, not graphemes. Locating a code point by offset is a very intensive operation in utf-8 as it requires full iteration with "jumps" of 2->4 bytes, while utf-32 has actual random access. Substring operations are faster consequently. Instead, as you said, locating graphemes requires full traversal in both encodings, but in utf-32 less jumps will be required. – Sturgis

@Wes: But what substring operations would need that? For example, finding a substring works just as well on UTF-8 as it does on UTF-32 (you're just finding a specific sequence of uint8s / uint32s). The index returned can directly be used for (say) slicing to the end of the string in both cases. – Kikelia

works just as well, but that's not random access. for instance just knowing the length of a string (in code points) would require a full traversal of the byte array, while with utf-32 it's just sizeof(codepoints) – Sturgis

Another advantage of UTF8 is you don't need to duplicate your API. Like those nasty windows W versions of the API. Why didn't they adopt UTF8? – Bed

Another way to describe UTF32's ability for random access is to say string slicing is O(1) in UTF32 and O(n) in UTF8 even in best cases. – Falla

utf-32 is not only more efficient for string operations (it supports random access; enough said!), but it's also simpler to manipulate by virtue of being a fixed-size array (I dare you work with utf-8 in C...) – Pawl

@Nawaz Note that some of the bits are used to identify what the size of the character is, so you don't get to use the entire 8 or 16 etc bits for your character. – Starstarboard

@ hippietrail D language uses utf-32 (till it supports utf8/16/32 natively) in iteration for example – Rigging

@TimČas I am not sure about "slower" argument: in most cases blocks are transferring, not bytes (all external devices). In the case of DMA I have 2 ideas: 1) I dont know the difference b/w 16 bits vs 32 bits modes, but often it's better to use full register size then just a part of it ALSO performance of 32 bits may be the same as of 16 bits (everything happens on clock signal) 2) if we have 200Mb/s then we can accept that the times to copy of 1024 bytes and 64 bytes is super close – Rigging

@étale-cohomologyRandom access to "characters" (what Unicode technically calls "grapheme clusters") in UTF-32 is a myth. Even fully-normalized UTF-32 uses combining characters (consider emoji!). And like I said, you pretty much never need code point random access. – Kikelia

@Rigging Memory-copy operations on byte streams typically do use word-sized moves. A common optimization of memcpy is to split the copy into unaligned copy (first 0-7 bytes), and then do 64-bit copies for the main part --- plus another partial at the end. All that matters in the end (for non-trivially-small sizes) is the amount of data, not what the base units are. – Kikelia

I think the question is not random or not, but how often jumps happen, about statistics, in UTF-8 they happen more often, so all iterations will be slower. – Rigging

@TimČas I very much disagree with you there. Slicing and getting specific characters is extremely useful and common. I think you're getting characters confused with graphemes; UTF-32 gives you O(1) indexing into a string of characters (not graphemes), but I'd guess that a very large percentage of string manipulation doesn't actually care about graphemes. – Pereyra

@RedwolfPrograms "Character" is ambigious in Unicode (unicode.org/glossary/#character), but people typically mean "grapheme cluster" or "code point" when they say it. I'm not sure which one you mean. But anyway, go on, name 1 scenario where you need to deal in anything but strings-as-whole-units or graphemes, other than rendering glyphs (where you need code points in order to reference TTF/OTF internal tables, making it a bit of a circular argument). – Kikelia

@TimČas Any situation where how the string actually looks just...doesn't matter. Which is most. E.g., I write a lot of interpreters. They don't care if an emoji is part of a grapheme or on its own, it's just a part of an identifier (potentially along with a ZWJ and something else). There has literally never been a situation where I've had to handle grapheme clusters in my code, but I do stuff involving string manipulation basically every day. – Pereyra

Answer 2

In short:

• UTF-8: Variable-width encoding, backwards compatible with ASCII. ASCII characters (U+0000 to U+007F) take 1 byte, code points U+0080 to U+07FF take 2 bytes, code points U+0800 to U+FFFF take 3 bytes, code points U+10000 to U+10FFFF take 4 bytes. Good for English text, not so good for Asian text.
• UTF-16: Variable-width encoding. Code points U+0000 to U+FFFF take 2 bytes, code points U+10000 to U+10FFFF take 4 bytes. Bad for English text, good for Asian text.
• UTF-32: Fixed-width encoding. All code points take four bytes. An enormous memory hog, but fast to operate on. Rarely used.

In long: see Wikipedia: UTF-8, UTF-16, and UTF-32.

Steeve

The reason UTF-16 works is that U+D800–U+DFFF are left as a gap in the BMP for the surrogate pair pairs. Clever. – Crites

@spurrymoses: I'm referring strictly to the amount of space taken up by the data bytes. UTF-8 requires 3 bytes per Asian character, while UTF-16 only requires 2 bytes per Asian character. This really isn't a major problem, since computers have tons of memory these days compared to the average amount of text stored in a program's memory. – Steeve

UTF-32 isn't rarely used anymore... on osx and linux wchar_t defaults to 4 bytes. gcc has an option -fshort-wchar which reduces the size to 2 bytes, but breaks the binary compatibility with std libs. – Woodhouse

@PandaWood ofcource UTF-8 can encode any character! But have you compared the memory requirement with that for UTF-16? You seem to be missing the point! – Kayo

If someone were to say UTF-8 is "not so good for Asian text" in the context of All Encoding Formats Including Those That Cannot Encode Unicode, they would of course be wrong. But that is not the context. The context of memory requirements comes from the fact that the question (and answer) is comparing UTF-8, UTF-16 and UTF-32, which will all encode Asian text but use differing amounts of memory/storage. It follows that their relative goodness would naturally be entirely in the context of memory requirements. "Not so good" != "not good". – Probative

@vine'th: wchar_t has not gained much popularity that I've seen. The fact that it's 16 bits wide on Windows and 32 bits wide on *nix is a probably contributor to its lack of acceptance. In *nix most projects eschew wchar_t and just use char with UTF-8. – Altimeter

Wikipedia remarks that in real world usage, it appeared that UTF-8 is smaller then UTF-16 even when using non English characters because of the amount of spaces or english word still used in text. – Biedermeier

Is there no reference source available more trustworthy than Wikipedia? (Not that stackoverflow is any better in that regard...) – Rash

@McGafter: Well of course there is. If you want trustworthiness, go straight to the horse's mouth at The Unicode Consortium. See chapter 2.5 for a description of the UTF-* encodings. But for obtaining a simple, high-level understanding of the encodings, I find that the Wikipedia articles are a much more approachable source. – Steeve

@PandaWood web pages contain a lot of ASCII characters that aren't part of the body text, so UTF-8 is a good choice for those no matter what language you're using. – Translator

While UTF-8 does take 3 bytes for most Asian characters vs 2 for UTF-16 (some Chinese characters in common use ended up in the multilingual plane where they take 4 bytes in both UTF-8 and UTF-16), in practice this does not make much difference because real documents often have a large number of ASCII characters mixed in. See utf8everywhere.org/#asian for side-by-side size comparisons of one real document: UTF-8 actually took 50% fewer bytes to encode a Japanese-language HTML page (the Wikipedia article on Japan, in Japanese) than UTF-16 did. – Critta

absolutely truth about Chinese language: I created 2 files with Notepad with Chinese text: I saved one in UTF-16 and another one in UTF-8. The ratio is utf8_size/utf16_size = 1.4 (about 4K vs 2K). Cyrillic ratio is different: utf16_size/utf8_size = 1.14 (about 13K vs 11K) – Rigging

Answer 3

• UTF-8 is variable 1 to 4 bytes.

• UTF-16 is variable 2 or 4 bytes.

• UTF-32 is fixed 4 bytes.

Oratorio

UTF8 is actually 1 to 6 bytes. – Salisbury

@Salisbury is technically correct because mapping the full range of UTF32/LE/BE includes U-00200000 - U-7FFFFFFF even though Unicode v6.3 ends at U-0010FFFF inclusive. Here's a nice breakdown of how to enc/dec 5 and 6 byte utf8: lists.gnu.org/archive/html/help-flex/2005-01/msg00030.html – Lade

backing up these with relevant references parts and their sources? – Libertinage

@Salisbury No, UTF-8 can not be 5 or 6 bytes. Unicode code points are limited to 21 bits, which limits UTF-8 to 4 bytes. (You could of course extend the principle of UTF-8 to encode arbitrary large integers, but it would not be Unicode.) See RFC 3629. – Sitin

Why are all you varying the size? UTF-8 1-4 bytes.. then 1-6. then why other UTFs? – Heber

Quoting Wikipedia: In November 2003, UTF-8 was restricted by RFC 3629 to match the constraints of the UTF-16 character encoding: explicitly prohibiting code points corresponding to the high and low surrogate characters removed more than 3% of the three-byte sequences, and ending at U+10FFFF removed more than 48% of the four-byte sequences and all five- and six-byte sequences. – Talc

Could the standard be extended in the future to allow any of these to use 5 bytes, or are they limited to 4 exactly in some technical way? – Starstarboard

@AaronFranke: the first byte can define up to 7 continuation bytes, so it can technically be extended up to 8 bytes (36 payload bits ~ 68 billion codepoints) per sequence. – Oratorio

@quas 7 continuation bytes at 6 payload bits makes 42. – Undermanned

@Deduplicator: I meant 6 continuation bytes, of course, thanks for noticing – Oratorio

@Oratorio Well, 0xFF could be invalid, or signal 7 continuation bytes... – Undermanned

@Deduplicator: you're right, this way it can be 42 bits indeed – Oratorio

Answer 4

Unicode defines a single huge character set, assigning one unique integer value to every graphical symbol (that is a major simplification, and isn't actually true, but it's close enough for the purposes of this question). UTF-8/16/32 are simply different ways to encode this.

In brief, UTF-32 uses 32-bit values for each character. That allows them to use a fixed-width code for every character.

UTF-16 uses 16-bit by default, but that only gives you 65k possible characters, which is nowhere near enough for the full Unicode set. So some characters use pairs of 16-bit values.

And UTF-8 uses 8-bit values by default, which means that the 127 first values are fixed-width single-byte characters (the most significant bit is used to signify that this is the start of a multi-byte sequence, leaving 7 bits for the actual character value). All other characters are encoded as sequences of up to 4 bytes (if memory serves).

And that leads us to the advantages. Any ASCII-character is directly compatible with UTF-8, so for upgrading legacy apps, UTF-8 is a common and obvious choice. In almost all cases, it will also use the least memory. On the other hand, you can't make any guarantees about the width of a character. It may be 1, 2, 3 or 4 characters wide, which makes string manipulation difficult.

UTF-32 is opposite, it uses the most memory (each character is a fixed 4 bytes wide), but on the other hand, you know that every character has this precise length, so string manipulation becomes far simpler. You can compute the number of characters in a string simply from the length in bytes of the string. You can't do that with UTF-8.

UTF-16 is a compromise. It lets most characters fit into a fixed-width 16-bit value. So as long as you don't have Chinese symbols, musical notes or some others, you can assume that each character is 16 bits wide. It uses less memory than UTF-32. But it is in some ways "the worst of both worlds". It almost always uses more memory than UTF-8, and it still doesn't avoid the problem that plagues UTF-8 (variable-length characters).

Finally, it's often helpful to just go with what the platform supports. Windows uses UTF-16 internally, so on Windows, that is the obvious choice.

Linux varies a bit, but they generally use UTF-8 for everything that is Unicode-compliant.

So short answer: All three encodings can encode the same character set, but they represent each character as different byte sequences.

Orangutan

It is inaccurate to say that Unicode assigns a unique integer to each graphical symbol. It assigns such to each code point, but some code points are invisible control characters, and some graphical symbols require multiple code points to represent. – Roundworm

@tchrist: yes, it's inaccurate. The problem is that to accurately explain Unicode, you need to write thousands of pages. I hoped to get the basic concept across to explain the difference between encodings – Orangutan

@Orangutan lol right so basically to explain Unicode you would have to write the Unicode Core Specification – L

@Roundworm More specifically, you can construct Chinese symbols out of provided primitives (but they're in the same chart, so you'll just end up using unreal amount of space - either disk or RAM - to encode them) instead of using the built-in ones. – Coldhearted

Best answer by far – Decortication

Note that the description of UTF-32 is incorrect. Each character is not 4 bytes wide. Each code point is 4 bytes wide, and some characters may require multiple code points. Computing string length is not just the number of bytes divided by 4, you have to walk the whole string and decode each code point to resolve these clusters. – Thoron

Answer 5

Unicode is a standard and about UTF-x you can think as a technical implementation for some practical purposes:

• UTF-8 - "size optimized": best suited for Latin character based data (or ASCII), it takes only 1 byte per character but the size grows accordingly symbol variety (and in worst case could grow up to 6 bytes per character)
• UTF-16 - "balance": it takes minimum 2 bytes per character which is enough for existing set of the mainstream languages with having fixed size on it to ease character handling (but size is still variable and can grow up to 4 bytes per character)
• UTF-32 - "performance": allows using of simple algorithms as result of fixed size characters (4 bytes) but with memory disadvantage

Burgle

«mainstream languages» not that mainstream in a lot of parts of the world ^^ – Nadabb

UTF-16 is actually size optimized for non ASCII chars. For it really depends with which languages it will be used. – Nadabb

@Nadabb totally agree, it is worth noting sets of Hanzi and Kanji characters for Asian part of world. – Burgle

Should be the top answer. This is too correct to be buried here. – Acriflavine

utf-8 might be faster than all of these just because developers spent the most effort optimizing it – Charlyncharm

Answer 6

I tried to give a simple explanation in my blogpost.

UTF-32

requires 32 bits (4 bytes) to encode any character. For example, in order to represent the "A" character code-point using this scheme, you'll need to write 65 in 32-bit binary number:

00000000 00000000 00000000 01000001 (Big Endian)

If you take a closer look, you'll note that the most-right seven bits are actually the same bits when using the ASCII scheme. But since UTF-32 is fixed width scheme, we must attach three additional bytes. Meaning that if we have two files that only contain the "A" character, one is ASCII-encoded and the other is UTF-32 encoded, their size will be 1 byte and 4 bytes correspondingly.

UTF-16

Many people think that as UTF-32 uses fixed width 32 bit to represent a code-point, UTF-16 is fixed width 16 bits. WRONG!

In UTF-16 the code point maybe represented either in 16 bits, OR 32 bits. So this scheme is variable length encoding system. What is the advantage over the UTF-32? At least for ASCII, the size of files won't be 4 times the original (but still twice), so we're still not ASCII backward compatible.

Since 7-bits are enough to represent the "A" character, we can now use 2 bytes instead of 4 like the UTF-32. It'll look like:

00000000 01000001

UTF-8

You guessed right.. In UTF-8 the code point maybe represented using either 32, 16, 24 or 8 bits, and as the UTF-16 system, this one is also variable length encoding system.

Finally we can represent "A" in the same way we represent it using ASCII encoding system:

01001101

A small example where UTF-16 is actually better than UTF-8:

Consider the Chinese letter "語" - its UTF-8 encoding is:

11101000 10101010 10011110

While its UTF-16 encoding is shorter:

10001010 10011110

In order to understand the representation and how it's interpreted, visit the original post.

Fellner

#3865342. – Fernandez

How does computers don't 'drop' UTF-32 encode numbers that contains alot of zeros? like representing 'A' will contain 26-27 zeros... – Tegan

@ArikJordanGraham If you are trying to ask a question, it is really unclear what it is. What do you mean by "drop"? (Or, well, "alot".) – Armilda

Answer 7

UTF-8

• has no concept of byte-order
• uses between 1 and 4 bytes per character
• ASCII is a compatible subset of encoding
• completely self-synchronizing e.g. a dropped byte from anywhere in a stream will corrupt at most a single character
• pretty much all European languages are encoded in two bytes or less per character

UTF-16

• must be parsed with known byte-order or reading a byte-order-mark (BOM)
• uses either 2 or 4 bytes per character

UTF-32

• every character is 4 bytes
• must be parsed with known byte-order or reading a byte-order-mark (BOM)

UTF-8 is going to be the most space efficient unless a majority of the characters are from the CJK (Chinese, Japanese, and Korean) character space.

UTF-32 is best for random access by character offset into a byte-array.

Umbrella

How does "self synchronizing" work in UTF-8? Can you give examples for 1 byte and 2 byte characters? – Intratelluric

@KorayTugay Valid shorter byte strings are never used in longer characters. For instance, ASCII is in the range 0-127, meaning all one-byte characters have the form 0xxxxxxx in binary. All two-byte characters begin with 110xxxxx with a second byte of 10xxxxxx. So let's say the first character of a two-byte character is lost. As soon as you see 10xxxxxx without a preceding 110xxxxxx, you can determine for sure that a byte was lost or corrupted, and discard that character (or re-request it from a server or whatever), and move on until you see a valid first byte again. – Botulin

if you have the offset to a character, you have the offset to that character -- utf8, utf16 or utf32 will work just the same in that case; i.e. they are all equally good at random access by character offset into a byte array. The idea that utf32 is better at counting characters than utf8 is also completely false. A codepoint (which is not the same as a character which again, is not the same as a grapheme.. sigh), is 32 bits wide in utf32 and between 8 and 32 bits in utf8, but a character may span multiple codepoints, which destroys the major advantage that people claim utf32 has over utf8. – Ideally

@Ideally But how often do you need to work with characters/graphemes rather than just codepoints? I have worked on a number of projects involving heavy string manipulation, and being able to slice/index codepoints in O(1) really is very helpful. – Pereyra

@RedwolfPrograms Today I don't, but I used to work in language anaylsis, where it was very important. – Ideally

@Ideally : that's not true - you're still assuming offsets come in full characters. When only individual bytes got corrupted along the way, only UTF-8 offers instant self synchronization. With UTF-32, say if one of the null bytes in a middle of a chain is missing, you actually need to spend a lot of effort trying to deduce exactly which char that null byte belonged to (since every UTF-32, as of today, has at least 1 \0 per char). That's why BOM is such a useless concept for UTF-8 – Cully

@RadvylfPrograms : i deal with the CJK BMP all the time, and I don't see anything wrong with UTF-8. – Cully

Answer 8

I'm surprised this question is 11yrs old and not one of the answers mentioned the #1 advantage of utf-8.

utf-8 generally works even with programs that are not utf-8 aware. That's partly what it was designed for. Other answers mention that the first 128 code points are the same as ASCII. All other code points are generated by 8bit values with the high bit set (values from 128 to 255) so that from the POV of a non-unicode aware program it just sees strings as ASCII with some extra characters.

As an example let's say you wrote a program to add line numbers that effectively does this (and to keep it simple let's assume end of line is just ASCII 13)

// pseudo code

function readLine
  if end of file
     return null
  read bytes (8bit values) into string until you hit 13 or end or file
  return string

function main
  lineNo = 1
  do {
    s = readLine
    if (s == null) break;
    print lineNo++, s
  }  

Passing a utf-8 file to this program will continue to work. Similarly, splitting on tabs, commas, parsing for ASCII quotes, or other parsing for which only ASCII values are significant all just work with utf-8 because no ASCII value appear in utf-8 except when they are actually meant to be those ASCII values

Some other answers or comments mentions that utf-32 has the advantage that you can treat each codepoint separately. This would suggest for example you could take a string like "ABCDEFGHI" and split it at every 3rd code point to make

ABC
DEF
GHI

This is false. Many code points affect other code points. For example the color selector code points that lets you choose between 👨🏻‍🦳👨🏼‍🦳👨🏽‍🦳👨🏾‍🦳👨🏿‍🦳. If you split at any arbitrary code point you'll break those.

Another example is the bidirectional code points. The following paragraph was not entered backward. It is just preceded by the 0x202E codepoint

• ‮This line is not typed backward it is only displayed backward

So no, utf-32 will not let you just randomly manipulate unicode strings without a thought to their meanings. It will let you look at each codepoint with no extra code.

FYI though, utf-8 was designed so that looking at any individual byte you can find out the start of the current code point or the next code point.

If you take a arbitrary byte in utf-8 data. If it is < 128 it's the correct code point by itself. If it's >= 128 and < 192 (the top 2 bits are 10) then to find the start of the code point you need to look the preceding byte until you find a byte with a value >= 192 (the top 2 bits are 11). At that byte you've found the start of a codepoint. That byte encodes how many subsequent bytes make the code point.

If you want to find the next code point just scan until the byte < 128 or >= 192 and that's the start of the next code point.

Num Bytes	1st code point	last code point	Byte 1	Byte 2	Byte 3	Byte 4
1	U+0000	U+007F	0xxxxxxx
2	U+0080	U+07FF	110xxxxx	10xxxxxx
3	U+0800	U+FFFF	1110xxxx	10xxxxxx	10xxxxxx
4	U+10000	U+10FFFF	11110xxx	10xxxxxx	10xxxxxx	10xxxxxx

Where xxxxxx are the bits of the code point. Concatenate the xxxx bits from the bytes to get the code point

Adaliah

Another design feature: UTF-8 unaware code can correctly sort UTF-8 strings in ascending codepoint order (however useful that may be) – Hydromedusa

Answer 9

In UTF-32 all of characters are coded with 32 bits. The advantage is that you can easily calculate the length of the string. The disadvantage is that for each ASCII characters you waste an extra three bytes.

In UTF-8 characters have variable length, ASCII characters are coded in one byte (eight bits), most western special characters are coded either in two bytes or three bytes (for example € is three bytes), and more exotic characters can take up to four bytes. Clear disadvantage is, that a priori you cannot calculate string's length. But it's takes lot less bytes to code Latin (English) alphabet text, compared to UTF-32.

UTF-16 is also variable length. Characters are coded either in two bytes or four bytes. I really don't see the point. It has disadvantage of being variable length, but hasn't got the advantage of saving as much space as UTF-8.

Of those three, clearly UTF-8 is the most widely spread.

Smokeless

Why would I want to calculate the length of the string while developing websites? Is there any advantage of choosing UTF-8/UTF-16 in web development? – Geisha

"The advantage is that you can easily calculate the length of the string" If you define length by the # of codepoints, then yes, you can just divide the byte length by 4 to get it with UTF-32. That's not a very useful definition, however : it may not relate to the number of characters. Also, normalization may alter the number of codepoints in the string. For example, the french word "été" can be encoded in at least 4 different ways, with 3 distinct codepoint lengths. – Naseberry

UTF-16 is possibly faster than UTF-8 while also no wasting memory like UTF-32 does. – Acriflavine

@MichalŠtein But it also gives you the worst of both worlds; it uses up more space than UTF-8 for ASCII, but it also has all of the same issues caused by having multiple codepoints per character (in addition to potential endianness issues). – Pereyra

Answer 10

I made some tests to compare database performance between UTF-8 and UTF-16 in MySQL.

Update Speeds

UTF-8

UTF-16

Insert Speeds

Delete Speeds

Pad

Just one short string doesn't mean anything, just one record even less, the time differences may have been due to other factors, Mysql's own internal mechanisms, if you want to do a reliable test, you would need to use at least 10,000 records with a 200 character string, and it would need to be a set of tests, with some scenarios, at least about 3, so it would isolate the encoding factor – Antimicrobial

Answer 11

Depending on your development environment you may not even have the choice what encoding your string data type will use internally.

But for storing and exchanging data I would always use UTF-8, if you have the choice. If you have mostly ASCII data this will give you the smallest amount of data to transfer, while still being able to encode everything. Optimizing for the least I/O is the way to go on modern machines.

Beaux

Arguably, a lot more important than space requirements is the fact, that UTF-8 is immune to endianness. UTF-16 and UTF-32 will inevitably have to deal with endianness issues, where UTF-8 is simply a stream of octets. – Finegrained

Answer 12

After reading through the answers, UTF-32 needs some love.

C#:

Data1 = RandomNumberGenerator.GetBytes(500_000_000);

sw = Stopwatch.StartNew();
int l = Encoding.UTF8.GetString(Data1).Length;
sw.Stop();
Console.WriteLine($"UTF-8: Elapsed - {sw.ElapsedMilliseconds * .001:0.000s}   Length - {l:###,###,###}");

sw = Stopwatch.StartNew();
l = Encoding.Unicode.GetString(Data1).Length;
sw.Stop();
Console.WriteLine($"Unicode: Elapsed - {sw.ElapsedMilliseconds * .001:0.000s}   Length - {l:###,###,###}");

sw = Stopwatch.StartNew();
l = Encoding.UTF32.GetString(Data1).Length;
sw.Stop();
Console.WriteLine($"UTF-32: Elapsed - {sw.ElapsedMilliseconds * .001:0.000s}   Length - {l:###,###,###}");

sw = Stopwatch.StartNew();
l = Encoding.ASCII.GetString(Data1).Length;
sw.Stop();
Console.WriteLine($"ASCII: Elapsed - {sw.ElapsedMilliseconds * .001:0.000s}   Length - {l:###,###,###}");

ASCII -- Elapsed 2.362s - Length 500,000,000

UTF-8 -- Elapsed 9.939s - Length 473,752,800

Unicode -- Elapsed 0.853s - Length 250,000,000

UTF-32 -- Elapsed 3.143s - Length 125,030,570

Hernadez

Any idea what makes Unicode the fastest? – Anemia

Yes, because Unicode is 1:1 conversion from bytes. There's nothing to derive, so it's basically doing a memcopy. UTF-32 is close to 1:1 as indicated by the returned length, but clearly there are edge cases that must be checked during conversion. – Hernadez

"Unicode" here means UTF-16, I presume? – Armilda

No, not exactly. Unicode is a standard and UTF-16 is a possible representation of Unicode. In the Microsoft ecosystem, they are very close, but not the same. – Hernadez

Answer 13

As mentioned, the difference is primarily the size of the underlying variables, which in each case get larger to allow more characters to be represented.

However, fonts, encoding and things are wickedly complicated (unnecessarily?), so a big link is needed to fill in more detail:

http://www.cs.tut.fi/~jkorpela/chars.html#ascii

Don't expect to understand it all, but if you don't want to have problems later it's worth learning as much as you can, as early as you can (or just getting someone else to sort it out for you).

Paul.

Fauver

or just use UTF-8 as default as it has become the de-facto standard, and find out if a new system supports it or not. if it doesn't, you can come back to this post. – Quiroz

@paul-w-homer Your link is broken. – Pierian

Answer 14

In short, the only reason to use UTF-16 or UTF-32 is to support non-English and ancient scripts respectively.

I was wondering why anyone would chose to have non-UTF-8 encoding when it is obviously more efficient for web/programming purposes.

A common misconception - the suffixed number is NOT an indication of its capability. They all support the complete Unicode, just that UTF-8 can handle ASCII with a single byte, so is MORE efficient/less corruptible to the CPU and over the internet.

Some good reading: http://www.personal.psu.edu/ejp10/blogs/gotunicode/2007/10/which_utf_do_i_use.html and http://utf8everywhere.org

Medicinal

I'm not sure, why you suggest, that using UTF-16 or UTF-32 were to support non-English text. UTF-8 can handle that just fine. And there are non-ASCII characters in English text, too. Like a zero-width non-joiner. Or an em dash. I'm afraid, this answer doesn't add much value. – Finegrained

This question is liable to downvoting because UTF-8 is still commonly used in HTML files even if the majority of the characters are 3-byte characters in UTF-8, – Willtrude

@Finegrained support is not the best wording, promote or better support would be more accurate – Quiroz

Sending a page like utf8everywhere.org is not what I would do in a SO answer. – Acriflavine