How many bytes does one Unicode character take?

Question

I am a bit confused about encodings. As far as I know old ASCII characters took one byte per character. How many bytes does a Unicode character require?

I assume that one Unicode character can contain every possible character from any language - am I correct? So how many bytes does it need per character?

And what do UTF-7, UTF-6, UTF-16 etc. mean? Are they different versions of Unicode?

I read the Wikipedia article about Unicode but it is quite difficult for me. I am looking forward to seeing a simple answer.

Agonized

en.wikipedia.org/wiki/Comparison_of_Unicode_encodings – Poignant

Sorry, there is no simple answer. I find the whole thing a bit of a mess. Unicode was billed as using two bytes and being able to represent all characters, but it turns out two bytes wasn't quite enough. – Vanbuskirk

"Simple answer": A unicode character takes 1-4 bytes. Unicode covers a lot of languages but not all. Last time I looked, for example Klingon was not an official Unicode character set. – Bocanegra

Klingon is not part of the Unicode standard itself, no. It uses Uniode's Private Use Area (U+F8D0 - U+F8FF) instead. – Qatar

Saviour question - thanks. My situation is storing data via SCORM 1.2 compliant LMSs... the standard for SCORM 1.2 'cmi.suspend_data' is 4096 bytes of data which a previous developer assumed meant we could store 4096 characters. Oh man was he wrong - I've just discovered why our bookmarking fails on long courses. So now I know since we're using UTF-8 it takes 4 bytes per character giving us 1024 characters. – Sempiternal

A lot of characters of human languages are not included in Unicode. – Rookie

@NicolasBarbulesco For example? – Marketable

This helped me: gammon.com.au/unicode – Byelostok

Answer 1

You won't see a simple answer because there isn't one.

First, Unicode doesn't contain "every character from every language", although it sure does try.

Unicode itself is a mapping, it defines codepoints and a codepoint is a number, associated with usually a character. I say usually because there are concepts like combining characters. You may be familiar with things like accents, or umlauts. Those can be used with another character, such as an a or a u to create a new logical character. A character therefore can consist of 1 or more codepoints.

To be useful in computing systems we need to choose a representation for this information. Those are the various unicode encodings, such as utf-8, utf-16le, utf-32 etc. They are distinguished largely by the size of of their codeunits. UTF-32 is the simplest encoding, it has a codeunit that is 32bits, which means an individual codepoint fits comfortably into a codeunit. The other encodings will have situations where a codepoint will need multiple codeunits, or that particular codepoint can't be represented in the encoding at all (this is a problem for instance with UCS-2).

Because of the flexibility of combining characters, even within a given encoding the number of bytes per character can vary depending on the character and the normalization form. This is a protocol for dealing with characters which have more than one representation (you can say "an 'a' with an accent" which is 2 codepoints, one of which is a combining char or "accented 'a'" which is one codepoint).

Leyla

OK. Then how many bytes takes one given character represented in one given codepoint ? For example, the non-breaking space. – Rookie

The combining characters make life of a programmer a hell when it comes to writing strlen(), substr() and other string manipulation functions on UTF8 arrays. This kind of work will be never complete and always buggy. – Azral

I wrote a demo that shows Windows-1252, UTF8 and UTF8-BOM encoded files interpreted with each encoding, and compares equality between the results: github.com/vladyrn/encodings_demo – Microeconomics

In practice no Western language ever uses two codepoints for a character. I'm not sure about CJK, but combining characters is something westeners only see when representing an astrocat emoji, as all characters using diacritics exist as unique codepoints as well. – Marketable

@Marketable This is not true. é that's two codepoints. In hex: ["0065", "0301"] – Dekaliter

@PeterR é is just U+00E9 (just google the character). Your hex is a composite, which I don't believe anyone uses in practice. – Marketable

Answer 2

Strangely enough, nobody pointed out how to calculate how many bytes is taking one Unicode char. Here is the rule for UTF-8 encoded strings:

Binary    Hex          Comments
0xxxxxxx  0x00..0x7F   Only byte of a 1-byte character encoding
10xxxxxx  0x80..0xBF   Continuation byte: one of 1-3 bytes following the first
110xxxxx  0xC0..0xDF   First byte of a 2-byte character encoding
1110xxxx  0xE0..0xEF   First byte of a 3-byte character encoding
11110xxx  0xF0..0xF7   First byte of a 4-byte character encoding

So the quick answer is: it takes 1 to 4 bytes, depending on the first one which will indicate how many bytes it'll take up.

Policewoman

I believe the maximum Hex value for a 4-byte character is 0xF7 (not 0xF4). – Manutius

Thank you so much! I was just control+f'ing through the IETF standard, and I didn't find anything about encoding, and the article I was reading didn't go into enough detail to tell how many bits are used to represent the number of trailing code points per "character". – Posology

This is now on the second page of my "introduction for new team members" cheat sheet, along with the hilarious first two comments – Fauteuil

0xF4 was not a mistake but a clarification. Unicode codepoints are in the range 0-0x10ffff so the last codepoint is coded as F4 8F BF BF. – Pleomorphism

@Manutius You are correct in principle, but UTF-8 does not use all of the available space, for compatibility with UTF-16. – Didynamous

Answer 3

I know this question is old and already has an accepted answer, but I want to offer a few examples (hoping it'll be useful to someone).

As far as I know old ASCII characters took one byte per character.

Right. Actually, since ASCII is a 7-bit encoding, it supports 128 codes (95 of which are printable), so it only uses half a byte (if that makes any sense).

How many bytes does a Unicode character require?

Unicode just maps characters to codepoints. It doesn't define how to encode them. A text file does not contain Unicode characters, but bytes/octets that may represent Unicode characters.

I assume that one Unicode character can contain every possible character from any language - am I correct?

No. But almost. So basically yes. But still no.

So how many bytes does it need per character?

Same as your 2nd question.

And what do UTF-7, UTF-6, UTF-16 etc mean? Are they some kind Unicode versions?

No, those are encodings. They define how bytes/octets should represent Unicode characters.

A couple of examples. If some of those cannot be displayed in your browser (probably because the font doesn't support them), go to http://codepoints.net/U+1F6AA (replace 1F6AA with the codepoint in hex) to see an image.

1. • U+0061 LATIN SMALL LETTER A: a
     • Nº: 97
     • UTF-8: 61
     • UTF-16: 00 61
2. • U+00A9 COPYRIGHT SIGN: ©
     • Nº: 169
     • UTF-8: C2 A9
     • UTF-16: 00 A9
   • U+00AE REGISTERED SIGN: ®
     • Nº: 174
     • UTF-8: C2 AE
     • UTF-16: 00 AE
3. • U+1337 ETHIOPIC SYLLABLE PHWA: ጷ
     • Nº: 4919
     • UTF-8: E1 8C B7
     • UTF-16: 13 37
   • U+2014 EM DASH: —
     • Nº: 8212
     • UTF-8: E2 80 94
     • UTF-16: 20 14
   • U+2030 PER MILLE SIGN: ‰
     • Nº: 8240
     • UTF-8: E2 80 B0
     • UTF-16: 20 30
   • U+20AC EURO SIGN: €
     • Nº: 8364
     • UTF-8: E2 82 AC
     • UTF-16: 20 AC
   • U+2122 TRADE MARK SIGN: ™
     • Nº: 8482
     • UTF-8: E2 84 A2
     • UTF-16: 21 22
   • U+2603 SNOWMAN: ☃
     • Nº: 9731
     • UTF-8: E2 98 83
     • UTF-16: 26 03
   • U+260E BLACK TELEPHONE: ☎
     • Nº: 9742
     • UTF-8: E2 98 8E
     • UTF-16: 26 0E
   • U+2614 UMBRELLA WITH RAIN DROPS: ☔
     • Nº: 9748
     • UTF-8: E2 98 94
     • UTF-16: 26 14
   • U+263A WHITE SMILING FACE: ☺
     • Nº: 9786
     • UTF-8: E2 98 BA
     • UTF-16: 26 3A
   • U+2691 BLACK FLAG: ⚑
     • Nº: 9873
     • UTF-8: E2 9A 91
     • UTF-16: 26 91
   • U+269B ATOM SYMBOL: ⚛
     • Nº: 9883
     • UTF-8: E2 9A 9B
     • UTF-16: 26 9B
   • U+2708 AIRPLANE: ✈
     • Nº: 9992
     • UTF-8: E2 9C 88
     • UTF-16: 27 08
   • U+271E SHADOWED WHITE LATIN CROSS: ✞
     • Nº: 10014
     • UTF-8: E2 9C 9E
     • UTF-16: 27 1E
   • U+3020 POSTAL MARK FACE: 〠
     • Nº: 12320
     • UTF-8: E3 80 A0
     • UTF-16: 30 20
   • U+8089 CJK UNIFIED IDEOGRAPH-8089: 肉
     • Nº: 32905
     • UTF-8: E8 82 89
     • UTF-16: 80 89
4. • U+1F4A9 PILE OF POO: 💩
     • Nº: 128169
     • UTF-8: F0 9F 92 A9
     • UTF-16: D8 3D DC A9
   • U+1F680 ROCKET: 🚀
     • Nº: 128640
     • UTF-8: F0 9F 9A 80
     • UTF-16: D8 3D DE 80

Okay I'm getting carried away...

Fun facts:

• If you're looking for a specific character, you can copy&paste it on http://codepoints.net/.
• I wasted a lot of time on this useless list (but it's sorted!).
• MySQL has a charset called "utf8" which actually does not support characters longer than 3 bytes. So you can't insert a pile of poo, the field will be silently truncated. Use "utf8mb4" instead.
• There's a snowman test page (unicodesnowmanforyou.com).

Evincive

The code units in UTF-16 are 16 bit wide. You showed them with a space in the middle, which is misleading. The UTF-16 representation for © should rather be 00A9 instead of 00 A9 (which would be UTF-16BE). – Copyist

What's the difference? Doesn't BE stand for big endian? He wrote it in big endian, and so a file written in big endian UTF-16 would be the same as UTF-16BE, right? – Abuzz

Corrections: 1) ASCII is 7 bits, a byte is 8 bits, so it is much more than half. 2) Unicode does define how to encode the code points. UTF-8, UTF-16 and UTF-32 are defined in the Unicode Standard. – Reproduce

@JonathanRosenne I think s/he meant it only uses half of the possible values representable with 8 bits, not that it uses half of the bits. – Item

@Aritz It might have been the intention. – Reproduce

I really like the examples. They highlight why one may prefer UTF-16 over UTF-8, for instance. Developers of different software may select different encodings based upon which Unicode characters are more likely to be used. In China/Japan for instance, UTF-16 (2-bytes) makes more sense than UTF-8 for them, because the same characters often would need twice as many bytes to encode in UTF-8 – Parabolize

last time I checked ⅞ ≠ ½. – Dilative

@Dilative Are you really one of those guys that when you see someone say 7 bits is half of 8 bits, you think "ooh, I'm smart. 7 isn't half of 8. 4 is half of 8", but then realize you forgot how binary works? 7bits is half of 8bits. 6bits is half of 7bits. 5 bits is half of 6 bits. 4 bits is half of 5 bits. 3 bits is half of 4 bits. 2 bits is half of 3 bits. 1 bit is half of 2 bits. Maybe you only understood the last one. – Dekaliter

Answer 4

Simply speaking Unicode is a standard which assigned one number (called code point) to all characters of the world (Its still work in progress).

Now you need to represent this code points using bytes, thats called character encoding. UTF-8, UTF-16, UTF-6 are ways of representing those characters.

UTF-8 is multibyte character encoding. Characters can have 1 to 6 bytes (some of them may be not required right now).

UTF-32 each characters have 4 bytes a characters.

UTF-16 uses 16 bits for each character and it represents only part of Unicode characters called BMP (for all practical purposes its enough). Java uses this encoding in its strings.

Mourn

Unicode is a 21-bit code set and 4 bytes is sufficient to represent any Unicode character in UTF-8. UTF-16 uses surrogates to represent characters outside the BMP (basic multilingual plane); it needs either 2 or 4 bytes to represent any valid Unicode character. UCS-2 was the 16-bit only variant of UTF-16 without support for surrogates or characters outside the BMP. – Penile

You are correct. UTF-8 original one had 6 bytes to accommodate a 32 bit. I actually didn't want to complicate things much as he was already confused with wiki doc :) – Mourn

This answer states that UTF-16 cannot encode BMP code points. This is incorrect, as these can be encoded just as they can in UTF-8 using surrogate pairs. (You must be thinking of the outdated UCS-2, before Unicode 2.0 came out, which encoded 16-bit code points only.) Also, Java doesn't quite use UTF-16, it uses a modified form thereof where the code point 0 is encoded differently. – Trista

@Trista - It is the opposite. The answer says that UTF-16 represents the BMP. – Rookie

I mistyped; I had meant to say "non-BMP". The error in the answer is that it says that UTF-16 represents BMP characters, which is inaccurate. UTF-16 can encode all unicode characters-- non-BMP characters are encoded via surrogate pairs. Perhaps the answerer was confused with UCS-2. – Trista

Answer 5

In Unicode, every character is represented by an integer from zero to 0x10FFFF. Doing this naively in 32-bit integers is called the UTF-32 encoding. To be less wasteful, UTF-8 and UTF-16 are encodings that require less space for the lower codepoints.

Note that what is called UTF-16 in implementations is often really just UCS2: the subset of codepoints that UTF-16 can fit in 32 bits.

The storage requirements are as follows.

In UTF-8:

1 byte:       0 -     7F  (ASCII)
2 bytes:     80 -    7FF  (all European plus some Middle Eastern)
3 bytes:    800 -   FFFF  (multilingual plane incl. the top 1792 and private-use)
4 bytes:  10000 - 10FFFF

In UTF-16:

2 bytes:      0 -   D7FF  (multilingual plane except the top 1792 and private-use)
4 bytes:   D800 - 10FFFF

In UTF-32:

4 bytes:      0 - 10FFFF

10FFFF is the last unicode codepoint by definition, and it's defined that way because it's UTF-16's technical limit.

It is also the largest codepoint UTF-8 can encode in 4 byte, but the idea behind UTF-8's encoding also works for 5 and 6 byte encodings to cover codepoints until 7FFFFFFF, ie. half of what UTF-32 can.

Marketable

Answer 6

In Unicode the answer is not easily given. The problem, as you already pointed out, are the encodings.

Given any English sentence without diacritic characters, the answer for UTF-8 would be as many bytes as characters and for UTF-16 it would be number of characters times two.

The only encoding where (as of now) we can make the statement about the size is UTF-32. There it's always 32bit per character, even though I imagine that code points are prepared for a future UTF-64 :)

What makes it so difficult are at least two things:

1. composed characters, where instead of using the character entity that is already accented/diacritic (À), a user decided to combine the accent and the base character (`A).
2. code points. Code points are the method by which the UTF-encodings allow to encode more than the number of bits that gives them their name would usually allow. E.g. UTF-8 designates certain bytes which on their own are invalid, but when followed by a valid continuation byte will allow to describe a character beyond the 8-bit range of 0..255. See the Examples and Overlong Encodings below in the Wikipedia article on UTF-8.
   • The excellent example given there is that the € character (code point U+20AC can be represented either as three-byte sequence E2 82 AC or four-byte sequence F0 82 82 AC.
   • Both are valid, and this shows how complicated the answer is when talking about "Unicode" and not about a specific encoding of Unicode, such as UTF-8 or UTF-16. Strictly speaking, as pointed out in a comment, this doesn't seem to be the case any longer or was even based on a misunderstanding on my part. The quote from the updated Wikipedia article reads: Longer encodings are called overlong and are not valid UTF-8 representations of the code point.

Caaba

Answer 7

There is a great tool for calculating the bytes of any string in UTF-8: http://mothereff.in/byte-counter

Update: @mathias has made the code public: https://github.com/mathiasbynens/mothereff.in/blob/master/byte-counter/eff.js

Earplug

Answer 8

Well I just pulled up the Wikipedia page on it too, and in the intro portion I saw "Unicode can be implemented by different character encodings. The most commonly used encodings are UTF-8 (which uses one byte for any ASCII characters, which have the same code values in both UTF-8 and ASCII encoding, and up to four bytes for other characters), the now-obsolete UCS-2 (which uses two bytes for each character but cannot encode every character in the current Unicode standard)"

As this quote demonstrates, your problem is that you are assuming Unicode is a single way of encoding characters. There are actually multiple forms of Unicode, and, again in that quote, one of them even has 1 byte per character just like what you are used to.

So your simple answer that you want is that it varies.

Frauenfeld

Answer 9

Unicode is a standard which provides a unique number for every character. These unique numbers are called code points (which is just unique code) to all characters existing in the world (some's are still to be added).

For different purposes, you might need to represent this code points in bytes (most programming languages do so), and here's where Character Encoding kicks in.

UTF-8, UTF-16, UTF-32 and so on are all Character Encodings, and Unicode's code points are represented in these encodings, in different ways.

UTF-8 encoding has a variable-width length, and characters, encoded in it, can occupy 1 to 4 bytes inclusive;

UTF-16 has a variable length and characters, encoded in it, can take either 1 or 2 bytes (which is 8 or 16 bits). This represents only part of all Unicode characters called BMP (Basic Multilingual Plane) and it's enough for almost all the cases. Java uses UTF-16 encoding for its strings and characters;

UTF-32 has fixed length and each character takes exactly 4 bytes (32 bits).

Polypody

Answer 10

For UTF-16, the character needs four bytes (two code units) if it starts with 0xD800 or greater; such a character is called a "surrogate pair." More specifically, a surrogate pair has the form:

[0xD800 - 0xDBFF]  [0xDC00 - 0xDFF]

where [...] indicates a two-byte code unit with the given range. Anything <= 0xD7FF is one code unit (two bytes). Anything >= 0xE000 is invalid (except BOM markers, arguably).

See http://unicodebook.readthedocs.io/unicode_encodings.html, section 7.5.

Moil

Answer 11

Check out this Unicode code converter. For example, enter 0x2009, where 2009 is the Unicode number for thin space, in the "0x... notation" field, and click Convert. The hexadecimal number E2 80 89 (3 bytes) appears in the "UTF-8 code units" field.

Chaffer

Answer 12

From Wiki:

UTF-8, an 8-bit variable-width encoding which maximizes compatibility with ASCII;

UTF-16, a 16-bit, variable-width encoding;

UTF-32, a 32-bit, fixed-width encoding.

These are the three most popular different encoding.

• In UTF-8 each character is encoded into 1 to 4 bytes ( the dominant encoding )
• In UTF16 each character is encoded into 1 to two 16-bit words and
• in UTF-32 every character is encoded as a single 32-bit word.

Centrepiece