How to know the preferred display width (in columns) of Unicode characters?

Question

In different encodings of Unicode, for example UTF-16le or UTF-8, a character may occupy 2 or 3 bytes. Many Unicode applications doesn't take care of display width of Unicode chars just like they are all Latin letters. For example, in 80-column text, which should contains 40 Chinese characters or 80 Latin letters in one line, but most application (like Eclipse, Notepad++, and all well-known text editors, I dare if there's any good exception) just count each Chinese character as 1 width as Latin letter. This certainly make the result format ugly and non-aligned.

For example, a tab-width of 8 will get the following ugly result (count all Unicode as 1 display width):

apple   10
banana  7
苹果      6
猕猴桃     31
pear    16

However, the expected format is (Count each Chinese character as 2 width):

apple   10
banana  7
苹果    6
猕猴桃  31
pear    16

The improper calculation on display width of chars make these editors totally useless when doing tab-align, and line wrapping and paragraph reformat.

Though, the width of a character may vary between different fonts, but in all cases of Fixed-size terminal font, Chinese character is always double width. That is to say, in despite of font, each Chinese character is preferred to display in 2 width.

One of solution is, I can get the correct width by convert the encoding to GB2312, in GB2312 encoding each Chinese character takes 2 bytes. however, some Unicode characters doesn't exist in GB2312 charset (or GBK charset). And, in general it's not a good idea to compute the display width from the encoded size in bytes.

To simply calculate all character in Unicode in range of (\u0080..\uFFFF) as 2 width is also not correct, because there're also many 1-width chars scattered in the range.

There's also difficult when calculate the display width of Arabic letters and Korean letters, because they construct a word/character by arbitrary number of Unicode code points.

So, the display width of a Unicode code point maybe not an integer, I deem that is ok, they can be grounded to integer in practice, at least better than none.

So, is there any attribute related to the preferred display width of a char in Unicode standard? Or any Java library function to calculate the display width?

Bayern

I guess the problem is that the width is dependent on the font, so you either count characters (which has the problem you mention), or you calculate string-in-font length, and use that. – Catabasis

Answer 1

Sounds like you're looking for something like wcwidth and wcswidth, defined in IEEE Std 1003.1-2001, but removed from ISO C:

The wcwidth() function shall determine the number of column positions required for the wide character wc. The wcwidth() function shall either return 0 (if wc is a null wide-character code), or return the number of column positions to be occupied by the wide-character code wc, or return -1 (if wc does not correspond to a printable wide-character code).

Markus Kuhn wrote an open source version, wcwidth.c, based on Unicode 5.0. It includes a description of the problem, and an acknowledgement of the lack of standards in the area:

In fixed-width output devices, Latin characters all occupy a single "cell" position of equal width, whereas ideographic CJK characters occupy two such cells. Interoperability between terminal-line applications and (teletype-style) character terminals using the UTF-8 encoding requires agreement on which character should advance the cursor by how many cell positions. No established formal standards exist at present on which Unicode character shall occupy how many cell positions on character terminals. These routines are a first attempt of defining such behavior based on simple rules applied to data provided by the Unicode Consortium. [...]

It implements the following rules:

• The null character (U+0000) has a column width of 0.
• Other C0/C1 control characters and DEL will lead to a return value of -1.
• Non-spacing and enclosing combining characters (general category code Mn or Me in the Unicode database) have a column width of 0.
• SOFT HYPHEN (U+00AD) has a column width of 1.
• Other format characters (general category code Cf in the Unicode database) and ZERO WIDTH SPACE (U+200B) have a column width of 0.
• Hangul Jamo medial vowels and final consonants (U+1160-U+11FF) have a column width of 0.
• Spacing characters in the East Asian Wide (W) or East Asian Full-width (F) category as defined in Unicode Technical Report #11 have a column width of 2.
• All remaining characters (including all printable ISO 8859-1 and WGL4 characters, Unicode control characters, etc.) have a column width of 1.

Wilinski

+1 Great explanation. Though Java doesn't have a wcwidth() function, but it's very easy to write it your own, follow the guide. See also @bobince's answer for more information on East Asian Width (N/W/H/F/Na). – Bayern

There's since Unicode 5.2 (2009) another range with Hangul Jamo medial vowels and final consonants at U+D7B0..U+D7FF. – Bronchiole

From what I can see, this implementation is returning 1 for a whole slew of codepoints such as U+2501 ("━") which are clearly two columns wide. I'm confused. – Kaiser

There's a comment at the top of the code clarifying that it is mainly intended for CJK and doesn't support multi-cell widths for characters not identified as such by the Unicode database: "Much less clear is the choice of width for the Not East Asian (Neutral) class. Existing practice does not dictate a width for any of these characters. [...]" – Wilinski

Answer 2

You are confusing code points, graphemes and encoding.

The encoding is how code points are converted into an octet stream for storage, transmission or processing. Both UTF-8 and UTF-16 are variable width encodings, with different code points needing a different number of octets (for UTF-8 anything from 1 to, IIRC, 6 and UTF-16 either 2 or 4).

Graphemes are "what we see as a character", these are what are displayed. One code point (e.g. LATIN LOWER CASE A) for one grapheme, but in other cases multiple code points might be needed (e.g. LATIN LOWER CASE A, COMBINING ACUTE and COMBINING UNDERSCORE to get an lower case with acute and underscore as used in Kwakwala). In some cases there is more than one combination of code points to create the same grapheme (e.g. LATIN LOWER CASE A WITH ACUTE and COMBINING UNDERSCORE), this is "normalisation",

I.e. the length of the encoding of a single grapheme will depend on the encoding and normalisation.

The display width of the grapheme will depend on the typeface, style and size independently of the encoding length.

For more information, see Wikipedia on Unicode and Unicode's home. There are also some excellent books, perhaps most notably "Fonts & Encodings" by Yannis Haralambous, O'Reilly.

Rotten

+1. Just a minor remark: a valid UTF-8 encoded code points takes up to 4 octets. – Samualsamuel

@Nemanja the original definition (for the original 31bit Universal character set) or the refeined RFC 3629/Unicode definition for 24bit Unicode. The latter is indeed limited to 4 octets as that is all that is needed for 24bits. – Rotten

You are right, but I'm not confused, though I didn't use the terminology correctly. You didn't get the point, I mean the fixed size terminal font here, and my question is about preferred display width, not precise display width. It's no doubt that, for example, all CJK characters take up 2 width, my question is whether Unicode gives such attribute, to handle Unicode in terminal window more correctly. Some characters (like combining) are constructed by several code points, in this case, I'd like to know whether there's defined function to calculate the preferred display width from a string. – Bayern

@谢继雷 I think you are mixing things up. Unicode defines very little about the display of a glyph, really you need to consider the typeface in use. In a fixed with typeface one expects to have all glyphs the same width, but this breaks down with CJK glyphs which really need more space than Latin glyphs. However if you look at your platform documentation there should be API functions to calculate the display space needed for a string in a given size and style of a given typeface. E.g. in Windows API one is GetTextExtentPoint32: msdn.microsoft.com/en-us/library/dd144938(VS.85).aspx – Rotten

@Rotten You are right. But it's not easy. To estimate the display width is useful to patch existing editors for better line wrapping, 50% more accuracy will result in a great improvement in the paragraph format. and, while GetTextExtentPoint32 is not portable and involves the graphics layer to refactor, just too hard to implement on those i18n-unfriendly codes. – Bayern

Answer 3

The Unicode property reflecting this concept is East_Asian_Width. It's not really reliable as a visual width in the context of general Unicode rendering, as non-Asian characters, combining characters etc. will fail to line up even in a monospaced font. (Your example certainly doesn't render lined-up for me.)

Java does not have the built-in ability to read this property for characters (though Android's extension does). You can get it from ICU4J if you really need it.

Monospermous

This is exactly what I want, and this property file is helpful: unicode.org/Public/UNIDATA/EastAsianWidth.txt It also shows that the varied width is randomly scattered over all. – Bayern

Answer 4

I believe that to do this correctly, you need to consider that component of the published Unicode Standard known as Unicode Standard Annex #14, the Unicode Line Breaking Algorithm.

If you were programming in Perl, what you want to know would be super easy, because Perl’s Unicode::LineBreak module implementing UAX#14 includes a class with a simple columns method that tells you the right answer for its string argument. These things work especially well on Asian languages, where absolutley nothing else will do. This module includes over 6,000 unit tests, is actively maintained, and its author is himself Asian, so it’s important to him to get these tricky bits exactly correct.

Most of the guts of the module are a library written in C. I have not looked at how to call its component C library from other languages thn Perl, but you might look into whether this might be possible.

Extricate

Answer 5

Regarding "Or any Java library function to calculate the display width?": if there is one I've never found it.

The simplest method of calulating the width of a character / string is to write it in the GNU unicode font ( http://unifoundry.com/unifont.html ) & measure the character width. Not clean, but so far it's worked for every encoding I can think of.

FWIW here's what I do:

java.awt.font.Font MONOSPACEFONT = Font.createFont(Font.TRUETYPE_FONT, 
    new File("unifont-5.1.20080907.ttf"));

java.awt.font.FontRenderContext FRC = new FontRenderContext(null, true, true);

int charWidth =  (int) (2.0*((java.awt.geom.Rectangle2D.Float) 
    MONOSPACEFONT.getStringBounds(stringToMeasure, FRC)).width);

... this should work pretty much anywhere you deploy your JVM (it runs fine in a headless environment).

Canula

I didn't test the code, but this seems to calculate the width in pixels, but not in columns. (I've updated the question title to reflect the intention) – Bayern

Answer 6

I know this question is very old, but I had the same issue and couldn't find anything very useful, so I'll leave my solution here in case it helps someone else: there is a document that lists the widths of unicode characters in 6 different classes: https://www.unicode.org/Public/UCD/latest/ucd/EastAsianWidth.txt

To match the columns assigned in my Linux terminal I parsed that file and constructed a range of codepoints for those in the "F" and "W" categories, and merged adjacent ranges: [\u1100-\u115F\u231A-\u231B\u2329-\u232A\u23E9-\u23EC\u23F0\u23F3\u25FD-\u25FE\u2614-\u2615\u2648-\u2653\u267F\u2693\u26A1\u26AA-\u26AB\u26BD-\u26BE\u26C4-\u26C5\u26CE\u26D4\u26EA\u26F2-\u26F3\u26F5\u26FA\u26FD\u2705\u270A-\u270B\u2728\u274C\u274E\u2753-\u2755\u2757\u2795-\u2797\u27B0\u27BF\u2B1B-\u2B1C\u2B50\u2B55\u2E80-\u2E99\u2E9B-\u2EF3\u2F00-\u2FD5\u2FF0-\u303E\u3041-\u3096\u3099-\u30FF\u3105-\u312F\u3131-\u318E\u3190-\u31E3\u31EF-\u321E\u3220-\u3247\u3250-\u4DBF\u4E00-\uA48C\uA490-\uA4C6\uA960-\uA97C\uAC00-\uD7A3\uF900-\uFAFF\uFE10-\uFE19\uFE30-\uFE52\uFE54-\uFE66\uFE68-\uFE6B\uFF01-\uFF60\uFFE0-\uFFE6\U00016FE0-\U00016FE4\U00016FF0-\U00016FF1\U00017000-\U000187F7\U00018800-\U00018CD5\U00018D00-\U00018D08\U0001AFF0-\U0001AFF3\U0001AFF5-\U0001AFFB\U0001AFFD-\U0001AFFE\U0001B000-\U0001B122\U0001B132\U0001B150-\U0001B152\U0001B155\U0001B164-\U0001B167\U0001B170-\U0001B2FB\U0001F004\U0001F0CF\U0001F18E\U0001F191-\U0001F19A\U0001F200-\U0001F202\U0001F210-\U0001F23B\U0001F240-\U0001F248\U0001F250-\U0001F251\U0001F260-\U0001F265\U0001F300-\U0001F320\U0001F32D-\U0001F335\U0001F337-\U0001F37C\U0001F37E-\U0001F393\U0001F3A0-\U0001F3CA\U0001F3CF-\U0001F3D3\U0001F3E0-\U0001F3F0\U0001F3F4\U0001F3F8-\U0001F43E\U0001F440\U0001F442-\U0001F4FC\U0001F4FF-\U0001F53D\U0001F54B-\U0001F54E\U0001F550-\U0001F567\U0001F57A\U0001F595-\U0001F596\U0001F5A4\U0001F5FB-\U0001F64F\U0001F680-\U0001F6C5\U0001F6CC\U0001F6D0-\U0001F6D2\U0001F6D5-\U0001F6D7\U0001F6DC-\U0001F6DF\U0001F6EB-\U0001F6EC\U0001F6F4-\U0001F6FC\U0001F7E0-\U0001F7EB\U0001F7F0\U0001F90C-\U0001F93A\U0001F93C-\U0001F945\U0001F947-\U0001F9FF\U0001FA70-\U0001FA7C\U0001FA80-\U0001FA88\U0001FA90-\U0001FABD\U0001FABF-\U0001FAC5\U0001FACE-\U0001FADB\U0001FAE0-\U0001FAE8\U0001FAF0-\U0001FAF8\U00020000-\U0002FFFD\U00030000-\U0003FFFD]

Codepoints in this range occupy 2 columns in the terminal, others 1.

For those who object that it's a bad requirement, mine was to produce helpful error messages in console utilities that indicate the point of parse failure (similar to gcc):

% echo -n "xxxxxxxxxxxxxxxxxfoo\u306fbar/asiaasd\x08asdfadsfsdfasdfdf/ay\u306fasdfsadf" | is_class foo
Failed to parse foo at 24: 0x2f:
…xxxxxxxxxxxxxxxxfooはbar/asiaasd\basdfadsfsdfasdfd…
-------------------------^
false

To do that requires knowing how many columns the terminal will advance for each character.

Tactile