Separating Unicode ligature characters

Question

Throughout the vast number of unicode characters, there are some that actually represent more than one character, like the U+FB00 ligature ff for two 'f' characters. Is there any way easy to convert characters like these into multiple single characters? Preferably something available in the standard Java API, but I can refer to an external library if need be.

Florio

I took the freedom to add the keyword ligature to your question. :) – Iritis

Thanks - I wasn't sure what they were called. :-) – Florio

not grapheme for fundamental unit? – Gamba

Answer 1

U+FB00 is a compatibility character. Normally Unicode doesn't support separate codepoints for ligatures (arguing that it's a layout decision if and when a ligature should be used and should not influence how the data is stored). A few of those still exist to allow round-trip conversion compatibility with older encodings that do represent ligatures as separate entities.

Luckily, the information which characters the ligature represents is present in the Unicode data file and most capable string handling systems have that data built-in.

In Java, you'll need to use the Normalizer class and the NFKC form:

String ff ="\uFB00";
String normalized = Normalizer.normalize(ff, Form.NFKC);
System.out.println(ff + " = " + normalized);

This will print

ff = ff

Spatial

@nonoitall: NFKD is no panacea: there are plenty of ligatures and other notionally combined forms it just does not work on at all. For example, it will not manage to decompose ß or ẞ to SS (even those there is a casefold thither!), nor Æ to AE or æ to ae, nor Œ to OE or œ to oe. It is also useless for turning ð or đ into d or ø into o. For all those things, you need the UCA (Unicode Collation Algorithm), not NFKD. NFD/NFKD also both have the annoying property of destroying singletons, if this matters to you. – Norrisnorrv

@tchrist: my understanding is that those decompositions you mention should not be done. They are not simply ligatures in the typographical sense, but real separate characters that are used differently! ß can be decomposed to ss if necessary (for example if you can only store ASCII), but they are not equivalent. The ff Ligature, on the other hand is only a typographical ligature. – Spatial

@tchrist, how did you apply UCA in java, is there an example of how to break down œ to oe for example? – Stockholm

Answer 2

The process you are talking about is called Normalization and is specified in the Unicode Normalization Forms technical note.

There is a class in the Java SE class library called java.text.Normalizer which implements this process. However, you need to read the Unicode document linked above to figure out which of the "normalization forms" you need to use to get the result you want. It is not straightforward ....

Stoltzfus

Answer 3

You could try the java.text.Normalizer, but I am not really sure if that works for ligatures.

Caffrey