How to properly Normalize a String with composite characters?

Question

Java Normalize already allows me to take accented characters and output non-accented characters. It does not, however, seem to deal with composite characters (Œ, Æ) very well at all.

Is there a way for Java to deal with these characters natively? I'd like to prevent having to keep a Map of these characters (as that was the reason we moved to using Normalize in the first place).

For example, an input of "Œ" should return "OE", in much the same way it already neatly decomposes characters such as "½" into "1/2".

Andean

Please elaborate It does not, however, seem to deal with composite characters (Œ, Æ) very well at all – Phraseograph

@SotiriosDelimanolis I think he wants Normalizer.normalize("Œ", Normalizer.Form.NFD).equals("OE"); to be true. Me too. – Asberry

@SotiriosDelimanolis I hope this clarifies it :) – Andean

See the diagram in unicode.org/reports/tr15/tr15-23.html. It implies that you need Normalizer.Form.NFKD instead. – Eltonelucidate

@Eltonelucidate hum, that seems not be enough (I get empty string as a result too) – Asberry

@Eltonelucidate I am using NFKD, as that DOES help for the NJ composite - but not here. – Andean

See the comments following this answer: https://mcmap.net/q/550659/-separating-unicode-ligature-characters – Eltonelucidate

@Eltonelucidate I don't quite see how that helps? The issue remained unsolved in those comments. – Andean

@WeckarE. I know, it helps in the sense that it's telling you it can't be solved ;-) – Eltonelucidate

@Eltonelucidate I choose to believe a solution not having been found yet and no solution existing are two very different things. – Andean

Yeah. The fact your question is upvoted so much means that a solution would be highly desirable to many. – Eltonelucidate

Ok, try this: lexsrv3.nlm.nih.gov/LexSysGroup/Projects/lvg/2013/docs/… - especially the bit near the end that mentions icu4j. – Eltonelucidate

AFAIK it's possible only with icu4j. – Elementary

Answer 1

TLDR; No, there is no way with native java to handle these uniformly.

Long Answer

As noted in this question, Separating Unicode ligature characters, the Java Normalizer implementation does not support all of the ligatures that exist in written language.

The reason for this is because Unicode does not support all of the ligatures that exist in written language. Ligatures are a debated subject when it comes to the storage of written language because an argument can be made that they are unimportant from a data viewpoint and that they are important from a layout view point.

The Data viewpoint claims that no information is lost and so it makes more sense to only use the decomposed forms and that the composed forms should not be in Unicode.

The Layout viewpoint claims that the composed ligature represents the proper layout of the written form of language and so should be represented in the data with a special code.

Possible Solution

I would suggest creating a Service that has an interface that handles ligatures only. Supply a concrete implementation that handles all that you currently need. In the future if new implementations are needed it will be simple to add them without modifying the original code by simply adding a new JAR to the program class-path that adds the missing ligatures.

The skeletal implementation may look like this.

Please note I have omitted the code that actually uses the ServiceLoader to locate the LigatureDecoder and LigatureEncoder implementations.

final class Ligatures {
  public static CharSequence compose ( CharSequence decomposedCharacters );
  public static CharSequence decompose ( CharSequence composedCharacters );
}

interface LigatureDecoder {
  CharSequence decompose ( CharSequence composedCharacters );
}

interface LigatureEncoder {
  CharSequence compose ( CharSequence decomposedCharacters );
}

Patency