Java support for non-BMP Unicode characters (i.e. codepoints > 0xFFFF) in their Regular Expression Library?

Question

I'm currently using Java 6 (I don't have the option of moving to Java 7) and I'm trying to use the java.util.regex package to do pattern matching of strings that contain Unicode characters.

I know that java.lang.String supports supplemental characters (i.e. characters with codepoints > 0xFFFF) (since Java 5), but I don't see a simple way to do do pattern matching with these characters. java.util.regex.Pattern still only allows hexadecimals to be represented using 4 digits (e.g. \uFFFF)

Does anyone know if I'm missing an API here?

Deegan

Tom Christiansen (of Perl fame) has answered several questions about Unicode and Java Regexes here on SO: stackoverflow.com/users/471272/tchrist – Estelaestele

Answer 1

I've never done pattern matching with supplemental characters, but I think it's as simple as encoding them (in patterns and strings) as two 16 bits numbers (a UTF-16 surrogate pair) \unnnn\ummmm . java.util.regex should be is clever enough to interpret those two numbers (Java chars) as a single character in patterns and strings (though Java will still see them as two chars, as elements of the string).

Two links:

Java Unicode encoding

http://java.sun.com/developer/technicalArticles/Intl/Supplementary/

From the last link (refering to Java 5) :

The java.util.regex package has been updated so that both pattern strings and target strings can contain supplementary characters, which will be handled as complete units.

Note also that, if you are using UTF8 as your encoding (for your source files), you can also write them directly (see section "Representing Supplementary Characters in Source Files" in the last link).

For example:

    String pat1 = ".*\uD840\uDC00{2}.*";
    String s1  = "HI \uD840\uDC00\uD840\uDC00 BYE";
    System.out.println(s1.matches(pat1) + " len=" + s1.length());

    String pat2 = ".*\u0040\u0041{2}.*";
    String s2 = "HI \u0040\u0041\u0040\u0041 BYE";
    System.out.println(s2.matches(pat2) + " len=" + s2.length());

This, compiled with Java 6, prints

true len=11
false len=11

which agrees with the above. In the first case, we have a single code point, represented as a pair of surrogate java chars (two 16 bits chars, one suplemental Unicode character), and the {2} quantifier applies to the pair(=codepoint). In the second, we have two distinct BMP characters, the quantifier applies to the last one - hence, no match.

Notice, however, that the string length is the same (because Java measures the string length counting Java characters, not Unicode code points).

Misjoinder

I was hoping that I could encode non-BMP Unicode directly into the regex (similar to \Unnnnnn), but I guess UTF-8 is the next best thing. Thanks. – Deegan

@Jin: It's UTF-16, not UTF-8. And UTF-16 is the only way Java can internally represent those characters in a String. – Abolish

To clarify the above comments: you can either enter the code point as \unnnnn\ummmmm as a UTF16 surrogate , OR directly type the character, in the charset encoding you are using for your source code (tipically UTF-8) (but not in the \u... form !). The second alternative can be handy, but also problematic, specially if you need to share your source code. – Misjoinder

@Jim: Come JDK7, you can use \x{HHHHH}. That is the only way to make them work in character classes, for example. – Fairchild

@Leonblow: Actually, it doesn’t work in character classes. UTF‑16 bites you there. You’ll have to wait for JDK7. Meanwhile, UTF-8 is your best bet. – Fairchild

Answer 2

The easiest solution is to use a UTF-8 encoding for your source code. Then just put the characters in directly. You should never ever ever have to specify separate code units in any program.

There is still an issue with character classes, however, because Java’s lamely exposed UTF-16 internal encoding messes them up. You can’t use full Unicode until JDK7, where even then you will have to specify logical code points using an indirect \x{HHHHH} notation. You still won’t be able to have any literal code point in a charclass, but you can dodge it with \x{H..H}.

Imperfect, but it’s a lot better than it was. UTF-16 is always a compromise. Systems that use UTF-8 or UTF-32 internally don’t have these restrictions. They also never make you specify code units that aren’t identical to code points.

Fairchild