Unicode-ready wordsearch - Question

Question

Is this code OK? I don't really have a clue which normalization-form I should us (the only thing I noticed is with NFD I get a wrong output).

#!/usr/local/bin/perl
use warnings;
use 5.014;
use utf8;
binmode STDOUT, ':encoding(utf-8)';

use Unicode::Normalize;
use Unicode::Collate::Locale;
use Unicode::GCString;

my $text = "my taxt täxt";
my %hash;

while ( $text =~ m/(\p{Alphabetic}+(?:'\p{Alphabetic}+)?)/g ) { #'
    my $word = $1;
    my $NFC_word = NFC( $word );
    $hash{$NFC_word}++;
}

my $collator = Unicode::Collate::Locale->new( locale => 'DE' ); 

for my $word ( $collator->sort( keys %hash ) ) {
    my $gcword = Unicode::GCString->new( $word );
    printf "%-10.10s : %5d\n", $gcword, $hash{$word};
}

Kaliope

It doesn't matter which normalization you use as long as you use the same one for all strings that you're comparing! – Pennypennyaliner

@Kerrek That is incorrect. Both Unicode::Collate (and its subclass U::C::Locale) and Unicode::GCString are specifically designed so that normalization does not matter. – Boycott

Answer 1

Wow!! I can’t believe nobody answered this. It’s a super duper great question. You almost had it right, too. I like that you’re using Unicode::Collate::Locale and Unicode::GCString. Good for you!

The reason you are getting “wrong” output is because you are not using the Unicode::GCString class's columns method to determine the print width of the stuff you’re printing.

printf is very stupid and just counts code points, not columns, so you have to write your own pad function that takes the GCS columns into account. For example, to do it manually, instead of writing this:

 printf "%-10.10s", $gstring;

You have to write this:

 $colwidth = $gcstring->columns();
 if ($colwidth > 10) {
      print $gcstring->substr(0,10);
 } else {
     print " " x (10 - $colwidth);
     print $gcstring;
 }

See how that works?

Now normalization doesn’t matter. Ignore Kerrek’s old comment. It is very wrong. The UCA is specifically designed not to let normalization enter into the matter. You have to bend over backwards to screw than up, like by passing in normalization => undef to the constructor in case you want to use its gmatch method or some such.

Boycott

But makes it a difference for the word-counting ($hash{key}++) if I do a normalization before the counting? – Kaliope