Perl substr based on bytes

Question

I'm using SimpleDB for my application. Everything goes well unless the limitation of one attribute is 1024 bytes. So for a long string I have to chop the string into chunks and save it.

My problem is that sometimes my string contains unicode character (chinese, japanese, greek) and the substr() function is based on character count not byte.

I tried to use use bytes for byte semantic or later substr(encode_utf8($str), $start, $length) but it does not help at all.

Any help would be appreciated.

Alcinia

Which version of perl you are using? – Epode

Answer 1

UTF-8 was engineered so that character boundaries are easy to detect. To split the string into chunks of valid UTF-8, you can simply use the following:

my $utf8 = encode_utf8($text);
my @utf8_chunks = $utf8 =~ /\G(.{1,1024})(?![\x80-\xBF])/sg;

Then either

# The saving code expects bytes.
store($_) for @utf8_chunks;

or

# The saving code expects decoded text.
store(decode_utf8($_)) for @utf8_chunks;

Demonstration:

$ perl -e'
    use Encode qw( encode_utf8 );

    # This character encodes to three bytes using UTF-8.
    my $text = "\N{U+2660}" x 342;

    my $utf8 = encode_utf8($text);
    my @utf8_chunks = $utf8 =~ /\G(.{1,1024})(?![\x80-\xBF])/sg;

    CORE::say(length($_)) for @utf8_chunks;
'
1023
3

Claman

@Borodin, Yes and no. It wouldn't make a difference to the regex engine, but it would make a difference to the reader. We want to match where the previous match left off, so let's not hide that. – Claman

@mob, Thanks, but it's UTF-8's designers that were clever. They specifically engineered UTF-8 so that character boundaries are easy to detect. – Claman

Answer 2

substr operates on 1-byte characters unless the string has the UTF-8 flag on. So this will give you the first 1024 bytes of a decoded string:

substr encode_utf8($str), 0, 1024;

although, not necessarily splitting the string on character boundaries. To discard any split characters at the end, you can use:

$str = decode_utf8($str, Encode::FB_QUIET);

Ciapas

This doesn't necessarily split the string on character boundaries, so the OP couldn't call decode_utf8 on an individual chunk (which might be OK). – Thyme

Incorrect; substr operates on characters (whatever their size in bytes may be), not bytes. – Downhearted

@mob: they could, passing Encode::FB_QUIET as the second parameter. – Ciapas