Get multibyte character count before match with preg_match() (PREG_OFFSET_CAPTURE parameter is unhelpfully counting bytes)

Question

I'm trying to search a UTF8-encoded string using preg_match.

preg_match('/H/u', "\xC2\xA1Hola!", $a_matches, PREG_OFFSET_CAPTURE);
echo $a_matches[0][1];

This should print 1, since "H" is at index 1 in the string "¡Hola!". But it prints 2. So it seems like it's not treating the subject as a UTF8-encoded string, even though I'm passing the "u" modifier in the regular expression.

I have the following settings in my php.ini, and other UTF8 functions are working:

mbstring.func_overload = 7
mbstring.language = Neutral
mbstring.internal_encoding = UTF-8
mbstring.http_input = pass
mbstring.http_output = pass
mbstring.encoding_translation = Off

Any ideas?

Phosphorate

Answer 1

Looks like this is a "feature", see http://bugs.php.net/bug.php?id=37391

'u' switch only makes sense for pcre, PHP itself is unaware of it.

From PHP's point of view, strings are byte sequences and returning byte offset seems logical (i don't say "correct").

Darmit

Great...and they don't provide a mb_preg_replace. – Phosphorate

Be aware that the same "rules" regarding utf-8 handling applies to the 5th parameter $offset. Sample: var_dump(preg_match('/#/u', "\xc3\xa4#",$matches,0,2)); – Underskirt

php is aware of the u modifier it's listed in the manual see "u (PCRE_UTF8)" php.net/manual/en/reference.pcre.pattern.modifiers.php – Olecranon

Answer 2

Although the u modifier makes both the pattern and subject be interpreted as UTF-8, the captured offsets are still counted in bytes.

You can use mb_strlen to get the length in UTF-8 characters rather than bytes:

$str = "\xC2\xA1Hola!";
preg_match('/H/u', $str, $a_matches, PREG_OFFSET_CAPTURE);
echo mb_strlen(substr($str, 0, $a_matches[0][1]));

Biff

"The u modifier is only to get the pattern interpreted as UTF-8, not the subject." This is not true. Compare e.g. preg_split('//', .) with preg_split('//u', .). Since this "x is interpret as UTF-8" is a bit vague, see this for the actual effects of the unicode mode. – Fumy

According to nl1.php.net/manual/en/… the u modifier has effect on both the pattern and the input. – Keiko

@LukaRamishvili Some people are of the opinion that it sucks at many things. – Stuffed

OK, php sucks at Unicode, but maybe with one constraint now: version<7.0. UString is coming wiki.php.net/rfc/ustring. UString aims to tackle the issues of working with Unicode strings – Driest

Fun fact, this produces different output depending upon your version. 3v4l.org/iHl4a – Cords

@tomalak and next ones. Of course, php doesn't manage unicode, because it works on bytes if you use old functions like substr, strlen, etc., but it is fully managed since a very long time via the extension mbstring, enabled by default in many distributions and servers. This is a choice to maintain backward compatibility. – Particularly

I have had NO TROUBLE with UTF-8 in PHP since I started converting all my old sites to Unicode 4-5 years ago. – Extended

@Tomalak "Man, it's 2019 and PHP still sucks abysmally at Unicode." Please confirm. – Urine

Answer 3

Try adding this (*UTF8) before the regex:

preg_match('(*UTF8)/H/u', "\xC2\xA1Hola!", $a_matches, PREG_OFFSET_CAPTURE);

Magic, thanks to a comment in https://www.php.net/manual/function.preg-match.php#95828

Suhail

Interesting, although I think you need the initial / before the (*UTF8). This doesn't work on my system, but it might on others. What does this output when you do echo $a_matches[0][1];? – Phosphorate

I used it like this on PHP 5.4.29, works like a charm: preg_match_all('/(*UTF8)[^A-Za-z0-9\s]/', $txt, $matches); – Gantry

Doesn't work for me on either PHP 5.6 or PHP 7 on Ubuntu 16.04. (*UTF8) before delimiter is an error, after has no effect. I suspect that it depends on how/where you got your php, specifically the settings that libpcre* was compiled with. – Severus

Does not change the offsets for me, but that's an interesting thing to know. The original documentation for that "feature" is: pcre.org/pcre.txt – Floorman

Answer 4

Excuse me for necroposting, but may be somebody will find it useful: code below can work both as replacement for preg_match and preg_match_all functions and returns correct matches with correct offset for UTF8-encoded strings.

     mb_internal_encoding('UTF-8');

     /**
     * Returns array of matches in same format as preg_match or preg_match_all
     * @param bool   $matchAll If true, execute preg_match_all, otherwise preg_match
     * @param string $pattern  The pattern to search for, as a string.
     * @param string $subject  The input string.
     * @param int    $offset   The place from which to start the search (in bytes).
     * @return array
     */
    function pregMatchCapture($matchAll, $pattern, $subject, $offset = 0)
    {
        $matchInfo = array();
        $method    = 'preg_match';
        $flag      = PREG_OFFSET_CAPTURE;
        if ($matchAll) {
            $method .= '_all';
        }
        $n = $method($pattern, $subject, $matchInfo, $flag, $offset);
        $result = array();
        if ($n !== 0 && !empty($matchInfo)) {
            if (!$matchAll) {
                $matchInfo = array($matchInfo);
            }
            foreach ($matchInfo as $matches) {
                $positions = array();
                foreach ($matches as $match) {
                    $matchedText   = $match[0];
                    $matchedLength = $match[1];
                    $positions[]   = array(
                        $matchedText,
                        mb_strlen(mb_strcut($subject, 0, $matchedLength))
                    );
                }
                $result[] = $positions;
            }
            if (!$matchAll) {
                $result = $result[0];
            }
        }
        return $result;
    }

    $s1 = 'Попробуем русскую строку для теста';
    $s2 = 'Try english string for test';

    var_dump(pregMatchCapture(true, '/обу/', $s1));
    var_dump(pregMatchCapture(false, '/обу/', $s1));

    var_dump(pregMatchCapture(true, '/lish/', $s2));
    var_dump(pregMatchCapture(false, '/lish/', $s2));

Output of my example:

    array(1) {
      [0]=>
      array(1) {
        [0]=>
        array(2) {
          [0]=>
          string(6) "обу"
          [1]=>
          int(4)
        }
      }
    }
    array(1) {
      [0]=>
      array(2) {
        [0]=>
        string(6) "обу"
        [1]=>
        int(4)
      }
    }
    array(1) {
      [0]=>
      array(1) {
        [0]=>
        array(2) {
          [0]=>
          string(4) "lish"
          [1]=>
          int(7)
        }
      }
    }
    array(1) {
      [0]=>
      array(2) {
        [0]=>
        string(4) "lish"
        [1]=>
        int(7)
      }
    }

Kneehigh

Can you explain what your code does instead of just pasting a code dump? And how does this answer the question? – Election

It does exactly what described in comments and returns CORRECT string offsets. It is the subject of the question. No idea why I had -2 for my answer. It is working for me. – Kneehigh

Well, that's why you should include an explanation of what your code does. People don't get what you are trying to do here. – Election

Edit my answer, added tests. – Kneehigh

To use $offset as (characters) instead of (bytes), you can add this near the top of the function: if ($offset) { $offset = strlen(mb_substr($subject, 0, $offset)); } – Bunde

Thanks. After searching on multiple questions, this is the only answer that does the trick. – Carrigan

An old comment of "necroposting", but still useful! Thank you @GuyFawkes, this helped with my current mess of code I'm working through. Cheers, jz – Unsteady

Answer 5

You can calculate the real UTF-8 offset by cutting the string to the offset returned by the preg_mach with the byte-counting substr and then measuring this prefix with the correct-counting mb_strlen.

$utf8Offset = mb_strlen(substr($text, 0, $offsetFromPregMatch), 'UTF-8');

Epiphragm

Answer 6

If all you want to do is find the multi-byte safe position of H try mb_strpos()

mb_internal_encoding('UTF-8');
$str = "\xC2\xA1Hola!";
$pos = mb_strpos($str, 'H');
echo $str."\n";
echo $pos."\n";
echo mb_substr($str,$pos,1)."\n";

Output:

¡Hola!
1
H

Elimination

That was just a simplified example, but this may be useful for others. – Phosphorate

Answer 7

I wrote small class to convert offsets returned by preg_match to proper utf offsets:

final class NonUtfToUtfOffset
{
    /** @var int[] */
    private $utfMap = [];

    public function __construct(string $content)
    {
        $contentLength = mb_strlen($content);

        for ($offset = 0; $offset < $contentLength; $offset ++) {
            $char = mb_substr($content, $offset, 1);
            $nonUtfLength = strlen($char);

            for ($charOffset = 0; $charOffset < $nonUtfLength; $charOffset ++) {
                $this->utfMap[] = $offset;
            }
        }
    }

    public function convertOffset(int $nonUtfOffset): int
    {
        return $this->utfMap[$nonUtfOffset];
    }
}

You can use it like that:

$content = 'aą bać d';
$offsetConverter = new NonUtfToUtfOffset($content);

preg_match_all('#(bać)#ui', $content, $m, PREG_OFFSET_CAPTURE);

foreach ($m[1] as [$word, $offset]) {
    echo "bad: " . mb_substr($content, $offset, mb_strlen($word))."\n";
    echo "good: " . mb_substr($content, $offsetConverter->convertOffset($offset), mb_strlen($word))."\n";
}

https://3v4l.org/8Y32J

Zulmazulu

Answer 8

You might want to look at T-Regx library.

pattern('/Hola/u')->match('\xC2\xA1Hola!')->first(function (Match $match) 
{
    echo $match->offset();     // characters
    echo $match->byteOffset(); // bytes
});

This $match->offset() is UTF-8 safe offset.

Hoxsie

Answer 9

The problem was solved to me just by using casual substr instead of expected mb_substr (PHP 7.4).

The mb_substr together with preg_match_all / PREG_OFFSET_CAPTURE (despite using or not using /u modifier)resulted in incorrect position when text contained euro sign symbol (€).

Also iconv and utf8_encode did not help, and I was not able to use htmlentities.

Just reverting to simple substr helped, and it worked with € and other characters correctly.

Unsupportable

So this answer is just confirming that previously posted answers worked for you? – Pyramidal

Answer 10

I think working with PREG_OFFSET_CAPTURE in this case only creates more work.

Demo of below scripts.

If the pattern only contains literal characters, then preg_ is overkill, just use mb_strpos() and bear in mind that the returned value will be false if the needle is not found in the haystack.

var_export(mb_strpos($str, 'H')); // 1

If you know that the needle will exist in the haystack, you can use preg_match_all() with the marvellous \G (continue) metacharacter and \X (multibyte any character) metacharacter.

echo preg_match_all('/\G(?![A-Z])\X/u', $str); // 1
// if needle not found, will return the mb length of haystack

If you don't know if the needle will exist in the haystack, just check if the returned count is equal to the multibyte length of the input string.

$mbLength = preg_match_all('/\G(?![A-Z])\X/u', $str, $m);
var_export(mb_strlen($str) !== $mbLength ? $mbLength : 'not found');

But if you are going to call an extra mb_ function anyhow, then make just one match, check if a match was made, and measure its multibyte length if so.

var_export(
    preg_match('/\X*?(?=[A-Z])/u', $str, $m) ? mb_strlen($m[0]) : 'not found' 
);

---

All this said, I've never seen the need to count the multibyte position of something unless the greater task was to isolate or replace a substring. If this is the case, avoid this step entirely and just use preg_match() or preg_replace() to more directly serve your needs.

Pyramidal