Symfony 2 / Doctrine user-defined DQL functions parameter have string length limit?

Question

I created my custom DQL function for Doctrine DQL:

class Translate extends FunctionNode {
    public $field;

    public function getSql(SqlWalker $sqlWalker) {
        $query = "TRANSLATE(" . $this->field->dispatch($sqlWalker) . ", 'àâäãáåÀÁÂÃÄÅçÇéèêëÉÈÊËîïìíÌÍÎÏñÑôöðòóÒÓÔÕÖùúûüÙÚÛÜýÿÝ', 'AAAAAAAAAAAACCEEEEEEEEIIIIIIIINNOOOOOOOOOOUUUUUUUUYYY')";
        return $query;
    }

    public function parse(Parser $parser) {
        $parser->match(Lexer::T_IDENTIFIER);
        $parser->match(Lexer::T_OPEN_PARENTHESIS);
        $this->field = $parser->StringPrimary();
        $parser->match(Lexer::T_CLOSE_PARENTHESIS);
    }
}

It seems to work well when using it.

But if the string parameter contains more than 307 chars, it is not working. There is no error but the script is ending.

$query = $this->createQueryBuilder('...');
$query->addSelect("TRANSLATE('less than 307 chars')"); // working
$query->addSelect("TRANSLATE('more than 307 chars')"); // NOT working

How can I use more than 307 chars?

Saleswoman

Have you tried to run just SQL with more than 307 chars? Maybe it's problem in max query size. – Dilution

Yes, and there is no limitation. – Saleswoman

And if i modify my DQL function (Translate class) to directly put my string, it's working : $query = "TRANSLATE('raw string with more than 307 chars working', 'àâäãáåÀÁÂÃÄÅçÇéèêëÉÈÊËîïìíÌÍÎÏñÑôöðòóÒÓÔÕÖùúûüÙÚÛÜýÿÝ', 'AAAAAAAAAAAACCEEEEEEEEIIIIIIIINNOOOOOOOOOOUUUUUUUUYYY')"; – Saleswoman

Did you consider writing a test case that verifies if the problem lies in the generated SQL or in the RDBMS? – Comeau

What is the exception error message exactly? – Alum

Old question, don't know the DQL answer, but: seems like a simple iconv to ASCII would do about the same thing without needing to provide all characters seperately. – Morentz

You might want to have a look at DoctrineExtensions as possible solution for your original problem, which might be related to "slugification". See also https://github.com/l3pp4rd/DoctrineExtensions/blob/master/doc/sluggable.md#transliterator – Nonplus

Have you tried to store this ASCII-only text in another column instead of using this function? – Marmolada

I tried to recreate your issue and couldn't. I tried 'SELECT TRANSLATE(c.notes) from DemoBundle:Entity c where c.id = 3' where 'c.notes' was a long text field. I also tried 'SELECT TRANSLATE("verylongstring...") from CranaContactsBundle:Contact c where c.id = 3'. The only change I made to your custom function class was to use the 'REPLACE()' SQL function instead of 'TRANSLATE()'. Can you post the whole query you are running, with the SQL? – Yaker

What is the script ending on? – Mouse

This has nothing to do with 307 character limit. It's probably that you have an invalid character at position 308 in your database. Try the SQL with a :substitution_variable rather than injecting text into your SQL. You'll probably get a more debuggable error message. – Nitid

Is the value from $this->field->dispatch($sqlWalker) returned in single quotes? – Suiting

Answer 1

Like wrikken had already explained, Transliteration should be done using iconv since that will save your server resources.

here's a simple function that will transliterate (convert non latin characters to their closest representations in latin charset) a string.

function transliterateString($str)
{
    $serverLocale = setlocale(LC_CTYPE, 0);
    setlocale(LC_CTYPE, 'en_US.UTF8');
    // transliterate the string using iconv
    $str = iconv('UTF-8', 'ASCII//TRANSLIT//IGNORE', $str); 
    setlocale(LC_CTYPE, $serverLocale); //  return the locale to what it was before
    return $str;
}   

Usage:

$string = "café"; 
echo $string;
echo transliterateString($string);

Above will output the below:

café
cafe

Divisionism