Python template safe substitution with the custom double-braces format

Question

I am trying to substitute variables in the format {{var}} with Python's Template.

from string import Template

class CustomTemplate(Template):
    delimiter = '{{'
    pattern = r'''
    \{\{(?:
    (?P<escaped>\{\{)|
    (?P<named>[_a-z][_a-z0-9]*)\}\}|
    (?P<braced>[_a-z][_a-z0-9]*)\}\}|
    (?P<invalid>)
    )
    '''

replacement_dict = {
    "test": "hello"
}

tpl = '''{
    "unaltered": "{{foo}}",
    "replaced": "{{test}}"
}'''

a = CustomTemplate(tpl)
b = a.safe_substitute(replacement_dict)

print(b)

The output:

{
    "unaltered": "{{foo",
    "replaced": "hello"
}

As you can see, the {{foo}} variable (which is not part of the replacement) has the closing brackets chopped off. I think it's the way the regex is written (the closing \}\})?

I want to solve this with Template, not with any other external libraries.

Cheboksary

You have to post the full regex and how the named groups relate to the replacement. – Richardson

Could try to get rid of the escaped and invalid groups. It looks like {{foo}} matches {{ and invalid (which is empty). So whatever it does when it matches could be the problem. – Richardson

The documentation of the Template library says that four named groups need to be supplied. – Cheboksary

Answer 1

I'm not sure how you got this to work. On linux, in python 3.4.3 (I thought I got this working with some version of 2.7) I needed to make tpl a string

tpl = '''
    "unaltered": "{{foo}}",
    "replaced": "{{test}}"
'''

to avoid getting a TypeError

>>> tpl = '''
...     "unaltered": "{{foo}}",
...     "replaced": "{{test}}"
... '''
>>> a = CustomTemplate(tpl)
>>> a.template
'\n    "unaltered": "{{foo}}",\n    "replaced": "{{test}}"\n'
>>> b = a.safe_substitute(replacement_dict)
>>> b
'\n    "unaltered": "{{foo}}",\n    "replaced": "hello"\n'

When I do this, {{foo}} is unaltered.

I tried the above code, and it looks like the code does not actually work with python 2.7.6. I'll see if I can find a way to make it work with 2.7.6, since that seems to be a common version with recent linux distros.

update:

Looks like this was a known bug as of 2007. http://bugs.python.org/issue1686 Far as I can tell, it was applied to python 3.2 in 2010, and python 2.7 in 2014. As far as getting this to work, you can either apply the patch for issue 1686, or you can override safe_substitute() in your class with the actual source code from this patch https://hg.python.org/cpython/file/8a98ee6baa1e/Lib/string.py.

This code works in 2.7.6, and 3.4.3

from string import Template
class CustomTemplate(Template):
    delimiter = '{{'
    pattern = r'''
    \{\{(?:
    (?P<escaped>\{\{)|
    (?P<named>[_a-z][_a-z0-9]*)\}\}|
    (?P<braced>[_a-z][_a-z0-9]*)\}\}|
    (?P<invalid>)
    )
    '''

    def safe_substitute(self, *args, **kws):
        if len(args) > 1:
            raise TypeError('Too many positional arguments')
        if not args:
            mapping = kws
        elif kws:
            mapping = _multimap(kws, args[0])
        else:
            mapping = args[0]
        # Helper function for .sub()
        def convert(mo):
            named = mo.group('named') or mo.group('braced')
            if named is not None:
                try:
                    # We use this idiom instead of str() because the latter
                    # will fail if val is a Unicode containing non-ASCII
                    return '%s' % (mapping[named],)
                except KeyError:
                    return mo.group()
            if mo.group('escaped') is not None:
                return self.delimiter
            if mo.group('invalid') is not None:
                return mo.group()
            raise ValueError('Unrecognized named group in pattern',
                             self.pattern)
        return self.pattern.sub(convert, self.template)

replacement_dict = {
    "test": "hello"
}

tpl = '''{
    "escaped": "{{{{",
    "unaltered": "{{foo}}",
    "replaced": "{{test}}",
    "invalid": "{{az"
}'''

a = CustomTemplate(tpl)
b = a.safe_substitute(replacement_dict)

print (b)

results:

Python 2.7.6 (default, Jun 22 2015, 17:58:13) 
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import template
{
    "escaped": "{{",
    "unaltered": "{{foo}}",
    "replaced": "hello",
    "invalid": "{{az"
}
>>> 

Forsake

That's very strange. My results differ from yours. I have posted the full example above. Can you please revisit? – Cheboksary

Answer 2

Problem

• developer reggie wishes to use custom placeholder delimiters with python PEP292 Template strings.

Solution (workaround)

• developer reggie can change the placeholder prefix character(s).
• this approach permits using the delimiter syntax from a completely different programming language.
• this approach fulfills a primary purpose of changing placeholder syntax, to prevent delimiter collision problems.
• this approach provides the most straightforward case of subclassing string.Template.

Pitfalls

• this approach does not support double-curly brace placeholders like {{var}}.
• this approach does not help in cases where developer cannot change the template syntax at will.

Example

import string
pass

class TemplateRubyish(string.Template):
  delimiter = '#'
  idpattern = r'[a-z][\w\,\+\=\:\-\.\x2f\x5c\*\(\)\[\]\x7c]*'

class TemplatePerlish(string.Template):
  delimiter = 'qq'
  idpattern = r'[a-z][\w\,\+\=\:\-\.\x2f\x5c\*\(\)\[\]\x7c]*'

replacement_dict = {}
replacement_dict.update({
  "age":      "34",
  "fname":    "Homer",
  "lname":    "Simpson",
});
pass

##
vout     = TemplateRubyish("""\
  Greetings #{fname} #{lname},
  You are #{age}ish years old.
""").safe_substitute(replacement_dict);
print(vout)
pass

##
vout     = TemplatePerlish("""\
  Greetings qq{fname} qq{lname},
  You are qq{age}ish years old.
""").safe_substitute(replacement_dict);
print(vout)
pass

Result

    Greetings Homer Simpson,
    You are 34ish years old.

    Greetings Homer Simpson,
    You are 34ish years old.

Clime