Can't use locals() in list comprehension in Python 3?

Question

The below works in Python 2 but not 3. Is there a way to access local variables in Python 3? Or an alternative solution to these cases?

[('{name_var}_{i:02d}of{maxpg:02d}.{date_var}').format(i, **locals()) 
  for i in range(start, end)]

Error in Python 3:

KeyError: 'local_var'

Below is a simpler toy example of above (works in Python 2 but not 3)

local_var = 'hello'
['{local_var}'.format(**locals()) for i in range(1)]

Error in Python 3:

KeyError: 'local_var'

Clench

Please make a clearer example for us, currently your code works in any version... – Shammer

are you using some broken shell by happenstance? the *** is not what the standard python repl shows. – Sawyor

Sorry for the confusion. I fixed the toy example to one that's representative and re-tested to confirm it works in Python 2 but not 3. The old one was only broken in pdb (for pdb-specific reasons). – Clench

On Python 3, the list comprehension has its own local scope. Variables from the outer scope won't show up in locals() unless the list comprehension uses them as closure variables. – Dogmatic

@Dogmatic Sorry for my ignorance, but what do you mean by the list comprehension using them as closures - what would be an example? – Clench

@bamboo77: If the list comprehension contains a direct name lookup for variable blah, something like [blah for thing in stuff], then the compiler will compile the code to go through the extra work needed to support accessing blah from within the comprehension. If it just calls locals, something like [locals()['blah'] for thing in stuff], the compiler doesn't activate the mechanisms needed to support the lookup. – Dogmatic

Answer 1

As explained by @user2357112 in a comment, list comprehensions have their own local scope (and thus locals() dict) in Python 3.

Compare:

>>> var=1
>>> [locals() for _ in range(1)]
[{'_': 0, '.0': <range_iterator object at 0x7f5b65cb7270>}]

With

>>> [l for l in (locals(), )]
[{'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <class '_frozen_importlib.BuiltinImporter'>, '__spec__': None, '__annotations__': {}, '__builtins__': <module 'builtins' (built-in)>, 'var': 1}]

In the first case, the function locals is called inside the list comprehension code while in the second the result of the function call is passed as an argument to the list comprehension.

The dis module confirms that:

>>> from dis import dis
>>> def f(): return [locals() for _ in range(1)]
... 
>>> dis(f)
  1           0 LOAD_CONST               1 (<code object <listcomp> at 0x7fc8173bd9c0, file "<stdin>", line 1>)
              2 LOAD_CONST               2 ('f.<locals>.<listcomp>')
              4 MAKE_FUNCTION            0
              6 LOAD_GLOBAL              0 (range)
              8 LOAD_CONST               3 (1)
             10 CALL_FUNCTION            1
             12 GET_ITER
             14 CALL_FUNCTION            1
             16 RETURN_VALUE

The locals function was not called. You see the call in the code of the list comprehension:

>>> dis(f.__code__.co_consts[1])
  1           0 BUILD_LIST               0
              2 LOAD_FAST                0 (.0)
        >>    4 FOR_ITER                10 (to 16)
              6 STORE_FAST               1 (_)
              8 LOAD_GLOBAL              0 (locals)
             10 CALL_FUNCTION            0
             12 LIST_APPEND              2
             14 JUMP_ABSOLUTE            4
        >>   16 RETURN_VALUE

While

>>> def g(): return [l for l in (locals(),)]
... 
>>> dis(g)
  1           0 LOAD_CONST               1 (<code object <listcomp> at 0x7f5b65cb8930, file "<stdin>", line 1>)
              2 LOAD_CONST               2 ('g.<locals>.<listcomp>')
              4 MAKE_FUNCTION            0
              6 LOAD_GLOBAL              0 (locals)
              8 CALL_FUNCTION            0
             10 BUILD_TUPLE              1
             12 GET_ITER
             14 CALL_FUNCTION            1
             16 RETURN_VALUE

The locals function is called before the list comprehension execution, the iter is built and passed to the list comprehension.

Concerning your specific problem, you can force the evaluation of locals outside of the list comprehension (note the i=i: this is not a positional argument):

>>> d = locals()
>>> ['{name_var}_{i:02d}of{maxpg:02d}.{date_var}'.format(i=i, **d) for i in range(start, end)]
['VAR_00of01.2019-01-01', 'VAR_01of01.2019-01-01', 'VAR_02of01.2019-01-01', 'VAR_03of01.2019-01-01', 'VAR_04of01.2019-01-01', 'VAR_05of01.2019-01-01', 'VAR_06of01.2019-01-01', 'VAR_07of01.2019-01-01', 'VAR_08of01.2019-01-01', 'VAR_09of01.2019-01-01']

If your version of Python is 3.6 or newer, you can use (f strings)[https://docs.python.org/3/whatsnew/3.6.html#pep-498-formatted-string-literals]

>>> [f'{name_var}_{i:02d}of{maxpg:02d}.{date_var}' for i in range(start, end)]
['VAR_00of01.2019-01-01', 'VAR_01of01.2019-01-01', 'VAR_02of01.2019-01-01', 'VAR_03of01.2019-01-01', 'VAR_04of01.2019-01-01', 'VAR_05of01.2019-01-01', 'VAR_06of01.2019-01-01', 'VAR_07of01.2019-01-01', 'VAR_08of01.2019-01-01', 'VAR_09of01.2019-01-01']

However, I think it's not a good idea to make a lookup in locals() for every iteration. You can build your format_string once and use it in the list comprehension:

>>> format_string = '{name_var}_{{i:02d}}of{maxpg:02d}.{date_var}'.format(**locals()) 
>>> format_string
'VAR_{i:02d}of01.2019-01-01'

Or (>= 3.6):

>>> format_string = f'{name_var}_{{i:02d}}of{maxpg:02d}.{date_var}'

Then you have:

>>> [format_string.format(i=i) for i in range(start, end)]
['VAR_00of01.2019-01-01', 'VAR_01of01.2019-01-01', 'VAR_02of01.2019-01-01', 'VAR_03of01.2019-01-01', 'VAR_04of01.2019-01-01', 'VAR_05of01.2019-01-01', 'VAR_06of01.2019-01-01', 'VAR_07of01.2019-01-01', 'VAR_08of01.2019-01-01', 'VAR_09of01.2019-01-01']

Hiltner