How to create dynamical scoped variables in Python?

Question

I am translating some code from lisp to Python.

In lisp, you can have a let construct with the variables introduced declared as special and thus having dynamic scope. (See http://en.wikipedia.org/wiki/Dynamic_scope#Dynamic_scoping)

How can I do likewise in Python? It seems the language does not support this directly, if true, what would be a good way to emulate it?

Probst

Answer 1

I feel Justice is plain right in his reasoning here.

On the other hand -- I can't resist implementing proof of concept for still another programing paradigm "unnatural" to Python -- I simply love doing this. :-)

So, I created a class whose objects'attributes are scopped just like you require (and can be created dynamically). As I said, it is just in a proof of concept state - but I think most usual errors, (like trying to access a variable ina scope it is not defined at all) should have errors raised, even if not the proper ones (IndexError due to a stack underflow instead of AttributeError, for example)

import inspect


class DynamicVars(object):
    def __init__(self):
        object.__setattr__(self, "variables", {})

    def normalize(self, stackframe):
        return [hash(tpl[0]) for tpl in stackframe[1:]]

    def __setattr__(self, attr, value):
        stack = self.normalize(inspect.stack())
        d = {"value": value, "stack": stack}
        if not attr in self.variables:
            self.variables[attr] = []
            self.variables[attr].append(d)
        else:
            our_value = self.variables[attr]
            if our_value[-1]["stack"] == stack:
                our_value[-1]["value"] = value
            elif len(stack) <= len(our_value):
                while our_value and stack !=  our_value["stack"]:
                    our_value.pop()
                our_value.append(d)
            else: #len(stack) > len(our_value):
                our_value.append(d)
    def __getattr__(self, attr):
        if not attr in self.variables:
            raise AttributeError
        stack = self.normalize(inspect.stack())
        while self.variables[attr]:
            our_stack = self.variables[attr][-1]["stack"]
            if our_stack == stack[-len(our_stack):]:
                break
            self.variables[attr].pop()
        else:
            raise AttributeError
        return self.variables[attr][-1]["value"]


# for testing:
def c():
    D = DynamicVars()
    D.c = "old"
    print D.c
    def a():
        print D.c
    a()
    def b():
        D.c = "new"
        a()
    b()
    a()
    def c():
        D.c = "newest"
        a()
        b()
        a()
    c()
    a()

c()

2020 update - Another similar question showed up, and I crafted a hack that needs no special namespace objects (but which resorts to using inner things from cPython, like updating the locals() to actual variables: https://mcmap.net/q/358841/-is-there-a-way-to-declare-that-a-function-should-use-the-scope-of-the-caller (works with Python 3.8)

Perr

Congratulations! Thanks to your hard work, the world of programming has yet one more solution that will make its way deep into the hearts of numerous critical applications! – Fosterling

And after all, Lisp's "special variables" are not so terrible, are they? They're like environment variables in bash. What's terrible is languages where dynamic scoping is the default. Fortunately not many of those are left. – Weingarten

I am glad most real languages don't use dynamic scope ... and yet I've written a ton of Emacs Lisp, which does; and it feels utterly natural to me. (Emacs Lisp recently got lexical scope as an option, and I've never even bothered using it :-) – Strake

Answer 2

Here's something that works a bit like Lisp's special variables, but fits a little better into Python.

_stack = []

class _EnvBlock(object):
    def __init__(self, kwargs):
        self.kwargs = kwargs
    def __enter__(self):
        _stack.append(self.kwargs)
    def __exit__(self, t, v, tb):
        _stack.pop()

class _Env(object):
    def __getattr__(self, name):
        for scope in reversed(_stack):
            if name in scope:
                return scope[name]
        raise AttributeError("no such variable in environment")
    def let(self, **kwargs):
        return _EnvBlock(kwargs)
    def __setattr__(self, name, value):
        raise AttributeError("env variables can only be set using `with env.let()`")

env = _Env()

You can use it like this:

with env.let(bufsize=8192, encoding="ascii"):
    print env.bufsize  # prints 8192
    a()  # call a function that uses env.bufsize or env.encoding

The effects of env.let last for the duration of the with block.

Note that if you use threads, you'll definitely want a different _stack for each thread. You could use threading.local to implement that.

Weingarten

This is intended as a compromise between "don't do that" and stack inspection (which seems like it would be slow and hard to verify). – Weingarten

Nice solution. It's quite explicit (so even if it surprises someone, it doesn't mess with the ordinary Python semantics). I find this kind of approach very useful for scientific plotting, since there are many settings that I'd like to set at some point in the call stack and it's a pain to have to carry them all the way through to the function where the actual plotting happens. – Fundamental

Answer 3

The Python idiom corresponding to Lisp "special" or dynamically-scoped variables is "thread local storage".

Here is a good discussion: What is "thread local storage" in Python, and why do I need it?

If you want to fully emulate Lisp's special variables, including the let statement, you can use a context manager:

from __future__ import with_statement # if Python 2.5
from contextlib import contextmanager
import threading

dyn = threading.local()

@contextmanager
def dyn_vars(**new):
    old = {}
    for name, value in new.items():
        old[name] = getattr(dyn, name, None)
        setattr(dyn, name, value)
    yield
    for name, value in old.items():
        setattr(dyn, name, value)

Example (patently silly, but it shows the reentrant feature):

def greet_self():
    print 'Hi', dyn.who_am_I

def greet_selves():
    with dyn_vars(who_am_I='Evil Twin'):
        greet_self()
    greet_self()

with dyn_vars(who_am_I='Tobia'):
    greet_selves()

Hildy

For Python 3.7+ that needs compatibility with async code (as well as threaded code), you'd use contextvars instead of thread locals, which allows you to associate contextual state with tasks (that all run in the same thread), not just threads.. – Ondrej

Answer 4

Dynamic Scoping Considered Harmful.

Don't use it; don't emulate it.

If you need to emulate it, define a dynamic_scope module to emulate this behavior and import the module in all source files. This module should have methods begin which is called in the first line of your functions that use dynamic scopes, end, get, and set. The get and set methods should implement looking up the call chain for variable names where the call chain is implemented by begin and end. Then refactor your code to eliminate dynamic scopes.

Fosterling

Dynamic scoping can be an incredibly useful feature in languages that support it well. I've made tiny (3-4 line) changes to large Common Lisp programs that would have taken huge (but mechanically simple) modifications without it. Sometimes it's the natural solution to a problem. That said, it's not at all natural in Python, and I wouldn't suggest porting it directly -- that does seem like a recipe for maintenance pain. – Uam

There are good uses for dynamic scoping, especially when setting up relatively global settings that you don't want to thread through the arguments of every function (for example, where to print the output of stdout). Of course, dynamically scoped variables should be well marked and their general use discouraged. Jason Orendorff's solution below is a good compromise for Python and it has helped to simplify some of my code. – Fundamental