Python: load variables in a dict into namespace

Question

I want to use a bunch of local variables defined in a function, outside of the function. So I am passing x=locals() in the return value.

How can I load all the variables defined in that dictionary into the namespace outside the function, so that instead of accessing the value using x['variable'], I could simply use variable.

Herniorrhaphy

That sounds like a terrible idea. – Killian

Less terrible than the concept of 'from module import *', since presumably you have more knowledge of what's in the dict. – Teeny

@JohnLaRooy why is this a terrible idea? – Obtest

@JohnLaRooy The dot notation is a wonderful way to write code. I'm curious, do you have a reason to prefer dictionaries to Namespaces? – Obtest

did you try:from argparse import Namespace; ns = Namespace(**mydict)? I am curious why you didn't accept that answer. – Obtest

@CharlieParker The dot notation is great, but it is not what DR was asking for. DR wanted to avoid the dot notation. – Barmy

locals().update(x) is the simplest solution. – Barmy

@JohnLaRooy Programming and Coding are generally terrible ideas since they prevent you from playing outside with other kids, but since we are coding, why not just help others code or just not comment if you do not know how to answer? – Exacting

Answer 1

Consider the Bunch alternative:

class Bunch(object):
  def __init__(self, adict):
    self.__dict__.update(adict)

so if you have a dictionary d and want to access (read) its values with the syntax x.foo instead of the clumsier d['foo'], just do

x = Bunch(d)

this works both inside and outside functions -- and it's enormously cleaner and safer than injecting d into globals()! Remember the last line from the Zen of Python...:

>>> import this
The Zen of Python, by Tim Peters
   ...
Namespaces are one honking great idea -- let's do more of those!

Proton

But this looks so elegant: globals().update(locals())! (I kid, I kid) – Grisgris

I use this globals().update(locals()) in simple plotting scripts where I have some simple difference when I load the module interactively or run in from the command line in a one off kind of way. I suppose I could just put the if name == 'main': at the top of the file and then not have functions ... but that seems just as inelagant. Anyway, if there is a better way to pass value from inside a function to the outer module, I would be interested. – Dich

Love this, but it is not recursive. Any solutions for nested dictionaries? – Anaclitic

@CleverGuy you can write it recursively as long as you don't use update: instead get k,v pairs with adict.items() and if type(v) is dict: self.__dict__[k]=Bunch(v), and then else: self.__dict__[k]=v for everything else. You can access the result like a namespace, including assigning new values to the nested spaces. – Whittington

what is wrong with: from argparse import Namespace; ns = Namespace(**mydict) ? – Obtest

But there's still an issue. Right now Bunch raises an error when being used in a python3 environment. Something about 'dict' object has no attribute 'iteritems' – Cachucha

Answer 2

Rather than create your own object, you can use argparse.Namespace:

from argparse import Namespace
ns = Namespace(**mydict)

To do the inverse:

mydict = vars(ns)

Psychosis

Finally, something safe, readable and already implemented in the standard library. I wonder why this honking great idea resides in the argparse module, though. – Clam

Now avaialble as types.SimpleNamespace – Rollins

This is a super solution and deserves upvotes. See repl.it/repls/NavyFlusteredSearchengine to see how easy it is – Intercollegiate

There's a subtle caveat in the "inverse" part: when doing mydict = vars(ns) both the new dictionary and the original namespace will point to the same object (changing either one will also change the other). Do mydict = vars(ns).copy() to avoid that. – Wheatworm

what do you think of the accepted answer talking about bunch? I am curious. – Obtest

Answer 3

This is perfectly valid case to import variables in one local space into another local space as long as one is aware of what he/she is doing. I have seen such code many times being used in useful ways. Just need to be careful not to pollute common global space.

You can do the following:

adict = { 'x' : 'I am x', 'y' : ' I am y' }
locals().update(adict)
blah(x)
blah(y)

Ripieno

def func(x): locals().update(x) print(pos) print(locals()) func({"pos":23}) --------------------------------------------------------------------------- NameError Traceback (most recent call last) <ipython-input-211-4eae42dbc044> in <module>() 3 print(pos) 4 print(locals()) ----> 5 func({"pos":23}) <ipython-input-211-4eae42dbc044> in func(x) 1 def func(x): 2 locals().update(x) ----> 3 print(pos) 4 print(locals()) 5 func({"pos":23}) NameError: name 'pos' is not defined – Accipiter

The docs explicitly warn not to update the return value of locals(). – Rigatoni

It like this answer, but it may mislead the reader into believing that this is a recommended appoach – Vitiated

what is wrong with: from argparse import Namespace; ns = Namespace(**mydict) ? – Obtest

Answer 4

Importing variables into a local namespace is a valid problem and often utilized in templating frameworks.

Return all local variables from a function:

return locals()

Then import as follows:

r = fce()
for key in r.keys():
   exec(key + " = r['" + key + "']")

Chiller

Works perfectly but is very slow... not a problem for simple test programs though. – Amygdalin

what is wrong with: from argparse import Namespace; ns = Namespace(**mydict) ? – Obtest

There isn't CharlieParker see answer posted by orodbhen. – Chiller

Answer 5

The Bunch answer is ok but lacks recursion and proper __repr__ and __eq__ builtins to simulate what you can already do with a dict. Also the key to recursion is not only to recurse on dicts but also on lists, so that dicts inside lists are also converted.

These two options I hope will cover your needs (you might have to adjust the type checks in __elt() for more complex objects; these were tested mainly on json imports so very simple core types).

1. The Bunch approach (as per previous answer) - object takes a dict and converts it recursively. repr(obj) will return Bunch({...}) that can be re-interpreted into an equivalent object.

class Bunch(object):
    def __init__(self, adict):
        """Create a namespace object from a dict, recursively"""
        self.__dict__.update({k: self.__elt(v) for k, v in adict.items()})

    def __elt(self, elt):
        """Recurse into elt to create leaf namespace objects"""
        if type(elt) is dict:
            return type(self)(elt)
        if type(elt) in (list, tuple):
            return [self.__elt(i) for i in elt]
        return elt
    
    def __repr__(self):
        """Return repr(self)."""
        return "%s(%s)" % (type(self).__name__, repr(self.__dict__))

    def __eq__(self, other):
        if hasattr(other, '__dict__'):
            return self.__dict__ == other.__dict__
        return NotImplemented
        # Use this to allow comparing with dicts:
        #return self.__dict__ == (other.__dict__ if hasattr(other, '__dict__') else other)

2. The SimpleNamespace approach - since types.SimpleNamespace already implements __repr__ and __eq__, all you need is to implement a recursive __init__ method:

import types
class RecursiveNamespace(types.SimpleNamespace):
    # def __init__(self, /, **kwargs):  # better, but Python 3.8+
    def __init__(self, **kwargs):
        """Create a SimpleNamespace recursively"""
        self.__dict__.update({k: self.__elt(v) for k, v in kwargs.items()})
        
    def __elt(self, elt):
        """Recurse into elt to create leaf namespace objects"""
        if type(elt) is dict:
            return type(self)(**elt)
        if type(elt) in (list, tuple):
            return [self.__elt(i) for i in elt]
        return elt

    # Optional, allow comparison with dicts:
    #def __eq__(self, other):
    #    return self.__dict__ == (other.__dict__ if hasattr(other, '__dict__') else other)

The RecursiveNamespace class takes keyword arguments, which can of course come from a de-referenced dict (ex **mydict)

---

Now let's put them to the test (argparse.Namespace added for comparison, although it's nested dict is manually converted):

from argparse import Namespace
from itertools import combinations
adict = {'foo': 'bar', 'baz': [{'aaa': 'bbb', 'ccc': 'ddd'}]}
a = Bunch(adict)
b = RecursiveNamespace(**adict)
c = Namespace(**adict)
c.baz[0] = Namespace(**c.baz[0])
for n in ['a', 'b', 'c']:
    print(f'{n}:', str(globals()[n]))
for na, nb in combinations(['a', 'b', 'c'], 2):
    print(f'{na} == {nb}:', str(globals()[na] == globals()[nb]))

The result is:

a: Bunch({'foo': 'bar', 'baz': [Bunch({'aaa': 'bbb', 'ccc': 'ddd'})]})
b: RecursiveNamespace(foo='bar', baz=[RecursiveNamespace(aaa='bbb', ccc='ddd')])
c: Namespace(foo='bar', baz=[Namespace(aaa='bbb', ccc='ddd')])
a == b: True
a == c: True
b == c: False

Although those are different classes, because they both (a and b) have been initialized to equivalent namespaces and their __eq__ method compares the namespace only (self.__dict__), comparing two namespace objects returns True. For the case of comparing with argparse.Namespace, for some reason only Bunch works and I'm unsure why (please comment if you know, I haven't looked much further as types.SimpleNameSpace is a built-in implementation).

You might also notice that I recurse using type(self)(...) rather than using the class name - this has two advantages: first the class can be renamed without having to update recursive calls, and second if the class is subclassed we'll be recursing using the subclass name. It's also the name used in __repr__ (type(self).__name__).

EDIT 2021-11-27:

1. Modified the Bunch.__eq__ method to make it safe against type mismatch.

2. Added/modified optional __eq__ methods (commented out) to allow comparing with the original dict and argparse.Namespace(**dict) (note that the later is not recursive but would still be comparable with other classes as the sublevel structs would compare fine anyway).

Adjacency

what is wrong with: from argparse import Namespace; ns = Namespace(**mydict) ? – Obtest

Try my example. Namespace(**dict) is not recursive. You did make me realize with a simple change I could also make both classes compare fine against a dict. To compare Namespace() with dicts the same ` __eq__` method could be used. – Adjacency

Also I can't explain why SimpleNamespace can't compare with argparse.Namespace when Bunch can, I would have to look at the actual implementation in CPython. – Adjacency

Answer 6

To extend Thomas's idea, the following preserves type of any supported iterable object that supports list-to-type conversion.

import types
import sys

class RecursiveNamespace(types.SimpleNamespace):
    def __init__(self, accepted_iter_types = [], **kwargs):
        self.supported_types = [list, tuple, set] + accepted_iter_types

        for key, val in kwargs.items():
            if type(val) == dict:
                setattr(self, key, RecursiveNamespace(**val))
            elif hasattr(val, '__iter__'): # object is iterable
                setattr(self, key, self.__make_iterable(val))
            else: 
                setattr(self, key, val)

    def __make_iterable(self, val):
        if(type(val) not in self.supported_types): # not a supoorted iterable type
            return val
        lst = [self.__recurse_in(v) for v in val]
        try:
            ret = type(val)(lst)  # the type is assumed to support list-to-type conversion
        except Exception as e:
            print(f"Failed to make iterable object of type {type(val)}", e, out=sys.stderr)
        return  ret 

    def __recurse_in(self, val):
        if type(val) == dict:
            return RecursiveNamespace(**val)
        elif(hasattr(val, '__iter__')): # if it's iterable
            return self.__make_iterable(val)        
        else:
            return val

    def __getitem__(self, key):
        return self.__dict__[key]

if __name__ == '__main__':
    data = {'a': 1, 
        'b': (2,3), 
        'c': [4,5],
        'd': set([6,'7',8]),
        'e': {
            'e_1': 9, 
            'e_2': {
                    'e_2_1': 10, 
                    'e_2_2': (11,)
                    },
            'e_3': [12,13]}
    }

    rname = RecursiveNamespace(**data)
    print(rname)
    print('%20s :'%(type(rname.a)), rname.a)
    print('%20s :'%(type(rname.b)), rname.b)
    print('%20s :'%(type(rname.c)), rname.c)
    print('%20s :'%(type(rname.d)), rname.d)
    print('%20s :'%(type(rname.e.e_2.e_2_2)), rname.e.e_2.e_2_2)

output:

   <class 'int'> : 1
 <class 'tuple'> : (2, 3)
  <class 'list'> : [4, 5]
   <class 'set'> : {8, '7', 6}
 <class 'tuple'> : (11,)

Sweeney

Answer 7

Used following snippet (PY2) to make recursive namespace from my dict(yaml) configs:

class NameSpace(object):
    def __setattr__(self, key, value):
        raise AttributeError('Please don\'t modify config dict')


def dump_to_namespace(ns, d):
    for k, v in d.iteritems():
        if isinstance(v, dict):
            leaf_ns = NameSpace()
            ns.__dict__[k] = leaf_ns
            dump_to_namespace(leaf_ns, v)
        else:
            ns.__dict__[k] = v

config = NameSpace()
dump_to_namespace(config, config_dict)

Lebna

Answer 8

There's Always this option, I don't know that it is the best method out there, but it sure does work. Assuming type(x) = dict

for key, val in x.items():  # unpack the keys from the dictionary to individual variables
    exec (key + '=val')

Thiouracil

This works well, until the intern catches you doing it and thinks "exec" is the best thing since Python. And just in case there are any interns reading this, don't do this. This damages the readability of your code, makes it hard to debug, is incredibly slow, and may cause a whole host of problems I can't even properly anticipate and explain. – Retention

If you are doing down this route, just update locals directly. e.g. locals().update(x). – Feeley

This can also be easily abused. What if the key contains user-provided data? You've just given your users full access to your application, it's credentials, every other credentials the application's user has (hopefully none but who knows at that point) and access to any unpatched, normally low-risk local privileges escalation vulnerability. In other words, if your applications is externally accessible you might as well save hackers some time and post your root password now. Your comment is also a good example of wasting developer's time with useless comments. – Adjacency

thanks for your comment @ThomasGuyot-Sionnest I hadn't thought of it that way since i never make apps that are used for others, it's mostly data analysis for myself, your comment is an example of elites pricks trolling old message boards. – Thiouracil