Why is the value of __name__ changing after assignment to sys.modules[__name__]?

Question

While trying to do something similar to what's in the ActiveState recipe titled Constants in Python by Alex Martelli, I ran into the unexpected side-effect (in Python 2.7) that assigning a class instance to an entry in sys.modules has -- namely that doing so apparently changes the value of __name__ to None as illustrated in the following code fragment (which breaks part of the code in the recipe):

class _test(object): pass

import sys
print '# __name__: %r' % __name__
# __name__: '__main__'
sys.modules[__name__] = _test()
print '# __name__: %r' % __name__
# __name__: None

if __name__ == '__main__': # never executes...
    import test
    print "done"

I'd like to understand why this is happening. I don't believe it was that way in Python 2.6 and earlier versions since I have some older code where apparently the if __name__ == '__main__': conditional worked as expected following the assignment (but no longer does).

FWIW, I also noticed that the name _test is getting rebound from a class object to None, too, after the assignment. It seems odd to me that they're being rebound to None rather than disappearing altogether...

Update:

I'd like to add that any workarounds for achieving the effect of if __name__ == '__main__':, given what happens would be greatly appreciated. TIA!

Chloroprene

Answer 1

This happens because you have overwrite your module when you did sys.modules[__name__] = _test() so your module was deleted (because the module didn't have any references to it anymore and the reference counter went to zero so it's deleted) but in the mean time the interpreter still have the byte code so it will still work but by returning None to every variable in your module (this is because python sets all the variables to None in a module when it's deleted).

class _test(object): pass

import sys
print sys.modules['__main__']
# <module '__main__' from 'test.py'>  <<< the test.py is the name of this module
sys.modules[__name__] = _test()
# Which is the same as doing sys.modules['__main__'] = _test() but wait a
# minute isn't sys.modules['__main__'] was referencing to this module so
# Oops i just overwrite this module entry so this module will be deleted
# it's like if i did:
#
#   import test
#   __main__ = test
#   del test
#   __main__ = _test()
#   test will be deleted because the only reference for it was __main__ in
#   that point.

print sys, __name__
# None, None

import sys   # i should re import sys again.
print sys.modules['__main__']
# <__main__._test instance at 0x7f031fcb5488>  <<< my new module reference.

EDIT:

A fix will be by doing like this:

class _test(object): pass

import sys
ref = sys.modules[__name__]  # Create another reference of this module.
sys.modules[__name__] = _test()   # Now when it's overwritten it will not be
                                  # deleted because a reference to it still
                                  # exists.

print __name__, _test
# __main__ <class '__main__._test'>

Hope this will explain things.

Changchun

And the "overwrite everything with None" behaviour comes from something the module destructor does deliberately to clear reference cycles between functions defined in the module and the module __dict__. – Kirkwall

Nice answer and excellent workaround. I'd never considered that removing a sys.modules[] reference to the module could result in its immediate destruction because of its reference count going to zero (especially considering it's being done by code in the module itself). Thanks! – Chloroprene

Based on your workaround suggestion, I've replaced the sys.modules[__name__] = _test() with a _ref, sys.modules[__name__] = sys.modules[__name__], _test() and all seems well again. – Chloroprene

ref = sys.modules['__main__'] is broken when test.py is imported. It should be ref = sys.modules[__name__] – Contend

@Nizam: Noted—and fixed. Good catch! – Chloroprene

Answer 2

If I assign anything to sys.modules['__main__'] I get a severely broken environment. Not this exact behaviour, but all my globals and builtins disappear.

sys.modules isn't documented to behave in any particular way when written to, only vaguely that you can use it for “reloading tricks” (and there are some significant traps even to that usage).

I wouldn't write a non-module to it and expect anything but pain. I think this recipe is totally misguided.

Shortcircuit

In the python REPL sys.modules['__main__'] refer to the built-in module (<module '__main__' (built-in)>), so if you overwrite it , the built-in module will be deleted so you will not have any access to any built-in function anymore. – Changchun

It's not misguided, it's sweet... of course, every time you write to something in the sys module, you should be aware that "Here be dragons". ;-) – Mira