How to make imports / closures work from IPython's embed?

Question

I sometimes use embed at a certain point in a script to quickly flesh out some local functionality. Minimal example:

#!/usr/bin/env python

# ...

import IPython
IPython.embed()

Developing a local function often requires a new import. However, importing a module in the IPython session does not seem to work, when used in a function. For instance:

In [1]: import os

In [2]: def local_func(): return os.path.sep

In [3]: local_func()
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-3-f0e5d4635432> in <module>()
----> 1 local_func()

<ipython-input-2-c530ce486a2b> in local_func()
----> 1 def local_func(): return os.path.sep

NameError: global name 'os' is not defined

This is rather confusing, especially since I can even use tab completion to write os.path.sep.

I noticed that the problem is even more fundamental: In general, functions created in the IPython embed session do not close over variables from the embed scope. For instance, this fails as well:

In [4]: x = 0

In [5]: def local_func(): return x

In [6]: local_func()
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-6-f0e5d4635432> in <module>()
----> 1 local_func()

<ipython-input-5-2116e9532e5c> in local_func()
----> 1 def local_func(): return x

NameError: global name 'x' is not defined

Module names are probably just the most common thing to "close over"...

Is there any solution to this problem?

Update: The problem not only applies for closures, but also nested list comprehensions.

Disclaimer: I'll post an (unsatisfactory) answer to the question myself -- still hoping for a better solution though.

Czardom

As of February 2017 I still observe the issue with sys.version being 2.7.12 (default, Nov 19 2016, 06:48:10) [GCC 5.4.0 20160609]. – Hectare

Answer 1

I also had the same problem. I used this trick to deal with the case when the embed() is called outside a function, so that globals() and locals() should be the same dictionary.

The simplest way is to call the following function after ipython is launched

ipy = get_ipython()
setattr(ipy.__class__, 'user_global_ns', property(lambda self: self.user_ns))

Another way is to subclass InteractiveShellEmbed

class InteractiveShellEmbedEnhanced(InteractiveShellEmbed):
    @property
    def user_global_ns(self):
        if getattr(self, 'embedded_outside_func', False):
            return self.user_ns
        else:
            return self.user_module.__dict__

    def init_frame(self, frame):
        if frame.f_code.co_name == '<module>':
            self.embedded_outside_func = True
        else:
            self.embedded_outside_func = False

and modify slightly the code of IPython.terminal.embed.embed() so that in it all InteractiveShellEmbed is changed to InteractiveShellEmbedEnhanced and call shell.init_frame(frame) after the line shell = InteractiveShellEmbed.instance(...).

This is based on the following observations:

• In an ipython session, we always have id(globals()) == id(ipy.user_module.__dict__) == id(ipy.user_global_ns) (user_global_ns is a class property of the super class of InteractiveShellEmbed, which returns ipy.user_module.__dict__)
• Also we have id(locals()) == id(ipy.user_ns)
• For normal ipython session, id(locals()) == id(globals())
• user_global_ns (a property) and user_ns (a dict) define the execution context
• In embedded ipython, ipy.user_module and ipy.user_ns are set in function ipy.__call__() and passed to ipy.mainloop(). They are not the same object since ipy.user_ns is constructed inside the functions.

If you are to launch ipython outside a function (like in a script), then it is safe to assume the globals() should be identical to locals().

With this setup, the following code should work while not working using the default embedded shell:

a=3
(lambda :a)()    # default behavior: name 'a' is not defined
import time
(lambda: time.time())()  # default behavior: name 'time' is not defined

(default behavior is due to a and time are not added to globals() and ipython does not make closures for local functions (the lambdas defined above) and insists to look up the variables in global scope. search closure in this page)

Ostracize

Very interesting. Do you think it makes sense to contribute that to iPython? Maybe this could close issues #62 and #10695. – Czardom

@Czardom I am not an expert in either python or IPython. I am not sure whether the code will work at all circumstances when launched outside a function. I can only share my code and let the experts tell whether it serves as a solution or points to the direction to solve it. I have put a more complete snippet to your #10695, which also provides helpers when IPython is embedded from a function. – Ostracize

Answer 2

This "bug" of IPython.embed() is still there today. But from the comment of developer, we could instead use:

from IPython import start_ipython
... # some random code
start_ipython(argv=[], user_ns = locals())

Or more aggressively, replace the user_ns = locals() to user_ns = locals() | globals()

Chap

Answer 3

Update: Again only a work-around, but somewhat simpler: globals().update(locals())

---

I don't have a general solution, but at least a work-around: After defining a local function, it is possible to add the locals() of the session to the func_globals of the function just defined, e.g.:

In [1]: import os

In [2]: def local_func(): return os.path.sep

In [3]: local_func.func_globals.update(locals())

In [4]: local_func()
Out[4]: '/'

However, one should be aware that this is only a "manual closure" and will not work as a regular closure in cases like this:

In [1]: x = 1

In [2]: def local_func(): return x

In [3]: local_func.func_globals.update(locals())

In [4]: local_func()
Out[4]: 1

In [5]: x = 42

In [6]: local_func() # true closure would return 42
Out[6]: 1

In [7]: local_func.func_globals.update(locals()) # but need to update again

In [8]: local_func()
Out[8]: 42

At least it can solve the notorious global name '...' is not defined problem for imports.

Czardom