Hook the global name lookup in a python interpreter

Question

Here is the thing, I have a proxy holding the reference to a remote module, and I put some of these proxies to the sys.modules such that I can use it just like local modules. But some other objects are put in the __builtin__ module at the remote environment (like a magic variable for convenience of debugging or referencing). I don't want to reference these vars like conn.__builtin__.var, and I have to either replace the local __builtin__ (which seems not working for replace sys.modules['__builtin__'] or to hook the global name finding rules. How? For a module you can just overload a getattr to do this. But in a interactive interpreter like IPython, who is the main module or how to do this? update: As pointed out by @Nizam Mohamed, yes I can get the __main__ module, but still I can't modify the name lookup role of it.

I'd like to turn the local environment completely to be the remote one (for a debugging console)

UPDATE

For now I just iterate all the __builtin__.__dict__ and if there is a name that isn't in the local __builtin__. I add the name to local's __builtin__. But it's not so dynamic compare to a name lookup rule say if I can't find the name in local __builtin__ try the remote one.

here is a similar discussion.

And this question gives a simulation of module by replace it with a object in sys.modules. But this won't work for __builtin__ name lookup, I've also tried to replace the __builtin__.__getattribute__ with a custom one that will first use the original lookup followed by a custom one when failed. But global name lookup of __builtin__ never called into the __builtin__.__getattribute__ even __builtin__.__getattribute__('name') returns the desired value, __builtin__.name or name never returns one.

Vasilek

Hooking imports is fiddly, but it can be done. importlib is the library to play with on modern versions of Python. I've written some import hooks here for another purpose that should illustrate what's needed. – Giro

@ThomasK actually, I've done the import part. I can have access to the remote modules. But there is a name say foo was add to the remote __builtin__.foo = 1, I don't want to access this var by saying conn.__builtin__.foo, which can refer directly as foo in the remote side. – Vasilek

Oh, sorry, I read too fast. In IPython, you can get a reference to the user namespace as get_ipython().user_ns. I'm not sure how far that will work with a dictionary-like object with overridden methods, though - that's up to Python itself. – Giro

this = __import__(__name__) refers to current module. – Oxidimetry

You could use an AST transformer. That wouldn't work for code written outside of the interactive environment, though. – Lisabethlisan

@Lisabethlisan I think its too complicate, but maybe it can be done. Can you give more instructions or examples. – Vasilek

@zoujyjs here is the documentation. – Lisabethlisan

Answer 1

Use AST transformation of IPython shell

As @asmeurer said, you can write a simple AST transformer to "hook" the variable name lookup. The base class ast.NodeTransformer provide a visit_Name method that you can manipulate. You just need to overload this method to redefine those variables existing in the remote module but not locally.

The following module can be used as an IPython extension:

testAST.py

import ast

modName = "undefined"
modAttr = []
user_ns = {}
class MyTransformer(ast.NodeTransformer):
    def visit_Name(self, node):
        if node.id in modAttr and not node.id in user_ns: 
          return self.getName(node)
        return node
    def getName(self, NameNode):
        return ast.Attribute(value=ast.Name(id=modName, ctx=ast.Load()), 
                             attr = NameNode.id, 
                             ctx  = NameNode.ctx)
def magic_import(self, line):
    global modName, modAttr, user_ns
    modName = str(line)
    if not self.shell.run_code( compile('import {0}'.format(line), '<string>', 'exec') ):
       user_ns = self.shell.user_ns
       modAttr = user_ns[line.strip()].__dict__
       self.shell.ast_transformers.append(MyTransformer())
       print modName, 'imported'
    
def load_ipython_extension(ip):
    ip.define_magic('magic_import', magic_import)

dummyModule.py

robot=" World"

Usage:

In [1]: %load_ext testAST
In [2]: %magic_import dummyModule
In [3]: print "Hello" , robot
Hello World

In [4]: dummyModule.robot_II = "Human" 
In [5]: print "Hi", robot_II
Hi Human

The benefit of this method is that any modification to the remote module takes effect immediately because the lookup is done in the language level and no object is copied and cached.

One drawback of this method is not being able to handle dynamic lookup. If that's important for you, maybe the python_line_transforms hook is more suitable.

Woeful

Answer 2

There is a way to get a list of all names the module will use. Although it does not modify the lookup mechanism, I believe it solves your problem.

Here is some code (hopefully understandable enough) you can put in a module, let's call it magic:

import sys

def magic():
    # Get the caller frame
    frame = sys._getframe().f_back

    # Unwind all internal Python import-related stuff
    while frame.f_code.co_filename.startswith('<'):
        frame = frame.f_back

    importer = frame

    # Iterate through names the module has/will use
    for name in importer.f_code.co_names:

        # If the module has not yet defined/imported this name
        if name not in importer.f_globals and \
                name not in importer.f_locals and \
                name not in __builtins__:

            # Replace the name in the importer's namespace
            # You'll have to replace the right-hand side by your specific code
            importer.f_globals[name] = 'hello world'

You can then import it to do magic:

import magic
magic.magic()

print(hi)

There are two downsides, however. First, dynamic lookups will fails:

import magic
magic.magic()

print(globals()['hi']) # KeyError: 'hi'

Although that particular case can be solved by looking at the strings in importer.f_code.co_consts, it will not work with more advanced dynamic lookups like print(globals()['h'+'i'])

The second downside is that it will not work in functions:

import magic
magic.magic()

def f():
    print(hi) # NameError: name 'hi' is not defined

f()

This is because in that case, the name hi is in f.__code__.co_names instead of the module's co_names.

One possible solution would be to modify magic to try to find all functions of the module and alter their code, but it won't work with functions defined inside functions, etc.

An other solution is to call magic in the function:

import magic

def f():
    magic.magic()
    print(hi)

f()

Geostatics