Dynamically importing Python module

Question

I have a trusted remote server that stores many custom Python modules. I can fetch them via HTTP (e.g. using urllib2.urlopen) as text/plain, but I cannot save the fetched module code to the local hard disk. How can I import the code as a fully operable Python module, including its global variables and imports?
I suppose I have to use some combination of exec and imp module's functions, but I've been unable to make it work yet.

Cheerly

are you downloading them over ssl or is every intermediate router trusted as well? :P – Nebiim

Actually, yes - both the module storage and the front-end server are parts of a single system (and are even located in the same server room). For now, suppose there are no security implications: even if I download the code over SSL, the question of how to interpret it still stands. – Cheerly

Answer 1

It looks like this should do the trick: importing a dynamically generated module

>>> import imp
>>> foo = imp.new_module("foo")
>>> foo_code = """
... class Foo:
...     pass
... """
>>> exec foo_code in foo.__dict__
>>> foo.Foo.__module__
'foo'
>>>

Also, as suggested in the ActiveState article, you might want to add your new module to sys.modules:

>>> import sys
>>> sys.modules["foo"] = foo
>>> from foo import Foo
<class 'Foo' …>
>>>

Keyser

Is there a trick so that I could allow the user to "from foo import Foo"? In this example, I get "ImportError: No module named foo". – Refine

After sys.modules["foo"] = foo, the sys module breaks and equals None. Is that supposed to happen or can you avoid that? See my question on that here. – Ounce

@DavidWolever It seems that imp.new_module is deprecated since Python 3.4 (ref). Use importlib.util.module_from_spec instead. – Arcane

Answer 2

Here's something I bookmarked a while back that covers something similar:

• Customizing the Python Import System

It's a bit beyond what you want, but the basic idea is there.

Eligibility

Answer 3

I recently encountered trying to do this while trying to write unit tests for source code examples I put into a project's readme (I wanted to avoid just linking to small files or duplicating the text in a way that could get out of sync).

I came up with the following

import sys
import types
from importlib import import_module


def compile_and_install_module(module_name: str, source_code: str) -> types.ModuleType:
    """Compile source code and install it as a module.

    End result is that `import <module_name>` and `from <module_name> import ...` should work.
    """
    module = types.ModuleType(module_name, "Module created from source code")

    # Execute source in context of empty/fake module
    exec(source_code, module.__dict__)

    # Insert fake module into sys.modules. It's now a real module
    sys.modules[module_name] = module

    # Imports should work now
    return import_module(module_name)

And a quick example of how you can use it

$ cat hello.py 
def foo():
    print("Hello world")


bar = 42

$ python
Python 3.9.5 (tags/v3.9.5:0a7dcbd, May  3 2021, 17:27:52) [MSC v.1928 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from compile import compile_and_install_module
>>> compile_and_install_module("hello", open("hello.py").read())
<module 'hello'>
>>> import hello
>>> hello.foo()
Hello world
>>> from hello import bar
>>> bar
42

You can remove the return value and import_lib import if you

Redistrict

The types.ModuleType seems to be the new way of creating modules dynamically, since imp is deprecated now. +1 to you. – Extroversion

Answer 4

Python3 version
(attempted to edit other answer but the edit que is full)

import imp

my_dynamic_module = imp.new_module("my_dynamic_module")
exec("""
class Foo:
    pass
""", my_dynamic_module.__dict__)

Foo = my_dynamic_module.Foo
foo_object = Foo()

# register it on sys
import sys
sys.modules[my_dynamic_module.__name__] = my_dynamic_module

Igor