Python: perform relative import when using __import__?

Question

Here are the files in this test:

main.py
app/
 |- __init__.py
 |- master.py
 |- plugin/
 |-  |- __init__.py
 |-  |- p1.py
 |-  |_ p2.py

The idea is to have a plugin-capable app. New .py or .pyc files can be dropped into plugins that adhere to my API.

I have a master.py file at the app level that contains global variables and functions that any and all plugins may need access to, as well as the app itself. For the purposes of this test, the "app" consists of a test function in app/__init__.py. In practice the app would probably be moved to separate code file(s), but then I'd just use import master in that code file to bring in the reference to master.

Here's the file contents:

main.py:

import app

app.test()
app.test2()

app/__init__.py:

import sys, os

from plugin import p1

def test():
        print "__init__ in app is executing test"
        p1.test()

def test2():
        print "__init__ in app is executing test2"
        scriptDir = os.path.join ( os.path.dirname(os.path.abspath(__file__)), "plugin" )
        print "The scriptdir is %s" % scriptDir
        sys.path.insert(0,scriptDir)
        m = __import__("p2", globals(), locals(), [], -1)
        m.test()

app/master.py:

myVar = 0

app/plugin/__init__.py:

<empty file>

app/plugin/p1.py:

from .. import master

def test():
    print "test in p1 is running"
    print "from p1: myVar = %d" % master.myVar

app/plugin/p2.py:

from .. import master

def test():
    master.myVar = 2
    print "test in p2 is running"
    print "from p2, myVar: %d" % master.myVar

Since I explicitly import the p1 module, everything works as expected. However, when I use __import__ to import p2, I get the following error:

__init__ in app is executing test
test in p1 is running
from p1: myVar = 0
__init__ in app is executing test2
The scriptdir is ....../python/test1/app/plugin
Traceback (most recent call last):
  File "main.py", line 4, in <module>
    app.test2()
  File "....../python/test1/app/__init__.py", line 17, in test2
    m = __import__("p2", globals(), locals(), [], -1)
  File "....../python/test1/app/plugin/p2.py", line 1, in <module>
    from .. import master
ValueError: Attempted relative import in non-package

Execution proceeds all the way through the test() function and errors out right as test2() tries to execute its __import__ statement, which in turn p2 tries to do a relative import (which does work when p1 is imported explicitly via the import statement, recall)

It's clear that using __import__ is doing something different than using the import statement. The Python docs state that using import simply translates to an __import__ statement internally but there has to be more going on than meets the eye.

Since the app is plugin-based, coding explicit import statements in the main app would of course not be feasible. Using import itself within the

What am I missing here? How can I get Python to behave as expected when manually importing modules using __import__? It seems maybe I'm not fully understanding the idea of relative imports, or that I'm just missing something with respect to where the import is occurring (i.e. inside a function rather than at the root of the code file)

EDIT: I found the following possible, but unsuccessful solutions:

m = __import__("p2",globals(),locals(),"plugin")

(returns the same exact error as above)

m = __import__("plugin",fromlist="p2")

(returns a reference to app.plugin, not to app.plugin.p2)

m = __import__("plugin.p2",globals(),locals())

(returns a reference to app.plugin, not to app.plugin.p2)

import importlib
m = importlib.import_module("plugin.p2")

(returns:)

Traceback (most recent call last):
  File "main.py", line 4, in <module>
    app.test2()
  File "....../python/test1/app/__init__.py", line 20, in test2
    m = importlib.import_module("plugin.p2")
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
ImportError: No module named plugin.p2

Lafrance

Possible duplicate of #3439582 – Gerardgerardo

Alternatively, you could do something like what I've proposed in my answer to this question...essentially get a list of the files in app/master/plugin and load them dynamically: #17251508 – Gerardgerardo

@Gerardgerardo Are you talking about eliminating master.py and just using the global namespace? Somehow this seems a bit dangerous at best - possible name collisions and such...? – Lafrance

You could do the dynamic importing in master, but the concern about namespace collisions is real. That said, anytime you're allowing dynamic imports, such collisions are possible (i.e. if p1 & p2 both have a 'test' method). I guess it comes down to the use-context of your program, can you instill conventions that help limit this possibility? You could also consider making your plugin architecture class (inheritance) based, which would further help reduce the risk – Gerardgerardo

I can certainly do my best to protect collisions... The problem still remains that the code in the plugins themselves needs to access stuff in the master.py file. master.py is going to contain functions or variables that all plugins may need access to. Therefore, I still need to be able to import master from the plugins to access it, and this leads back to the initial problem where using from .. fails if I use __import__ but works if I explicitly import using a module-level import statement... – Lafrance

Yeah, this sounds like a great case for an inheritance pattern: class Master contains the common things (with unimplemented methods that define your API), and then p1, p2 inherit from Master to fill-in the details. – Gerardgerardo

This works for functions, but it doesn't solve the problem of a common variable accessible to all of the plugins. For example the host program might want to export its version, or even something like a database connection object, to all plugins. Making master a class will turn all of these into instance variables and then I'll end up having to pass all of this information into each plugin manually... The method I'm using above DOES work for common variables, but won't work for the __import__ method, so... yeah. – Lafrance

let us continue this discussion in chat – Gerardgerardo

Answer 1

I never did find a solution, so I ended up deciding to restructure the program.

What I did was set up the main app as a class. Then, I also changed each plugin into a class. Then, as I load plugins using import, I also instantiate the class inside each plugin which has a predefined name, and pass in the reference to the main app class.

This means that each class can directly read and manipulate variables back in the host class simply by using the reference. It is totally flexible because anything that the host class exports is accessible by all the plugins.

This turns out to be more effective and doesn't depend on relative paths and any of that stuff. It also means one Python interpreter could in theory run multiple instances of the host app simultaneously (on different threads for example) and the plugins will still refer back to the correct host instance.

Here's basically what I did:

main.py:

import os, os.path, sys

class MyApp:

    _plugins = []

    def __init__(self):
        self.myVar = 0

    def loadPlugins(self):
        scriptDir = os.path.join ( os.path.dirname(os.path.abspath(__file__)), "plugin" )   
        sys.path.insert(0,scriptDir)
        for plug in os.listdir(scriptDir):
            if (plug[-3:].lower() == ".py"):
                m = __import__(os.path.basename(plug)[:-3])
                self._plugins.append(m.Plugin(self))

    def runTests(self):
        for p in self._plugins:
            p.test()

if (__name__ == "__main__"):
    app = MyApp()
    app.loadPlugins()
    app.runTests()

plugin/p1.py:

class Plugin:

    def __init__(self, host):
        self.host = host

    def test(self):
        print "from p1: myVar = %d" % self.host.myVar

plugin/p2.py:

class Plugin:

    def __init__(self, host):
        self.host = host

    def test(self):
        print "from p2: variable set"
        self.host.myVar = 1
        print "from p2: myVar = %d" % self.host.myVar

There is some room to improve this, for example, validating each imported .py file to see if it's actually a plugin and so on. But this works as expected.

Lafrance

Answer 2

I've had a similar problem.
__import__ only imports submodules if all parent __init__.py files are empty. You should use importlib instead

import importlib

p2 = importlib.import_module('plugin.p2')

Metallize

Sorry, didn't work... see above. I didn't make it clear but I'm using Python 2, so maybe that's why... I think importlib is mostly a Python3 thing? – Lafrance

You can use it in 2.7 – Stiff

Answer 3

Have you tried the following syntax:

How to use python's import function properly __import__()

It worked for me with a similar problem...

Stephanotis

Answer 4

I have managed to find a solution to the problem.
By taking your example the following static import is needed to be dynamic

from .plugin import p2

the "." near plugin means there is a need to relative import and not absolute import.

I was able to do that with the following code snipset:

plugin  = __import__('plugin', globals(), locals(), level=1, fromlist=['p2'])
p2      = getattr(plugin, 'p2')

level=1 Relative import parameter
fromlist Specify which sub modules to take from plugin module

As you mentioned, plugin holds the reference to 'plugin', thus additional getattr is needed to grep p2 from plugin

Becoming