Dynamic class loading in Python 2.6: RuntimeWarning: Parent module 'plugins' not found while handling absolute import

Question

I am working on a plugin system where plugin modules are loaded like this:

def load_plugins():
   plugins=glob.glob("plugins/*.py")
   instances=[]
   for p in plugins:
      try:
         name=p.split("/")[-1]
         name=name.split(".py")[0]
         log.debug("Possible plugin: %s", name)
         f, file, desc=imp.find_module(name, ["plugins"])
         plugin=imp.load_module('plugins.'+name, f, file, desc)
         getattr(plugin, "__init__")(log)
         instances=instances+plugin.get_instances()
      except Exception as e:
         log.info("Failed to load plugin: "+str(p))
         log.info("Error: %s " % (e))
         log.info(traceback.format_exc(e))
   return instances

The code works, but for each import statement in the plugin code i get a warning like this:

plugins/plugin.py:2: RuntimeWarning: Parent module 'plugins' not found while handling absolute import
  import os

No errors are reported for the main program code, and the plugins work.

Can somebody explain what the warning means and what I doing wrong. Do I need to create an empty plugins module separately and import it to keep python happy?

Antoneantonella

for the record: 'plugins' in Parent module 'plugins' not found comes from the name value passed to imp.load_module, eg. "plugins.something" in imp.load_module("plugins.something"). In my case the name value was like ".something" and thus message contained '' instead of 'plugins'. – Piperine

Answer 1

If the directory plugins were a real package (contained __init__.py fine), you could easily use pkgutils to enumerate its plugin files and load them.

import pkgutil
# import our package
import plugins
list(pkgutil.iter_modules(plugins.__path__))

However, it can work without a plugin package anyway, try this:

import pkgutil
list(pkgutil.iter_modules(["plugins"]))

Also it is possible to make a package that only exists at runtime:

import types
import sys
plugins = types.ModuleType("plugins")
plugins.__path__ = ["plugins"]

sys.modules["plugins"] = plugins
import plugins.testplugin

However that hack that was mostly for fun!

Middleaged

Answer 2

If the plugins directory does not have an __init__.py, it isn't a package, so when you create plugins.whatever, Python warns you that such a thing shouldn't really exist. (It couldn't be created by "import plugins.whatever" no matter what your path is.)

Also,

• Don't split on /, which is unportable. Use os.path.split.
• Don't use .split(".py") to get the name without the extension, which is buggy. Use os.path.splitext.
• Don't use getattr with a string literal. getattr(plugin, "__init__") is spelled plugin.__init__.
• I am confused why you are calling a module-level __init__ function. This doesn't seem right. Perhaps you want a "set_logger" function or better, to instantiate a class that takes a logger.
• Don't use L = L + some_other_list to extend a list, use the extend method, which has better performance and is more idiomatic.
• Don't squash unknown exceptions by except Exception. If you cannot plan to do something sane in response to an exception, your program cannot go on sanely.

Jurisprudent

Note that the plugins directory is not intended to be a package, and not imported (i want to cherry pick files from it). The rest of the code is compiled as an egg and run directly using "python -m". I never add the plugin directory to the sys.path. – Antoneantonella

Right, but if you have a plugin foo, you have told it that it is plugins.foo, so it thinks it's part of the plugins package (which doesn't exist). When within foo.py you import os (or if you had imported anythingelse), you get that warning because plugins doesn't really exist. – Jurisprudent

(This means your options are to ignore the warning, change your naming scheme, or change your structure to match what Python expects.) – Jurisprudent

except Exception does not catch all exceptions, notably not SystemExit or KeyboardInterrupt. Sincie Py 2.6 not GeneratorExit either. – Middleaged

I think I understand that, thanks for the explanation. And thanks for the help with my general Python. It will take a while to master the API and know how things should be done. EDIT: Ninja post by Mike: Is there no way to create an empty package? – Antoneantonella

A package is a directory with an __init__.py file in it. – Jurisprudent

@kaizer.se, I had forgotten that a couple common exceptions do not subclass Exception. The base advice remains: you shouldn't silence exceptions you cannot understand and continue with your program; does this do something reasonable in response to MemoryError? – Jurisprudent

Then the solution becomes to add an empty package with the init.py to the egg. It seems to work well. Thanks for the help. – Antoneantonella

I don't see anything idiomatically wrong with using + to combine two lists. Otherwise great answer. – Suffuse

@Mark, in the original code, we're using Shlemiel the painter's algorithm joelonsoftware.com/articles/fog0000000319.html – Jurisprudent

@MikeGraham +1 for the reminder to use list.extend (just yesterday I used "+" and forgot about extend). +2 for the link to a joelOn post from 2001. #tripdownmemorylane. – Sobersided

"Don't squash unknown exceptions by except Exception." -- I think this is sane in the context of loading plugins. You don't want your whole application shutting down just because a plugin messed up, especially in a GUI-only environment. – Fetich

Answer 3

The problem here is with the dot ('.') in the module name:

imp.load_module('plugins.'+name, f, file, desc)

Don't include a '.' after 'plugins', or Python will think it's a module path.

Trace

Answer 4

You can try adding below statement at the beginning of import statements.

from __future__ import absolute_import

Bromley

Answer 5

The Python imp documentation has been updated since this was answered. It now addresses this issue specifically in the find_module() method.

This function does not handle hierarchical module names (names containing dots). In order to find P.M, that is, submodule M of package P, use find_module() and load_module() to find and load package P, and then use find_module() with the path argument set to P.__path__. When P itself has a dotted name, apply this recipe recursively.

Note that P.__path__ is already a list when supplying it to find_module(). Also note what the find_module() documentation says about finding packages.

If the module is a package, file is None, pathname is the package path and the last item in the description tuple is PKG_DIRECTORY.

So from the OP's question, to import the plugin without the RuntimeError warnings, follow the directions in the updated Python documentation:

# first find and load the package assuming it is in
# the current working directory, '.'

f, file, desc = imp.find_module('plugins', ['.'])
pkg = imp.load_module('plugins', f, file, desc)

# then find the named plugin module using pkg.__path__
# and load the module using the dotted name

f, file, desc = imp.find_module(name, pkg.__path__)
plugin = imp.load_module('plugins.' + name, f, file, desc)

Sequela