How to import all submodules?

Question

I have a directory structure as follows:

| main.py
| scripts
|--| __init__.py
   | script1.py
   | script2.py
   | script3.py

In main.py, if I import scripts, this apparently does not allow me to use scripts.script1. I know I can use from scripts import * to access the modules in the scripts package, but then I can only use them directly as scripts1, scripts2 etc.

How can I write the code so that I can refer to scripts.script1 inside main.py?

I tried using pkgutils.walk_packages, as well as the __all__ attribute of the package, to get the submodule names, but I couldn't figure out a way to use those strings to do the import.

Hackworth

will pkgutil work if you deploy the application as a zipped egg? you can have a look at "import pkg_resources", just in case – Coprophagous

I am very confused by this questions and the answers. Why this is needed at all. It's my understanding that you have two options to include all your local "modules" in python (what other languages just refer to as files. a) you include a blank init file in every folder and that ALONE tells python that this is a modules folder and to import it all before running or b) to avoid having init files everywhere, you put a preamble at the top of your files to add everything to the path. There are other considerations, like code navigation in IDEs. – Edelweiss

Answer 1

Edit: Here's one way to recursively import everything at runtime...

(Contents of __init__.py in top package directory)

import pkgutil

__all__ = []
for loader, module_name, is_pkg in pkgutil.walk_packages(__path__):
    __all__.append(module_name)
    _module = loader.find_module(module_name).load_module(module_name)
    globals()[module_name] = _module

I'm not using __import__(__path__+'.'+module_name) here, as it's difficult to properly recursively import packages using it. If you don't have nested sub-packages, and wanted to avoid using globals()[module_name], though, it's one way to do it.

There's probably a better way, but this is the best I can do, anyway.

Original Answer (For context, ignore othwerwise. I misunderstood the question initially):

What does your scripts/__init__.py look like? It should be something like:

import script1
import script2
import script3
__all__ = ['script1', 'script2', 'script3']

You could even do without defining __all__, but things (pydoc, if nothing else) will work more cleanly if you define it, even if it's just a list of what you imported.

Candelabra

Sorry for not explaining fully. I have edited the main post. I am looking to import the submodules dynamically at run-time. – Hackworth

@Hackworth - Sorry for the misunderstanding. See the changes above... It's ugly, but it works. There's probably a better way, though. – Candelabra

Thanks! :) I found that using __import__ and not needing to import the scripts by their names actually worked out nicely, because I could just iterate over the modules without needing their names. – Hackworth

Using exec() is overkill - you can just poke the loaded module into globals() and problem is solved. – Perrins

Niall, can you elaborate? I'm not sure what you mean. – Kolk

See here for an alternative to using exec. – Lungfish

For some reason this fails when my modules attempt to import other modules within the package. – Bronwyn

Is this just me or does this not work with relative imports? If I have script1/file1.py with the line from .. import script2, I get an error ValueError: attempted relative import beyond top-level package. – Veg

Also, can you do from . import * instead of the exec bit? – Veg

+1 to those who reported its incompatibility with relative imports within the package. I am facing the same problem. For me it says SystemError: Parent module '' not loaded, cannot perform relative import when one of my submodule do from .blah import Blah. – Capstan

I get a RuntimeError: maximum recursion depth exceeded using this! – Slowpoke

If you need relative imports to work: add import importlib and replace the line loading module with _module = importlib.import_module('{}.{}'.format(__name__, module_name)). – Fulks

This answer should document that __path__ is to be defined by the user. It is not automatic. – Corporate

Answer 2

This is based on the answer that kolypto provided, but his answer does not perform recursive import of packages, whereas this does. Although not required by the main question, I believe recursive import applies and can be very useful in many similar situations. I, for one, found this question when searching on the topic.

This is a nice, clean way of performing the import of the subpackage's modules, and should be portable as well, and it uses the standard lib for python 2.7+ / 3.x.

import importlib
import pkgutil


def import_submodules(package, recursive=True):
    """ Import all submodules of a module, recursively, including subpackages

    :param package: package (name or actual module)
    :type package: str | module
    :rtype: dict[str, types.ModuleType]
    """
    if isinstance(package, str):
        package = importlib.import_module(package)
    results = {}
    for loader, name, is_pkg in pkgutil.walk_packages(package.__path__):
        full_name = package.__name__ + '.' + name
        try:
            results[full_name] = importlib.import_module(full_name)
        except ModuleNotFoundError:
            continue
        if recursive and is_pkg:
            results.update(import_submodules(full_name))
    return results

Usage:

# from main.py, as per the OP's project structure
import scripts
import_submodules(scripts)

# Alternatively, from scripts.__init__.py
import_submodules(__name__)

Twyla

I believe pkgutil.walk_packages() is recursive on its own, so your function doesn't need to be recursive. – Statutable

pkgutil.walk_packages() does claim to. However, it does not do so. Just run this on some package tree, with recursive=True and recursive=False. You'll see the difference. – Twyla

To make this more resilient, there are cases where a package is listed but not actually present so to guard against it: try: results[full_name] = importlib.import_module(full_name) and except ModuleNotFoundError: continue – Cygnus

Answer 3

Simply works, and allows relative import inside packages:

def import_submodules(package_name):
    """ Import all submodules of a module, recursively

    :param package_name: Package name
    :type package_name: str
    :rtype: dict[types.ModuleType]
    """
    package = sys.modules[package_name]
    return {
        name: importlib.import_module(package_name + '.' + name)
        for loader, name, is_pkg in pkgutil.walk_packages(package.__path__)
    }

Usage:

__all__ = import_submodules(__name__).keys()

Purposive

Answer 4

To just load all submodules of a package, you can use this simple function:

import importlib
import pkgutil

def import_submodules(module):
    """Import all submodules of a module, recursively."""
    for loader, module_name, is_pkg in pkgutil.walk_packages(
            module.__path__, module.__name__ + '.'):
        importlib.import_module(module_name)

Use case: load all database models of a Flask app, so that Flask-Migrate could detect changes to the schema. Usage:

import myproject.models
import_submodules(myproject.models)

Statutable

Answer 5

I got tired of this problem myself, so I wrote a package called automodinit to fix it. You can get it from http://pypi.python.org/pypi/automodinit/. Usage is like this:

1. Include the automodinit package into your setup.py dependencies.
2. Add the following to the beginning of the __init__.py file:

__all__ = ["I will get rewritten"]
# Don't modify the line above, or this line!
import automodinit
automodinit.automodinit(__name__, __file__, globals())
del automodinit
# Anything else you want can go after here, it won't get modified.

That's it! From now on importing a module will set __all__ to a list of .py[co] files in the module and will also import each of those files as though you had typed:

for x in __all__: import x

Therefore the effect of from M import * matches exactly import M.

automodinit is happy running from inside ZIP archives and is therefore ZIP safe.

Perrins

Answer 6

Not nearly as clean as I would like, but none of the cleaner methods worked for me. This achieves the specified behaviour:

Directory structure:

| pkg
|--| __init__.py
   | main.py
   | scripts
   |--| __init__.py
      | script1.py
      | script2.py
      | script3.py

Where pkg/scripts/__init__.py is empty, and pkg/__init__.py contains:

import importlib as _importlib
import pkgutil as _pkgutil
__all__ = [_mod[1].split(".")[-1] for _mod in
           filter(lambda _mod: _mod[1].count(".") == 1 and not 
                               _mod[2] and __name__ in _mod[1],
                  [_mod for _mod in _pkgutil.walk_packages("." + __name__)])]
__sub_mods__ = [".".join(_mod[1].split(".")[1:]) for _mod in
                filter(lambda _mod: _mod[1].count(".") > 1 and not 
                                    _mod[2] and __name__ in _mod[1],
                       [_mod for _mod in 
                        _pkgutil.walk_packages("." + __name__)])]
from . import *
for _module in __sub_mods__:
    _importlib.import_module("." + _module, package=__name__)

Although it's messy, it should be portable. I've used this code for several different packages.

Profile

:-| Just curious -- what about the cleaner methods didn't work? – Bleb

Tbh, I don't remember.. sorry! The limitation may not even exist/apply anymore, so it's probably worth checking out. I'm not sure if it addresses the question, but nowadays I use __all__ = [_mod[1] for _mod in _pkgutil.iter_modules(__path__) if not _mod[2]]. I'm pretty sure the iter_modules function didn't exist back then. – Profile

Answer 7

I was writing a small personal library and adding new modules all the time so I wrote a shell script to look for scripts and create the __init__.py's. The script is executed just outside of the main directory for my package, pylux.

I know it probably isn't the answer you're looking for, but it servered its purpose for me and it might be useful to someone else, too.

#!/bin/bash

echo 'Traversing folder hierarchy...'

CWD=`pwd`


for directory in `find pylux -type d -exec echo {} \;`;
do
    cd $directory
    #echo Entering $directory
    echo -n "" > __init__.py

    for subdirectory in `find . -type d -maxdepth 1 -mindepth 1`;
    do
        subdirectory=`echo $subdirectory | cut -b 3-`
        #echo -n '    ' ...$subdirectory
        #echo -e '\t->\t' import $subdirectory
        echo import $subdirectory >> __init__.py
    done

    for pyfile in *.py ;
    do
        if [ $pyfile = $(echo __init__.py) ]; then
            continue
        fi
        #echo -n '    ' ...$pyfile
        #echo -e '\t->\t' import `echo $pyfile | cut -d . -f 1`
        echo import `echo $pyfile | cut -d . -f 1` >> __init__.py
    done
    cd $CWD

done


for directory in `find pylux -type d -exec echo {} \;`;
do
    echo $directory/__init__.py:
    cat $directory/__init__.py | awk '{ print "\t"$0 }'
done

Thermosphere

Couldn't the same logic be written in __init__.py? Using os.listdir and __import__? – Isidor

Hmm. I suppose so, but I'd have to look into it a bit more. – Thermosphere

Answer 8

I've played around with Joe Kington's Answer and have built a solution that uses globals and get/setattr and thus doesn't need eval. A slight modification is that instead of directly using the packages __path__ for walk_packages, I use the packages parent directory and then only import modules starting with __name__ + ".". This was done to reliably get all subpackages from walk_packages - in my use case I had a subpackage named test which caused pkgutil to iterate over the test package from python's library; furthermore, using __path__ would not recurse into the packages subdirectories. All these issues were observed using jython and python2.5, the code below is only tested in jython thus far.

Also note that OPs question only talks about importing all modules from a package, this code recursively imports all packages too.

from pkgutil import walk_packages
from os import path

__all__ = []
__pkg_prefix = "%s." % __name__
__pkg_path = path.abspath(__path__[0]).rsplit("/", 1)[0] #parent directory

for loader, modname, _ in walk_packages([__pkg_path]):
    if modname.startswith(__pkg_prefix):
        #load the module / package
        module = loader.find_module(modname).load_module(modname)
        modname = modname[len(__pkg_prefix):] #strip package prefix from name
        #append all toplevel modules and packages to __all__
        if not "." in modname:
            __all__.append(modname)
            globals()[modname] = module
        #set everything else as an attribute of their parent package
        else:
            #get the toplevel package from globals()
            pkg_name, rest = modname.split(".", 1)
            pkg = globals()[pkg_name]
            #recursively get the modules parent package via getattr
            while "." in rest:
                subpkg, rest = rest.split(".", 1)
                pkg = getattr(pkg, subpkg)
            #set the module (or package) as an attribute of its parent package
            setattr(pkg, rest, module)

As a future improvement I'll try to make this dynamic with a __getattr__ hook on the package, so the actual modules are only imported when they are accessed...

Fanchie

Answer 9

This works nicely for me in Python 3.3. Note that this works only for submodules which are in files in the same directory as the __init__.py. With some work however it can be enhanced for supporting submodules in directories too.

from glob import iglob
from os.path import basename, relpath, sep, splitext

def import_submodules(__path__to_here):
    """Imports all submodules.
    Import this function in __init__.py and put this line to it:
    __all__ = import_submodules(__path__)"""
    result = []
    for smfile in iglob(relpath(__path__to_here[0]) + "/*.py"):
        submodule = splitext(basename(smfile))[0]
        importstr = ".".join(smfile.split(sep)[:-1])
        if not submodule.startswith("_"):
            __import__(importstr + "." + submodule)
            result.append(submodule)
    return result

Schou

Answer 10

To ensure, that submodule will not be loaded at different location than module.__path__, you can use this approach:

def import_submodules(module, recursive=False):
    """Import all submodules of a module, recursively."""
    from sys import modules
    from pkgutil import walk_packages
    from importlib.util import module_from_spec
    module_stack = [walk_packages(
        module.__path__,
        module.__name__ + '.')
    ]
    while module_stack:
        gen = module_stack.pop()
        for loader, module_name, is_pkg in gen:
            _spec = loader.find_spec(module_name)
            _module = module_from_spec(_spec)
            _spec.loader.exec_module(_module)
            modules[module_name] = _module
            yield _module
            if recursive:
                module_stack.append(
                     walk_packages(
                         _module.__path__,
                         _module.__name__ + '.'
                     )
                )

Vigesimal