List all the modules that are part of a python package?

Question

Is there a straightforward way to find all the modules that are part of a python package? I've found this old discussion, which is not really conclusive, but I'd love to have a definite answer before I roll out my own solution based on os.listdir().

Valued

Bonus question: how do you import the found modules nicely? – Valued

What's wrong with reading the source directory? What more information do you need? What's wrong with ls (or dir)? – Vivavivace

@S.Lott: There are more general solutions available, python packages are not always in directories in the filesystem, but can also be inside zips. – Hassiehassin

why reinvent the wheel? If python acquires hypermodules in Python 4, pkgutil and updated with that, my code will still work. I like to use abstractions that are available. Use the obvious method provided, it is tested and known to work. Reimplementing that.. now you have to find and work around every corner case yourself. – Hassiehassin

@S.Lott: Ah, this is all about programmatic discovery of submodules. The code I posted comes from an application that loads plugins that are submodules to the plugins package -- there is no need to keep a manual index in the program, since pkgutil can list the plugins available. – Hassiehassin

@kaizer.se. What manual index? I don't get the question. What's wrong with ls? When I want to know the modules in a package, I use ls -r on the filesystem. Or I unzip the egg and use ls -r. Why is that inadequate? What more is required? – Vivavivace

@S.Lott: So everytime the application starts, it will unzip its own egg if installed inside one just to check this? Please submit a patch against my project to reinvent the wheel in this function: git.gnome.org/cgit/kupfer/tree/kupfer/plugins.py#n17. Please consider both eggs and normal directories, do not exceed 20 lines. – Hassiehassin

@kaiser.se: How does "everytime the application starts, it will unzip its own egg" have anything to do with this question? Please clarify this question. Why is an ls not adequate? Please focus on why -- in this specific question -- the ls is not adequate. I only want clarification on the meaning of this question. – Vivavivace

@S.Lott: Are you asking about a manual ls by me in the shell, or actual os.popen("ls").read() or do you really mean os.listdir? – Hassiehassin

ls in the shell is not adequate. The program should discover itself whenever I or some other dev adds a new plugin by saving a new module (say new.py) inside the plugin subpackage. The program will display a list of discovered plugins. – Hassiehassin

@S.Lott: Why you don't understand that it is relevant is something you can't understand. Discovering this programmatically is about that the application takes interest in the content of a package, not the user. – Hassiehassin

Sorry, is something I can't understand. – Hassiehassin

@static_rtti: Is it possible for you to explain what problem you are solving? Do you recognize the "detect submodules of a package at runtime" usecase? – Hassiehassin

Of course I mean programmatically! Otherwise I wouldn't have mentioned "rolling out my own solution with os.listdir()" – Valued

Answer 1

Yes, you want something based on pkgutil or similar -- this way you can treat all packages alike regardless if they are in eggs or zips or so (where os.listdir won't help).

import pkgutil

# this is the package we are inspecting -- for example 'email' from stdlib
import email

package = email
for importer, modname, ispkg in pkgutil.iter_modules(package.__path__):
    print "Found submodule %s (is a package: %s)" % (modname, ispkg)

How to import them too? You can just use __import__ as normal:

import pkgutil

# this is the package we are inspecting -- for example 'email' from stdlib
import email

package = email
prefix = package.__name__ + "."
for importer, modname, ispkg in pkgutil.iter_modules(package.__path__, prefix):
    print "Found submodule %s (is a package: %s)" % (modname, ispkg)
    module = __import__(modname, fromlist="dummy")
    print "Imported", module

Hassiehassin

what is this importer returned by pkgutil.iter_modules? Can I use it to import a module instead of using this seemly "hackish" __import__(modname, fromlist="dummy") ? – Calm

I was able to use the importer like this: m = importer.find_module(modname).load_module(modname) and then m is the module, so for example: m.myfunc() – Finis

@Finis I was using ur method with python 2.7, but now I need to move on with python 3.4, so you know that in python 3 pkutil.iter_modules yields (module_finder, name, ispkg) instead of (module_loader, name, ispkg). What can I do to make it work like the previous one ? – Granulose

Your first example produces the following error: "AttributeError: 'module' object has no attribute '_path_'" Has this anything to do with Python version? (I use Python 2.7) – Empathic

@Apostolos, you are using only one underscore on either side of path (ie _path_). There should be two on either side, for a total of four (ie __path__). – Wellread

No, I used two undercores, but stackoverflow eats one considering it as italics. – Empathic

@Empathic Here is the answer: Python Glossary "Technically, a package is a Python module with an __path__ attribute." - So we can't get the path from any module, only a package. – Natty

Whatever. I can't remember what is all about after a year and a half. Thanks, anyway! – Empathic

Answer 2

The right tool for this job is pkgutil.walk_packages.

To list all the modules on your system:

import pkgutil
for importer, modname, ispkg in pkgutil.walk_packages(path=None, onerror=lambda x: None):
    print(modname)

Be aware that walk_packages imports all subpackages, but not submodules.

If you wish to list all submodules of a certain package then you can use something like this:

import pkgutil
import scipy
package=scipy
for importer, modname, ispkg in pkgutil.walk_packages(path=package.__path__,
                                                      prefix=package.__name__+'.',
                                                      onerror=lambda x: None):
    print(modname)

iter_modules only lists the modules which are one-level deep. walk_packages gets all the submodules. In the case of scipy, for example, walk_packages returns

scipy.stats.stats

while iter_modules only returns

scipy.stats

The documentation on pkgutil (http://docs.python.org/library/pkgutil.html) does not list all the interesting functions defined in /usr/lib/python2.6/pkgutil.py.

Perhaps this means the functions are not part of the "public" interface and are subject to change.

However, at least as of Python 2.6 (and perhaps earlier versions?) pkgutil comes with a walk_packages method which recursively walks through all the modules available.

Lindemann

walk_packages is now in the documentation: docs.python.org/library/pkgutil.html#pkgutil.walk_packages – Geotectonic

Your second example produces the following error: "AttributeError: 'module' object has no attribute '_path_'" - I didn't test it with 'scipy' but with a few other packages. Has this anything to do with Python version? ( I use Python 2.7) – Empathic

@Apostolos: There should be two underscores (_) before and after path -- that is, use package.__path__ rather than package._path_. It might be easier to try cutting & pasting the code rather than re-typing it. – Lindemann

There were two of them, when I wrote the comment! :) But they have been stripped by the system. My bad; I should have put three undercores. But then, this would be OK if I wanted to use italics, which I didn't! ... It's a loss-loss situation. :) Anyway, when I run the code I used two of them, of course. (I copy-pasted the code.) – Empathic

@Apostolos: Make sure the variable package is pointing to a package, not a module. Modules are files whereas packages are directories. All packages have the __path__ attribute (... unless someone deleted the attribute for some reason.) – Lindemann

I see what you mean @Lindemann ... Indeed, I tried with 'pip' and it works fine. Thanks! – Empathic

Answer 3

This works for me:

import types

for key, obj in nltk.__dict__.iteritems():
    if type(obj) is types.ModuleType: 
        print key

Companionway

This fails in two ways 1. packages don't always explicitly import their submodules into the top-level namespace 2. packages may import other 3rd-party modules into their top-level namespace – Mantua

Answer 4

Thanks to all previous answers, I've just merged them all into one function, which can be easily used to retrieve submodules:

def list_submodules(module) -> list[str]:
    """
    Args:
        module: The module to list submodules from.
    """
    # We first respect __all__ attribute if it already defined.
    submodules = getattr(module, "__all__", None)
    if submodules:
        return submodules

    # Then, we respect module object itself to get imported submodules.
    # Warning: Initially, the module object will respect the `__init__.py`
    # file, if its not exists, the object can partially load submoudles
    # by coda, so can lead `inspect` to return incomplete submodules list.
    import inspect
    submodules = [o[0] for o in inspect.getmembers(module)
                    if inspect.ismodule(o[1])]
    if submodules:
        return submodules

    # Finally we can just scan for submodules via pkgutil.
    import pkgutil
    # pkgutill will invoke `importlib.machinery.all_suffixes()`
    # to determine whether a file is a module, so if you get any
    # submoudles that are unexpected to get, you need to check
    # this function to do the confirmation.
    # If you want to retrive a directory as a submoudle, you will
    # need to clarify this by putting a `__init__.py` file in the
    # folder, even for Python3.
    return [x.name for x in pkgutil.iter_modules(module.__path__)]

Then you can just call it like:

import module
print(list_submodules(module))

path = ...
module = importlib.import_module(path)
print(list_submodules(module))

Publia

Answer 5

I was looking for a way to reload all submodules that I'm editing live in my package. It is a combination of the answers/comments above, so I've decided to post it here as an answer rather than a comment.

package=yourPackageName
import importlib
import pkgutil
for importer, modname, ispkg in pkgutil.walk_packages(path=package.__path__, prefix=package.__name__+'.', onerror=lambda x: None):
    try:
        modulesource = importlib.import_module(modname)
        reload(modulesource)
        print("reloaded: {}".format(modname))
    except Exception as e:
        print('Could not load {} {}'.format(modname, e))

Membranophone

Answer 6

In case you are not only interested in listing module names, but you also want to get a reference to the module objects, this answer is for you:

To list modules, use either pkgutil.iter_modules if you need just the direct children of a module, or pkgutil.walk_packages if you need all descendants of a module. Both return ModuleInfo tuples.

To import modules, there are various suggestions in the existing answers, most of which are not great choices:

• __import__ works if you import a top level module __import__('foo'), but __import__('foo.bar') will also return the foo module, not foo.bar! You can work around this restriction, but it is cumbersome.
• MetaPathFinder.find_module: has been deprecated since Python 3.4 and was removed in 3.12
• MetaPathFinder.find_spec replaces find_module, you can use it by accessing the ModuleInfo.module_finder attribute, but it's a bit verbose:

import pkgutil

submodules = [
    module_info.module_finder.find_spec(
        f"{my_module.__name__}.{module_info.name}"
    ).loader.load_module()
    for module_info in pkgutil.iter_modules(my_module.__path__)
]

My preferred method is to use importlib.import_module in combination with pkgutil.iter_modules:

import importlib
import pkgutil
from types import ModuleType


def get_submodules(module: ModuleType) -> list[ModuleType]:
    return [
        importlib.import_module(f"{module.__name__}.{module_info.name}")
        for module_info in pkgutil.iter_modules(module.__path__)
    ]

a few notes on this solution:

• you can replace pkgutil.iter_modules with pkgutil.walk_packages if needed
• importlib.import_module returns the module specified by the path, not the module at the root of the path, like __import__
• with f"{module.__name__}.{module_info.name}" we make sure that all modules are referenced by an absolute path (modules can be loaded with shorter paths if the parent module has been imported before, but this can cause issues if you want to filter or compare modules)

Jael

Answer 7

Here's one way, off the top of my head:

>>> import os
>>> filter(lambda i: type(i) == type(os), [getattr(os, j) for j in dir(os)])
[<module 'UserDict' from '/usr/lib/python2.5/UserDict.pyc'>, <module 'copy_reg' from '/usr/lib/python2.5/copy_reg.pyc'>, <module 'errno' (built-in)>, <module 'posixpath' from '/usr/lib/python2.5/posixpath.pyc'>, <module 'sys' (built-in)>]

It could certainly be cleaned up and improved.

EDIT: Here's a slightly nicer version:

>>> [m[1] for m in filter(lambda a: type(a[1]) == type(os), os.__dict__.items())]
[<module 'copy_reg' from '/usr/lib/python2.5/copy_reg.pyc'>, <module 'UserDict' from '/usr/lib/python2.5/UserDict.pyc'>, <module 'posixpath' from '/usr/lib/python2.5/posixpath.pyc'>, <module 'errno' (built-in)>, <module 'sys' (built-in)>]
>>> [m[0] for m in filter(lambda a: type(a[1]) == type(os), os.__dict__.items())]
['_copy_reg', 'UserDict', 'path', 'errno', 'sys']

NOTE: This will also find modules that might not necessarily be located in a subdirectory of the package, if they're pulled in in its __init__.py file, so it depends on what you mean by "part of" a package.

Justify

sorry, that has no use. Aside the false positives, it will only find already-imported submodules of packages too. – Hassiehassin