How to load all modules in a folder?

Question

Could someone provide me with a good way of importing a whole directory of modules?
I have a structure like this:

/Foo
    bar.py
    spam.py
    eggs.py

I tried just converting it to a package by adding __init__.py and doing from Foo import * but it didn't work the way I had hoped.

Zayin

Can you define "didn't work"? What happened? What error message did you get? – Unroll

Is this Pythonic or recommended? "Explicit is better than implicit." – Gailgaile

Explicit is indeed better. But would you really like to address these annoying 'not found' messages everytime you add a new module. I think that if you have a package directory containing many small module-functions, then this is the nicest way to go about it. My assumptions are: 1. module are rather simple 2. you use the package for code tidiness – Featherstitch

Note that most (all?) answers on this page will not give you autocomplete in some IDEs and give errors in some linters (they won't "know" what the module is exporting since they don't execute the code). If you're writing a public package and want good 'developer experience', I believe manually declaring what you export is the only way. – Dario

Answer 1

List all python (.py) files in the current folder and put them as __all__ variable in __init__.py

from os.path import dirname, basename, isfile, join
import glob
modules = glob.glob(join(dirname(__file__), "*.py"))
__all__ = [ basename(f)[:-3] for f in modules if isfile(f) and not f.endswith('__init__.py')]

Francinefrancis

Basically so I can drop python files into a directory with no further configuration and have them be executed by a script running somewhere else. – Zayin

@NiallDouglas this answer is for a specific question which OP asked, he didn't have a zip file and pyc files can be included easily, and you are forgetting .pyd or .so libs etc too – Francinefrancis

True, the OP didn't ask for running inside zip support, however he did appear to be asking for boilerplate code which would import the contents of a directory. That boilerplate ought to cope with most common situations - it's not hard, just use pkgutil, not os.listdir(), like the other poster below. – Credo

The only thing i would add is if not os.path.basename(f).startswith('_') or at the very least if not f.endswith('__init__.py') to the end of the list comprehension – Pressurize

If you have installed 'path.py' (pip install path.py), you can use __all__ = [f.namebase for f in path(__file__).parent.glob('*.py')], which I think looks cleaner. I'm also in agreement that __init__.py should be filtered from the list. – Jaws

os.listdir() is a good alternative to glob.glob(): maybe subjective, but IMHO is a bit more standard/readable way to get files in a dir. Or os.walk() if you want recursion – Guinea

To make it more robust, also make sure os.path.isfile(f) is True. That would filter out broken symlinks and directories like somedir.py/ (corner-case, I admit, but still...) – Guinea

How can I add the filter @MestreLion? I see no place for if statement in the code above... – Hendrik

@TomášZato added if condition to list comprehension – Francinefrancis

Add from . import * after setting __all__ if you want submodules to be available using . (e.g. as module.submodule1, module.submodule2, etc.). – Disinfest

I check again today and it seems not working any more #1057931 – Tartuffe

To be crossplatform you want to use os.path.join() to construct your path, in case the os.path.sep isn't '/' on the user's troublemaking OS (like Windows): modules = glob.glob(os.path.join(dirname(__file__), '*.py')) – Hooker

@EvanFosmark not exactly, since if the script is running, the files already have been imported. But great point, if the file is dropped on a server, maybe no one will notice when restarting the app. – Udela

__all__ = [f for f in glob.glob(__file__.replace('__init__', '*[!_*]'))] – Bedlamite

Answer 2

Add the __all__ Variable to __init__.py containing:

__all__ = ["bar", "spam", "eggs"]

See also http://docs.python.org/tutorial/modules.html

Helices

Yes, yes, but is there any way of having it be dynamic? – Zayin

Combination of os.listdir(), some filtering, stripping of .py extension and __all__. – Warnock

Not working for me as sample code here github.com/namgivu/python-import-all/blob/master/error_app.py . Maybe I miss something there? – Tartuffe

I found out myself - to use the variable/object defined in those modules, we have to use the full reference path e.g. moduleName.varName ref. https://mcmap.net/q/47754/-use-39-import-module-39-or-39-from-module-import-39 – Tartuffe

@NamGVU: This code in my answer to a related question will import all the public sub-modules' names into the the package's namespace. – Defense

Answer 3

Update in 2017: you probably want to use importlib instead.

Make the Foo directory a package by adding an __init__.py. In that __init__.py add:

import bar
import eggs
import spam

Since you want it dynamic (which may or may not be a good idea), list all py-files with list dir and import them with something like this:

import os
for module in os.listdir(os.path.dirname(__file__)):
    if module == '__init__.py' or module[-3:] != '.py':
        continue
    __import__(module[:-3], locals(), globals())
del module

Then, from your code do this:

import Foo

You can now access the modules with

Foo.bar
Foo.eggs
Foo.spam

etc. from Foo import * is not a good idea for several reasons, including name clashes and making it hard to analyze the code.

Tactical

Not bad, but don't forget that you can import .pyc and .pyo files too. – Zayin

tbh, i find __import__ hackish, i think it would be better to add the names to __all__ and then put from . import * at the bottom of the script – Siltstone

I think this is nicer than the glob version. – Medalist

Instead of del, you can do import os as _os – Hereabout

__import__ is not for general uses, it used by interpreter, use importlib.import_module() instead. – Druggist

importlib didn't exist when I wrote this answer, and import works just fine. But yes, you can use importlib as well in 2.7 and later. – Tactical

what is del module for, and why is it outside the for-loop block? – Tarrance

To delete the module variable, which is otherwise keeping a reference to the last module name and to be honest I don't know why I put it in there, it's not really very useful. In any case, this answer is old, use importlib instead. :-) – Tactical

The first example was very helpful, thanks! Under Python 3.6.4 I had to do from . import eggs etc. in the __init__.py before Python could import. With only import eggs I get ModuleNotFoundError: No module named 'eggs' when trying to import Foo in the main.py in the directory above. – Cassaba

What is this difference between this and __all__=['bar,'eggs',ham']? It seems cleaner and more in keeping with the original intention of the import statement. Is there anything else? – Cornstalk

The idea here is to load all modules in a folder, which means a module will be loaded simply if it's added to the folder, without changing any code. I agree it's generally a thing to avoid. – Tactical

Please define 'today'. I assume you mean from a particular version of Python? Which? Thanks. – Fi

@MichaelScheper Any version of Python that includes importlib. And if your version doesn't, please upgrade. – Tactical

This answer was very helpful! But it didn't work right out of the box for some reason on Py3.5.6. I had to append the . to the beginning of the module string for this to work , otherwise it gave a "no module named <filename>` ImportError. Also, when switching to importlib.import_module( ) you have to add the argument importlib.import_module( modulebase, parent=__name__ ) for it to go into the module's namespace. Let me know if you want a full code example. – Limpopo

Answer 4

Python, include all files under a directory:

For newbies who just can't get it to work who need their hands held.

1. Make a folder /home/el/foo and make a file main.py under /home/el/foo Put this code in there:

   from hellokitty import *
   spam.spamfunc()
   ham.hamfunc()
   

2. Make a directory /home/el/foo/hellokitty

3. Make a file __init__.py under /home/el/foo/hellokitty and put this code in there:

   __all__ = ["spam", "ham"]
   

4. Make two python files: spam.py and ham.py under /home/el/foo/hellokitty

5. Define a function inside spam.py:

   def spamfunc():
     print("Spammity spam")
   

6. Define a function inside ham.py:

   def hamfunc():
     print("Upgrade from baloney")
   

7. Run it:

   el@apollo:/home/el/foo$ python main.py 
   spammity spam
   Upgrade from baloney
   

Uptotheminute

Not just for newbies, but also experienced Python devs who like clear answers. Thanks. – Fi

But using import * is considered bad Python coding practice. How do you do this without that? – Lingam

@RachaelBlake I think it's considered bad practice in cases where you're importing all the functions from one namespace into the global namespace. Because you could have overlaps there. In this situation, notice that the namespace of the imported filenames is required (I.E., spam.spamfunc(),). spamfunc and hamfunc aren't in the global namespace, so there's no potential for overlap. – Scattering

Answer 5

Expanding on Mihail's answer, I believe the non-hackish way (as in, not handling the file paths directly) is the following:

1. create an empty __init__.py file under Foo/
2. Execute

import pkgutil
import sys


def load_all_modules_from_dir(dirname):
    for importer, package_name, _ in pkgutil.iter_modules([dirname]):
        full_package_name = '%s.%s' % (dirname, package_name)
        if full_package_name not in sys.modules:
            module = importer.find_module(package_name
                        ).load_module(full_package_name)
            print module


load_all_modules_from_dir('Foo')

You'll get:

<module 'Foo.bar' from '/home/.../Foo/bar.pyc'>
<module 'Foo.spam' from '/home/.../Foo/spam.pyc'>

Renteria

This is most of the way there to a correct answer - it handles ZIP archives, but doesn't write init nor import. See automodinit below. – Credo

Another thing: the above example doesn't check sys.modules to see if the module is already loaded. Without that check the above will load the module a second time :) – Credo

When I run load_all_modules_from_dir('Foo/bar') with your code I get "RuntimeWarning: Parent module 'Foo/bar' not found while handling absolute import" - to suppress this, I have to set full_package_name = '.'.join(dirname.split(os.path.sep) + package_name]) and also import Foo.bar – Ona

Those RuntimeWarning messages can also be avoided by not using full_package_name at all: importer.find_module(package_name).load_module(package_name). – Mayers

The RuntimeWarning errors can also be avoided (in an arguably ugly way) by importing the parent (AKA dirname). One way to do that is - if dirname not in sys.modules: pkgutil.find_loader(dirname).load_module(dirname). Of course, that only works if dirname is a single-component relative path; no slashes. Personally, I prefer @Artfunkel's approach of using the base package_name instead. – Orthocephalic

Answer 6

I know I'm updating a quite old post, and I tried using automodinit, but found out it's setup process is broken for python3. So, based on Luca's answer, I came up with a simpler answer - which might not work with .zip - to this issue, so I figured I should share it here:

within the __init__.py module from yourpackage:

#!/usr/bin/env python
import os, pkgutil
__all__ = list(module for _, module, _ in pkgutil.iter_modules([os.path.dirname(__file__)]))

and within another package below yourpackage:

from yourpackage import *

Then you'll have all the modules that are placed within the package loaded, and if you write a new module, it'll be automagically imported as well. Of course, use that kind of things with care, with great powers comes great responsibilities.

Flanna

Answer 7

I got tired of this problem myself, so I wrote a package called automodinit to fix it. You can get it from http://pypi.python.org/pypi/automodinit/.

Usage is like this:

1. Include the automodinit package into your setup.py dependencies.
2. Replace all __init__.py files like this:

__all__ = ["I will get rewritten"]
# Don't modify the line above, or this line!
import automodinit
automodinit.automodinit(__name__, __file__, globals())
del automodinit
# Anything else you want can go after here, it won't get modified.

That's it! From now on importing a module will set __all__ to a list of .py[co] files in the module and will also import each of those files as though you had typed:

for x in __all__: import x

Therefore the effect of "from M import *" matches exactly "import M".

automodinit is happy running from inside ZIP archives and is therefore ZIP safe.

Niall

Credo

pip can't download automodinit because there's nothing uploaded on pypi for it. – Odessa

Thanks for the bug report on github. I've fixed this in v0.13. Niall – Credo

Answer 8

import pkgutil
__path__ = pkgutil.extend_path(__path__, __name__)
for imp, module, ispackage in pkgutil.walk_packages(path=__path__, prefix=__name__+'.'):
  __import__(module)

Huntsman

Answer 9

I have also encountered this problem and this was my solution:

import os

def loadImports(path):
    files = os.listdir(path)
    imps = []

    for i in range(len(files)):
        name = files[i].split('.')
        if len(name) > 1:
            if name[1] == 'py' and name[0] != '__init__':
               name = name[0]
               imps.append(name)

    file = open(path+'__init__.py','w')

    toWrite = '__all__ = '+str(imps)

    file.write(toWrite)
    file.close()

This function creates a file (in the provided folder) named __init__.py, which contains an __all__ variable that holds every module in the folder.

For example, I have a folder named Test which contains:

Foo.py
Bar.py

So in the script I want the modules to be imported into I will write:

loadImports('Test/')
from Test import *

This will import everything from Test and the __init__.py file in Test will now contain:

__all__ = ['Foo','Bar']

Gilson

Answer 10

When from . import * isn't good enough, this is an improvement over the answer by ted. Specifically, the use of __all__ is not necessary with this approach.

"""Import all modules that exist in the current directory."""
# Ref https://stackoverflow.com/a/60861023/
from importlib import import_module
from pathlib import Path

for f in Path(__file__).parent.glob("*.py"):
    module_name = f.stem
    if (not module_name.startswith("_")) and (module_name not in globals()):
        import_module(f".{module_name}", __package__)
    del f, module_name
del import_module, Path

Note that module_name not in globals() is intended to avoid reimporting the module if it's already imported, as this can risk cyclic imports.

Panthea

Is the line line really needed if this is encapsulated in a function? – Heger

@Heger The del lines are not needed if the snippet is in a function. – Panthea

Answer 11

Anurag Uniyal answer with suggested improvements!

#!/usr/bin/python
# -*- encoding: utf-8 -*-

import os
import glob

all_list = list()
for f in glob.glob(os.path.dirname(__file__)+"/*.py"):
    if os.path.isfile(f) and not os.path.basename(f).startswith('_'):
        all_list.append(os.path.basename(f)[:-3])

__all__ = all_list  

Lipocaic

Answer 12

This is the best way i've found so far:

from os.path import dirname, join, isdir, abspath, basename
from glob import glob
pwd = dirname(__file__)
for x in glob(join(pwd, '*.py')):
    if not x.startswith('__'):
        __import__(basename(x)[:-3], globals(), locals())

Rattler

abspath and isdir is unused ;) – Guttate

Answer 13

Using importlib the only thing you've got to add is

from importlib import import_module
from pathlib import Path

__all__ = [
    import_module(f".{f.stem}", __package__)
    for f in Path(__file__).parent.glob("*.py")
    if "__" not in f.stem
]
del import_module, Path

Grimalkin

This leads to a legitimate mypy issue: error: Type of __all__ must be "Sequence[str]", not "List[Module]". Defining __all__ is not required if this import_module based approach is used. – Panthea

Answer 14

I'd like to add to Anurag Uniyal's answer. You can make it even simpler and get rid of a lot of the imports. Contents of the __init__.py file:

from os import listdir
from os.path import dirname
__all__ = [i[:-3] for i in listdir(dirname(__file__)) if not i.startswith('__') and i.endswith('.py')]

Mighell

Answer 15

I've created a module for that, which doesn't rely on __init__.py (or any other auxiliary file) and makes me type only the following two lines:

import importdir
importdir.do("Foo", globals())

Feel free to re-use or contribute: http://gitlab.com/aurelien-lourot/importdir

Torytoryism

Answer 16

Anurag's example with a couple of corrections:

import os, glob

modules = glob.glob(os.path.join(os.path.dirname(__file__), "*.py"))
__all__ = [os.path.basename(f)[:-3] for f in modules if not f.endswith("__init__.py")]

Reverse

Answer 17

See that your __init__.py defines __all__. The modules - packages doc says

The __init__.py files are required to make Python treat the directories as containing packages; this is done to prevent directories with a common name, such as string, from unintentionally hiding valid modules that occur later on the module search path. In the simplest case, __init__.py can just be an empty file, but it can also execute initialization code for the package or set the __all__ variable, described later.

...

The only solution is for the package author to provide an explicit index of the package. The import statement uses the following convention: if a package’s __init__.py code defines a list named __all__, it is taken to be the list of module names that should be imported when from package import * is encountered. It is up to the package author to keep this list up-to-date when a new version of the package is released. Package authors may also decide not to support it, if they don’t see a use for importing * from their package. For example, the file sounds/effects/__init__.py could contain the following code:

__all__ = ["echo", "surround", "reverse"]

This would mean that from sound.effects import * would import the three named submodules of the sound package.

Alodie

Answer 18

Look at the pkgutil module from the standard library. It will let you do exactly what you want as long as you have an __init__.py file in the directory. The __init__.py file can be empty.

Flocky

Answer 19

Here is a solution, with which you do not have to write the file name. Just add this code snippet to your __init__.py

from inspect import isclass
from pkgutil import iter_modules
from pathlib import Path
from importlib import import_module

# iterate through the modules in the current package
package_dir = Path(__file__).resolve().parent
for (_, module_name, _) in iter_modules([package_dir]):

    # import the module and iterate through its attributes
    module = import_module(f"{__name__}.{module_name}")
    for attribute_name in dir(module):
        attribute = getattr(module, attribute_name)

        if isclass(attribute):            
            # Add the class to this package's variables
            globals()[attribute_name] = attribute

Source

Weitzel

Answer 20

None of the solutions was working for me in Python 3.9.5, Flask 2.2.2, the module being a directory 2 levels down the cwd. This is my solution:

import importlib
import pathlib
import re

path = pathlib.Path(__file__).parent.absolute()
names = [x.name[:-3] for x in path.iterdir() if x.is_file() and re.search("^[a-z]*\.py$", x.name)]
for name in names:
    importlib.import_module(f".{name}", __name__)

Lavish

Answer 21

Just import them by importlib and add them to __all__ (add action is optional) in recurse in the __init__.py of package.

/Foo
    bar.py
    spam.py
    eggs.py
    __init__.py

# __init__.py
import os
import importlib
pyfile_extes = ['py', ]
__all__ = [importlib.import_module('.%s' % filename, __package__) for filename in [os.path.splitext(i)[0] for i in os.listdir(os.path.dirname(__file__)) if os.path.splitext(i)[1] in pyfile_extes] if not filename.startswith('__')]
del os, importlib, pyfile_extes

Mooch

Where is pyfile_extes defined? – Huxley

sorry for missing it, now fixed. It is the extention of python file you want to import, usually just py – Mooch

Answer 22

I had a nested directory structure i.e. I had multiple directories inside the main directory that contained the python modules.

I added the following script to my __init__.py file to import all the modules

import glob, re, os 

module_parent_directory = "path/to/the/directory/containing/__init__.py/file"

owd = os.getcwd()
if not owd.endswith(module_parent_directory): os.chdir(module_parent_directory)

module_paths = glob.glob("**/*.py", recursive = True)

for module_path in module_paths:
    if not re.match( ".*__init__.py$", module_path):
        import_path = module_path[:-3]
        import_path = import_path.replace("/", ".")
        exec(f"from .{import_path} import *")

os.chdir(owd)

Probably not the best way to achieve this, but I couldn't make anything else work for me.

Piker