Importing module from string variable using "__import__" gives different results than a normal import statement

Question

I'm working on a documentation (personal) for nested matplotlib (MPL) library, which differs from MPL own provided, by interested submodule packages. I'm writing Python script which I hope will automate document generation from future MPL releases.

I selected interested submodules/packages and want to list their main classes from which I'll generate list and process it with pydoc.

The problem is that I can't find a way to instruct Python to load a submodule from a string. Here is an example of what I tried:

import matplotlib.text as text
x = dir(text)

i = __import__('matplotlib.text')
y = dir(i)

j = __import__('matplotlib')
z = dir(j)

And here is a 3-way comparison of above lists through pprint:

I don't understand what's loaded in y object - it's base matplotlib plus something else, but it lacks information that I wanted and that is main classes from matplotlib.text package. It's the top blue coloured part on screenshot (x list).

Lightship

Can you explain why you need to use __import__(str) rather than the standard import statemetn? – Gearhart

It's because I'll process lists which items are MPL submodules and get their methods paths – Lightship

@Gearhart - c'mon - there are endless ideas where you'd want this functionality. When you have a textual configuration of libraries, you can load them by name, which wouldn't quite work with the import statement. Here's one example of use: djangosnippets.org/snippets/3048 – Transparency

Answer 1

The __import__ function can be a bit hard to understand.

If you change

i = __import__('matplotlib.text')

to

i = __import__('matplotlib.text', fromlist=[''])

then i will refer to matplotlib.text.

In Python 3.1 or later, you can use importlib:

import importlib

i = importlib.import_module("matplotlib.text")

Some notes

• If you're trying to import something from a sub-folder e.g. ./feature/email.py, the code will look like importlib.import_module("feature.email")

• Before Python 3.3 you could not import anything if there was no __init__.py in the folder with file you were trying to import (see caveats before deciding if you want to keep the file for backward compatibility e.g. with pytest).

Dorita

importlib should be avaliable on pypi for < python2.7 – Strigil

For anyone who's coming here from Google. It should be noted that if you're trying to import something from a sub-folder (for example, ./feature/email.py) the code will look like importlib.import_module("feature.email") – Arabella

Finally, also remember that you can't import anything if there is no __init__.py in the folder with file you are trying to import. – Arabella

@Dorita This is for import moduleName where moduleName is string. How about from moduleName import * ? – Garrison

@NamGVU: What is the problem? The question title is "import module from string variable". Hasn't that been answered? – Dorita

You have perfectly answered the OP. I just expand the topic. Thank you any way! – Garrison

Just found answer for my question here in case any one needs it https://mcmap.net/q/20621/-import-from-module-by-importing-via-string – Garrison

and what if I want to import something like ../../feature/email.py ? – Palua

I found the answer on how to import from parent directory here. – Palua

Due to implicit namespace packages, it is possible to import from folders without an __init__.py. – Magnificent

Answer 2

importlib.import_module is what you are looking for. It returns the imported module.

import importlib

# equiv. of your `import matplotlib.text as text`
text = importlib.import_module('matplotlib.text')

You can thereafter access anything in the module as text.myclass, text.myfunction, etc.

Phenomenology

An alternative is imp.load_source(..) using the imp module as suggested in this answer stackoverflow.com/questions/67631/… – Dissociable

@Phenomenology This is for import moduleName where moduleName is string. How about from moduleName import * ? – Garrison

Anyone figure out @NamGVU question? – Singe

Answer 3

spent some time trying to import modules from a list, and this is the thread that got me most of the way there - but I didnt grasp the use of ___import____ -

so here's how to import a module from a string, and get the same behavior as just import. And try/except the error case, too. :)

  pipmodules = ['pycurl', 'ansible', 'bad_module_no_beer']
  for module in pipmodules:
      try:
          # because we want to import using a variable, do it this way
          module_obj = __import__(module)
          # create a global object containging our module
          globals()[module] = module_obj
      except ImportError:
          sys.stderr.write("ERROR: missing python module: " + module + "\n")
          sys.exit(1)

and yes, for python 2.7> you have other options - but for 2.6<, this works.

Smearcase

Answer 4

Apart from using the importlib one can also use exec method to import a module from a string variable.

Here I am showing an example of importing the combinations method from itertools package using the exec method:

MODULES = [
    ['itertools','combinations'],
]

for ITEM in MODULES:
    import_str = "from {0} import {1}".format(ITEM[0],', '.join(str(i) for i in ITEM[1:]))
    exec(import_str)

ar = list(combinations([1, 2, 3, 4], 2))
for elements in ar:
    print(elements)

Output:

(1, 2)
(1, 3)
(1, 4)
(2, 3)
(2, 4)
(3, 4)

Betthel

Answer 5

You can also use exec built-in function that execute any string as a Python code.

In [1]: module = 'pandas'
   ...: function = 'DataFrame'
   ...: alias = 'DF'

In [2]: exec(f"from {module} import {function} as {alias}")

In [3]: DF
Out[3]: pandas.core.frame.DataFrame

For me this was the most readable way to solve my problem.

Cumulate

Answer 6

Module auto-install & import from list

Below script works fine with both submodules and pseudo submodules.

# PyPI imports
import pkg_resources, subprocess, sys

modules   = {'lxml.etree', 'pandas', 'screeninfo'}
required  = {m.split('.')[0] for m in modules}
installed = {pkg.key for pkg in pkg_resources.working_set}
missing   = required - installed

if missing:
    subprocess.check_call([sys.executable, '-m', 'pip', 'install', '--upgrade', 'pip'])
    subprocess.check_call([sys.executable, '-m', 'pip', 'install', *missing])

for module in set.union(required, modules):
    globals()[module] = __import__(module)

Tests:

print(pandas.__version__)
print(lxml.etree.LXML_VERSION)

Nullifidian

Answer 7

I developed these 3 useful functions:

def loadModule(moduleName):
    module = None
    try:
        import sys
        del sys.modules[moduleName]
    except BaseException as err:
        pass
    try:
        import importlib
        module = importlib.import_module(moduleName)
    except BaseException as err:
        serr = str(err)
        print("Error to load the module '" + moduleName + "': " + serr)
    return module

def reloadModule(moduleName):
    module = loadModule(moduleName)
    moduleName, modulePath = str(module).replace("' from '", "||").replace("<module '", '').replace("'>", '').split("||")
    if (modulePath.endswith(".pyc")):
        import os
        os.remove(modulePath)
        module = loadModule(moduleName)
    return module

def getInstance(moduleName, param1, param2, param3):
    module = reloadModule(moduleName)
    instance = eval("module." + moduleName + "(param1, param2, param3)")
    return instance

And everytime I want to reload a new instance I just have to call getInstance() like this:

myInstance = getInstance("MyModule", myParam1, myParam2, myParam3)

Finally I can call all the functions inside the new Instance:

myInstance.aFunction()

The only specificity here is to customize the params list (param1, param2, param3) of your instance.

Desiccator