Import hooks for PyQt4.QtCore

Question

I'm am attempting to setup some import hooks through sys.meta_path, in a somewhat similar approach to this SO question. For this, I need to define two functions find_module and load_module as explained in the link above. Here is my load_module function,

import imp

def load_module(name, path):
    fp, pathname, description = imp.find_module(name, path)

    try:
        module = imp.load_module(name, fp, pathname, description)
    finally:
        if fp:
             fp.close()
    return module

which works fine for most modules, but fails for PyQt4.QtCore when using Python 2.7:

name = "QtCore"
path = ['/usr/lib64/python2.7/site-packages/PyQt4']

mod = load_module(name, path)

which returns,

Traceback (most recent call last):
   File "test.py", line 19, in <module>
   mod = load_module(name, path)
   File "test.py", line 13, in load_module
   module = imp.load_module(name, fp, pathname, description)
SystemError: dynamic module not initialized properly

The same code works fine with Python 3.4 (although imp is getting deprecated and importlib should ideally be used instead there).

I suppose this has something to do with the SIP dynamic module initialization. Is there anything else I should try with Python 2.7?

Note: this applies both with PyQt4 and PyQt5.

Edit: this may be related to this question as indeed,

cd /usr/lib64/python2.7/site-packages/PyQt4
python2 -c 'import QtCore'

fails with the same error. Still I'm not sure what would be a way around it...

Edit2: following @Nikita's request for a concrete use case example, what I am trying to do is to redirect the import, so when one does import A, what happens is import B. One could indeed think that for this it would be sufficient to do module renaming in find_spec/find_module and then use the default load_module. However, it is unclear where to find a default load_module implementation in Python 2. The closest implementation I have found of something similar is future.standard_library.RenameImport. It does not look like there is a backport of the complete implementation of importlib from Python 3 to 2.

A minimal working example for the import hooks that reproduces this problem can be found in this gist.

Vidavidal

If it may be useful, to give some general context for what I'm trying to do, see the SiQt package, and this problem is discussed in this github issue. – Vidavidal

i really don't understand your problem but what's wrong with __import__('PyQt4.QtCore'). does it lead to infinite recursion? – Culinarian

@Culinarian Nothing is wrong with __import__('A'), but it is equivalent to using import A. What I want is to change what happens when you do that, and in particular run import B, when you import A. This can be done with import hooks in sys.meta_path, but they require lower level functions such as imp.load_module. – Vidavidal

@rth, indeed in the docs about imporylib in Python 2.7 it's written: "This module is a minor subset of what is available in the more full-featured package of the same name from Python 3.1 that provides a complete implementation of import.". There're thoughts about custom imports in PEP302, I'll look at it and share my thoughts in the answer update. – Shaveling

Answer 1

UPD: This part in not really relevant after answer updates, so see UPD below.

Why not just use importlib.import_module, which is available in both Python 2.7 and Python 3:

#test.py

import importlib

mod = importlib.import_module('PyQt4.QtCore')
print(mod.__file__)

on Ubuntu 14.04:

$ python2 test.py 
/usr/lib/python2.7/dist-packages/PyQt4/QtCore.so

Since it's a dynamic module, as said in the error (and the actual file is QtCore.so), may be also take a look at imp.load_dynamic.

Another solution might be to force the execution of the module initialization code, but IMO it's too much of a hassle, so why not just use importlib.

UPD: There are things in pkgutil, that might help. What I was talking about in my comment, try to modify your finder like this:

import pkgutil

class RenameImportFinder(object):

    def find_module(self, fullname, path=None):
        """ This is the finder function that renames all imports like
             PyQt4.module or PySide.module into PyQt4.module """
        for backend_name in valid_backends:
            if fullname.startswith(backend_name):
                # just rename the import (That's what i thought about)
                name_new = fullname.replace(backend_name, redirect_to_backend)
                print('Renaming import:', fullname, '->', name_new, )
                print('   Path:', path)


                # (And here, don't create a custom loader, get one from the
                # system, either by using 'pkgutil.get_loader' as suggested
                # in PEP302, or instantiate 'pkgutil.ImpLoader').

                return pkgutil.get_loader(name_new) 

                #(Original return statement, probably 'pkgutil.ImpLoader'
                #instantiation should be inside 'RenameImportLoader' after
                #'find_module()' call.)
                #return RenameImportLoader(name_orig=fullname, path=path,
                #       name_new=name_new)

    return None

Can't test the code above now, so please try it yourself.

P.S. Note that imp.load_module(), which worked for you in Python 3 is deprecated since Python 3.3.

Another solution is not to use hooks at all, but instead wrap the __import__:

print(__import__)

valid_backends = ['shelve']
redirect_to_backend = 'pickle'

# Using closure with parameters 
def import_wrapper(valid_backends, redirect_to_backend):
    def wrapper(import_orig):
        def import_mod(*args, **kwargs):
            fullname = args[0]
            for backend_name in valid_backends:
                if fullname.startswith(backend_name):
                    fullname = fullname.replace(backend_name, redirect_to_backend)
                    args = (fullname,) + args[1:]
            return import_orig(*args, **kwargs)
        return import_mod
    return wrapper

# Here it's important to assign to __import__ in __builtin__ and not
# local __import__, or it won't affect the import statement.
import __builtin__
__builtin__.__import__ = import_wrapper(valid_backends, 
                                        redirect_to_backend)(__builtin__.__import__)

print(__import__)

import shutil
import shelve
import re
import glob

print shutil.__file__
print shelve.__file__
print re.__file__
print glob.__file__

output:

<built-in function __import__>
<function import_mod at 0x02BBCAF0>
C:\Python27\lib\shutil.pyc
C:\Python27\lib\pickle.pyc
C:\Python27\lib\re.pyc
C:\Python27\lib\glob.pyc

shelve renamed to pickle, and pickle is imported by default machinery with the variable name shelve.

Shaveling

I agree with your two first ideas, unfortunately they do not work, I have tried it before. a) As far as I understand, importlib.import_module is too high level to put in a sys.meta_path import hooks. What happens is when you import a package it will look in sys.meta_path, and if the load_module function uses importlib.import_module it will look in sys.meta_path again where it will find the same load_module function etc, so you get an infinite recursion problem... What is needed is something of lower lever such as imp.find_module or `importlib.machinery.SourceFileLoader – Vidavidal

b) I have tried imp.load_dynamic, it produces the same result (since it must be called by imp.load_module I suppose). c) Yes, I know I'd rather not initialize that module by hand. What I don't understand is why I have to (i.e. what operation importlib.import_module does and imp.load_module doesn't, that make this necessary). The same is true for all PyQt4/PyQt4 submodules. What I'm trying to achieve is import SiQt.QtCore when PyQt4.QtCoreis imported. I know this is possible since python future.standard_library.RenameImport does it in PY2 (essentially it's just import renaming). – Vidavidal

@rth, by the link you provided about import hooks, it says that the meta path finder will call find_spec/find_module recursively for each part of the path. E.g. mpf.find_spec("PyQt4", None, None) and then one more mpf.find_spec("PyQt4.QtCore", PyQt4.__path__, None). So if you're hooking in place of find_spec or in some other part of mpf, may be replace PyQt4 with SiQt in the name string and then call the default machinery to let it load SiQt by itself. If I'm wrong, please, provide some code used for hooks to better understand what you are trying to accomplish. – Shaveling

I agree that using the default machinery for load_module would have been nice. See Edit2 in the question above. – Vidavidal

@rth, if hooks won't work, you can wrap the __import__, check out the answer update. – Shaveling

Answer 2

When finding a module which is part of package like PyQt4.QtCore, you have to recursively find each part of the name without .. And imp.load_module requires its name parameter be full module name with . separating package and module name.

Because QtCore is part of a package, you shoud do python -c 'import PyQt4.QtCore' instead. Here's the code to load a module.

import imp

def load_module(name):
    def _load_module(name, pkg=None, path=None):
        rest = None
        if '.' in name:
            name, rest = name.split('.', 1)
        find = imp.find_module(name, path)
        if pkg is not None:
            name = '{}.{}'.format(pkg, name)
        try:
            mod = imp.load_module(name, *find)
        finally:
            if find[0]:
                find[0].close()
        if rest is None:
            return mod 
        return _load_module(rest, name, mod.__path__)
    return _load_module(name)

Test;

print(load_module('PyQt4.QtCore').qVersion())  
4.8.6

Nagana