Cython package with __init__.pyx: Possible?

Question

Is it possible to create a Python 2.7 package using __init__.pyx (compiled to __init__.so)? If so how? I haven't had any luck getting it to work.

Here is what I have tried:

• setup.py:

  #!/usr/bin/env python
  
  from distutils.core import setup
  from distutils.extension import Extension
  from Cython.Distutils import build_ext
  
  foo = Extension(name='foo.__init__', sources=['foo/__init__.pyx'])
  bar = Extension(name='foo.bar', sources=['foo/bar.pyx'])
  
  setup(name='foo',
        packages = ['foo'],
        cmdclass={'build_ext':build_ext},
        ext_modules = [foo, bar])
  

• foo/__init__.pyx:

  import foo.bar
  
  cpdef hello_world():
      print "hello world"
      foo.bar.blah()
  

• foo/bar.pyx:

  cpdef blah():
      print "blah"
  

The above has the following behavior:

$ python -c 'import foo; foo.hello_world()'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named foo

I saw Python issue #15576 which was fixed by this Hg commit. Looking at the equivalent Git commit in the Git mirror of the Python Hg repository, I see that the commit is reachable from the Python v2.7.5 tag (as well as all subsequent v2.7.x versions). Was there a regression?

Pd

Just out of curiousity: Why should you do that? – Denounce

Answer 1

According to this really old mailing list post it works if you also have an __init__.py file (the __init__.py file is not used, but seems to be necessary for the directory to be treated as a module, and hence the __init__.so file to be loaded).

If I add __init__.py:

# an exception just to confirm that the .so file is loaded instead of the .py file
raise ImportError("__init__.py loaded when __init__.so should have been loaded")

then your example works on Linux Python 2.7.3:

$ python -c 'import foo; foo.hello_world()'
hello world
blah

This has all the signs of a buggy corner case so probably isn't recommended. Note that on Windows this doesn't seem to work for me giving

ImportError: DLL load failed: %1 is not a valid Win32 application.

---

Addendum (for a little extra context):

This behaviour doesn't seem to be explicitly documented. In the original description of packages from around Python 1.5 era they say:

without the __init__.py, a directory is not recognized as a package

and

Tip: the search order is determined by the list of suffixes returned by the function imp.get_suffixes(). Usually the suffixes are searched in the following order: ".so", "module.so", ".py", ".pyc". Directories don't explicitly occur in this list, but precede all entries in it.

The observed behaviour is certainly consistent with this — __init__.py needed to treat a directory as a package, but .so file is loaded in preference to .py file — but it's hardly unambiguous.

From a Cython point of view this behaviour seems to be been used to compile the standard library (in which case __init__.py would always have been present), or in the testcases given https://github.com/cython/cython/blob/master/tests/build/package_compilation.srctree (and a few other examples too). In these the "srctree" file looks to be expanded into a variety of folders containing __init__.py (and other files) then compiled. It's possible that only having __init__.so was simply never tested.

Disbranch

Is this trick mentioned in the official documentation? (it apparently wasn't at the time of that mailing list thread, but maybe things have changed since then) – Pd

I don't think so. (I found it by accident and then came across the newsgroup post slightly after and realised what I'd done.) I'll have a little look and update my answer if I can find a better source. It also works with an __init__.pyc file too, it seems. – Disbranch

Suggestions for improvement: Rather than assert False, raising an ImportError might be better. Or maybe there's some low-level stuff that can be done with the imp module as a fall-back in case this trick stops working in a future version of Python. – Pd

@RichardHansen I've updated it to raise ImportError (I agree - it's clearly better!) and added a link to the little documentation I can find that supports it. I think it's basically undocumented – Disbranch

Answer 2

Try using a relative import.

in __init__:

from . import bar

Might also be from . import foo. Haven't used python 2 cython in a while.

Veta