Cython C-level interface of package: *.pxd files are not found

Question

In a nutshell

I try to compile a cython extension called extension2 that cimports a file extension from a self-created package. When building extension2, I get the error that extension.pxd is not found though this file is exactly at the sepcified path.

Details

I am building two packages involving cython, a package A and a package B that depends on A. A is a subpacke of a namespace package nsp. That is, the folder structure looks as follows:

├── nsp
│   └── A
|       ├── extension.pxd
|       ├── extension.pyx
│       └── __init__.py
└── setup.py

Here, setup.py reads as follows:

from setuptools import setup
from setuptools.extension import Extension

# factory function
def my_build_ext(pars):
    # import delayed:
    from setuptools.command.build_ext import build_ext as _build_ext

    # include_dirs adjusted: 
    class build_ext(_build_ext):
        def finalize_options(self):
            _build_ext.finalize_options(self)
            # Prevent numpy from thinking it is still in its setup process:
            __builtins__.__NUMPY_SETUP__ = False
            import numpy
            self.include_dirs.append(numpy.get_include())

    #object returned:
    return build_ext(pars)

extensions = [Extension(nsp.A.extension, ['nsp/A/extension.cpp'])]

setup(
    cmdclass={'build_ext' : my_build_ext},
    setup_requires=['numpy'],
    install_requires=['numpy'], 
    packages=['nsp.A'],
    ext_modules=extensions
    package_data={
        'nsp/A': ['*.pxd', '*.pyx']
    },
)

The setup file is inspired by add-numpy-get-include-argument-to-setuptools-without-preinstalled-numpy and distributing-cython-modules. The cython files were already successfully transformed to .cpp files with another script.

I install the the package A with

pip install .

in the directory of the setup.py. Everything works as desired, and I can find all files of the package under ...\Anaconda3\Lib\site-packages\nsp\A, including the *.pxd files.

Now I seek to create a *.cpp file for an extension2 in order to package it later in the second package B. The file extension2.pxd reads

from nsp.A.extension cimport mymethod

The script to create the *.cpp file reads

from distutils.core import setup, Extension
from Cython.Build import cythonize
import numpy as np
import sys
print(sys.executable)

NAME = 'extension2'
extensions = [Extension(NAME, [NAME+'.pyx'],
                        include_dirs=[np.get_include()]
                        )
              ]

setup(name=NAME,
      ext_modules = cythonize(extensions, language="c++", 
                              compiler_directives=compiler_directives),
      include_dirs=[np.get_include()]
      ) 

When I run this script with python myscript build_ext --inplace, I get an error indicating the pxd file is missing:

from nsp.A.extension cimport mymethod
^
------------------------------------------------------------

.\extension2.pxd:11:0: 'nsp\A\extension.pxd' not found

However, this file exists exactly there. (sys.executable is the Anaconda3 folder that contains the installed package) How can I resolve the issue?

Additional info

I am using python 3.7 on Windows x64

Opinion

And you are sure pip is the right pip? I.e. pip for python3.7 and not python 2? – Preoccupy

Otherwise, check the python path in setup.py – Preoccupy

@Preoccupy I am certain that I am using the correct Python installation. I have only one installation, and importing the compiled files (not the pxd) works well. For the same reason I believe the path in setup.py is correct. – Opinion

Answer 1

Cython does not support implicit namespace packages as of yet. That is, cython searches only subdirectories that contain a file init.*, whereby * can be anything from py, pyc, pyx, and pxd.

I have created a bugtracker report for this issue, in case you want to follow up on whether the issue has been fixed in a newer version yet (I worked with Cython 0.29.14).

Until then, a workaround is to create an empty file __init__.pxd in the folder nsp. This file should be ignored by python, as it is not a *.py file, and lets cython search the subdirectories for packages. The file structure then reads as follows:

├── nsp
│   ├── __init__.pxd
│   └── A
|       ├── extension.pxd
|       ├── extension.pyx
│       └── __init__.py
└── setup.py

To install the additional file __init__.pxd in the namespace package, change the packages argument of setup(...) to packages=['nsp', 'nsp.A'] and the package_data argument to package_data={'': ['*.pxd', '*.pyx']}.

Edit:

The bug has been known to the cython developers and will be fixed in version 3. See Fix for cimport from PEP420 namespace.

Opinion