How to test functions cdef'd in Cython?

Question

I have a .pyx file in which I define some functions, e.g.

cdef double foo(double a) nogil:
    return 3. * a

How could I unit test the behavior of such functions outside the pyx file? Since they are cdef'd, I am not able to simply import them...

Eruptive

How about writing a def or cpdef that calls foo? – Interracial

Is there a good solution to this that does not involve cpdef? – Eer

Answer 1

To test cdef-fuctionality you need to write your tests in Cython. One could try to use cpdef-functions, however not all signatures can be used in this case (for example signatures using pointers like int *, float * and so on).

To access the cdef-functions you will need to "export" them via a pxd-file (the same can be done also for cdef-functions of extension types ):

#my_module.pyx:
cdef double foo(double a) nogil:
    return 3. * a

#my_module.pxd:
cdef double foo(double a) nogil

Now the functionality can be cimported and tested in a Cython-tester:

#test_my_module.pyx
cimport my_module

def test_foo():
    assert my_module.foo(2.0)==6.0
    print("test ok")

test_foo()

And now

>>> cythonize -i my_module.pyx
>>> cythonize -i test_my_module.pyx 
>>> python -c "import test_my_module"
test ok

Where to go from there depends on your testing infrastructure.

---

For example if you use unittest-module, then you could use pyximport to cythonize/load the test-module inspect it and convert all test cases into unittest-test cases or use unittest directly in your cython code (probably a better solution).

Here is a proof of concept for unittest:

#test_my_module.pyx
cimport my_module
import unittest

class CyTester(unittest.TestCase): 
    def test_foo(self):
        self.assertEqual(my_module.foo(2.0),6.0)

Now we only need to translate and to import it in pure python to be able to unittest it:

#test_cy.py 
import pyximport;
pyximport.install(setup_args = {"script_args" : ["--force"]},
                  language_level=3)

# now drag CyTester into the global namespace, 
# so tests can be discovered by unittest
from test_my_module import *

And now:

>>> python -m unittest test_cy.py
.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

Btw, there is no need to cythonize pyx-modules explicitly - pyximport does it for us automatically.

A word of warning: pyximport caches cythonized c-files in ~/.pyxbld (or similar on other OSes) and as long as test_my_module.pyx has not changed the extension isn't rebuild, even if its depenencies where changed. This might be a problem (among others), when my_module changes and it leads to binary incompatibility (luckily python warns if this is the case).

By passing setup_args = {"script_args" : ["--force"]} we force a rebuild.

Another option is to delete the cached-files (one could use a temporary directory, for example created with tempfile.TemporaryDirectory(), via pyximport.install(build_dir=...)), which has the advantage of keeping the system clean.

The explicit language_level (what is language_level?) is needed in order to prevent warnings.

---

If you use a virtual environment and install you cython-package via setup.py (or a similar workflow), you need to make sure that *.pxd files are also included into installation, i.e. your setup-file needs to be augmented with:

from setuptools import setup, find_packages, Extension
# usual stuff for cython-modules here
...

kwargs = {
      # usual stuff for cython-modules here
      ...

      #ensure pxd-files:
      'package_data' : { 'my_module': ['*.pxd']},
      'include_package_data' : True,
      'zip_safe' : False  #needed because setuptools are used
}

setup(**kwargs)

Merchantable

Answer 2

Although mentioned earlier, the easiest way is to change the cdef declaration for cpdef :

cpdef double foo(double a) nogil:
    return 3. * a

No need to change anything else. For most purposes they are practically the same, cpdef has slightly more overhead but plays nicer with inheritance, see details here:

The directive cpdef makes two versions of the method available; one fast for use from Cython and one slower for use from Python. Then:

This does slightly more than providing a python wrapper for a cdef method: unlike a cdef method, a cpdef method is fully overridable by methods and instance attributes in Python subclasses. It adds a little calling overhead compared to a cdef method.

Klehm

And if my cdef function has the following signature: cdef foo(double* a)? – Merchantable

Then you can use your answer, which is still correct :) – Klehm