How to apply decorators to Cython cpdef functions

Question

I've been playing around with Cython lately and I came across this error when applying a decorator to a Cython function

Cdef functions/classes cannot take arbitrary decorators

Here is the code I was tinkering with:

import functools

def memoize(f):
    computed = {}
    @functools.wraps(f)
    def memoized_f(main_arg, *args, **kwargs):
        if computed.get(main_arg):
            return computed[main_arg]
        computed[main_arg] = f(main_arg, *args, **kwargs)
        return computed[main_arg]
    return memoized_f

@memoize
cpdef int fib(int n):
    return 1 if n < 2 else fib(n - 1) + fib(n - 2)

The error suggests that cdef functions can only take certain decorators. Is it possible to write your own decorators that you can apply to cdef functions?

---

EDIT: For future readers:

The g = plus_one(_g) trick mentioned in @DavidW's answer sort of works. It does not work with recursion. e.g. doing fib = memoize(fib) in my example code does not memoize the recursive calls to fib, although it does memoize the top-level call. i.e. calling fib(5) will memoize the result of the fib(5) call, but it will not memoize the recursive calls (i.e. fib(4), fib(3), fib(2), fib(1))

As @DavidW points out, cdef, cpdef functions are fully determined at compile time; the decoration is a runtime thing and does not update the actual function.

Forfar

Just a quick comment about my trick: note that I've used different names. If you name your cdef function _fib, did fib = memorize(_fib) and made sure you recursively called fib (not _fib) I think it would work, but you'd add a lot of Python overhead on each call. – Laniferous

Answer 1

No - you can't easily write decorators for cdef functions. The decorators cdef functions take are things like cython.boundscheck which control the Cython code generation rather than user generated functions.

The main difference between a cdef function and a def function is that a cdef function has a C interface while a def function becomes a Python callable so can be used from Python (but calling it is slightly less efficient because the arguments have to be passed in terms of PyObjects). [The inside of both a cdef and def function is compiled by Python so the only performance difference comes from the calling overhead]

The usual use of a decorator is to take an arbitrary Python callable and make some modification to it. For example

def plus_one(f):
    def wrapper(*args,**kwargs):
       return f(*args,**kwargs) + 1
    return wrapper

now try to use it on a cdef function

cdef int g(double x, double y):
    # some implementation...

The first problem is that g is translated to C code like int g(double x, double y) which can be represented by a function pointer but not as an arbitrary Python callable like plus_one expects. Second, wrapper has no way of knowing (from a C function pointer) what gs arguments are called (can't do **kwargs) or any easy way of doing the *args expansion.

You can make something like a decorator by taking a specific function pointer type and returning a Python callable:

cdef plus_one(int (*f)(double, double):
    def wrapper(double x, double y):
       return f(x, y) + 1
    return wrapper

cdef int _g(double x, double y):
    # some implementation

g = plus_one(_g) # kind of like a decorator

However, you've lost the whole benefit of using a cdef function since g is now a generic Python callable with all the overhead that goes with it.

---

Addendum: an alternative way of putting it is that decorators are a runtime Python feature (usually run at module import). cdef functions are a compile-time C feature. While it probably wouldn't be impossible to implement something like "compile-time decorators" it would be a pretty significant change to Cython.

Laniferous

Got it. Thanks for the explanation. – Forfar