how to reload a cython module interactively using pyximport

Question

When writing python code, my typical workflow is to use the interactive prompt and do something like

write function
repeat until working:
  test function
  edit function

Once I'm pretty sure everything's ok, I'll run the code in non-interactive mode and collect the results.

Sometimes the functions run a little too slow and must be optimized.

I'm interested in using cython to optimize these slow functions, but I want to keep my interactive workflow, i.e., run the functions, make changes, run them again.

Is there an easy way to do this?

So far I've tried putting my cython functions in a separate module "my_functions.pyx":

def fun1(int x):
    return x + 130

def fun2(int x):
    return x / 30

Then running (at the interative prompt)

import pyximport; pyximport.install()
import my_functions as mf
mf.fun1(25)

This works the first time, but I want to make changes to my cython functions and reload them in the same interactive session.

running

import my_functions as mf

doesn't update the functions at all. And running

reload(mf)

gives an error: No module named my_functions

The only thing that works is to quit the current session, restart ipython, and import the module all over again. But this sort of kills the benefits of running interactively.

Is there a better way to optimize functions with cython interactively?

If not, can you describe some other ways to approach optimizing code with cython?

Any help is appreciated.

Anvers

The problem is that cython modules are compiled to shared libraries, which cannot be reloaded by the reload function. AFAIK at the moment you simply cannot unload a shared library, hence you'll have to restart the interpreter. Note that you can avoid to restart ipython if you use the Qt console: use Ctrl+. to restart the kernel only, without restarting the whole console. – Fotina

Answer 1

I found a poorly documented feature in the "pyximport.install" function that allows a cython module to be reloaded. With this feature set to True, you can load/reload your cython modules interactively without having to restart python.

If you initialize your cython module with:

import pyximport
pyximport.install(reload_support=True)
import my_functions as mf

You can make changes to your cython module, and then reload with:

reload(mf)

Hopefully this will be of use to someone.

Anvers

Awesome. Anyone know why it's not the default? – Pacemaker

In case it helps someone: for some reason I have to do both reload(mf) and then do import my_functions as mf a second time, in order for it to update. But then it works. – Sit

I am trying to use this to reload new versions of the .SO files. It doesn't seem to be working. In my python prompt, I did import pyximport; pyximport.install(reload_support=True); import my_module as m. If I replace the .SO file with a new one and try reload(m) it's still using the old one. Any idea how to get this working? – Departure

I am having the same issue as @Vinayak. I am guessing the problem is leated to the .SO file being compiled in a way that does not support reload – Republican

@SalvatoreCosentino I didn't get this to work, so I ended up separating/splitting my program into two different parts. An unchanging part which simply imports the .SO file and runs a function within it and the .SO file with the core logic in it. The unchanging script checks for updates and when an updated .SO file is found, it downloads it, replaces the existing .SO file and restarts itself. When no update is found, it imports the .SO file and runs a function within it. This approach worked for me on both Windows and OSX. – Departure

Thank you @Vinayak. In my case I just gave up on the REPL for now and just running a test script that would compile the Cython code when required. – Republican

@Departure - Can you provide code for the solution you mentioned? – Glassworks

@Glassworks You can take a look at my solution on my question about the same thing here: https://mcmap.net/q/538912/-reloading-cython-modules-on-the-fly – Departure