Reloading Cython modules on the fly

Question

I am trying to auto update Cython .so modules that my python program uses on the fly. After I download the new module and del module and import module Python seems to still be importing the older version.

From this question, I've tried this but it didn't work:

from importlib import reload
import pyximport
pyximport.install(reload_support=True)
import module as m
reload(m)

From this question, I've also tried this and it didn't work either:

del sys.modules['module']
del module
import module

I've also tried this with no success:

from importlib import reload
import my_module

my_module = reload(my_module)

Any idea how I can get Cython .SO files imported on the fly?

---

EDIT: Adding code for update check and download

update_filename = "my_module.cpython-37m-darwin.so"

if __name__ == '__main__':
    response = check_for_update()
    if response != "No new version available!":
        print (download_update(response))

def check_for_update():
    print("MD5 hash: {}".format(md5(__file__)))
    s = setup_session()
    data = {
        "hash": md5(__file__),
        "type": "md5",
        "platform": platform.system()
    }
    response = s.post(UPDATE_CHECK_URL, json=data)
    return response.text

def download_update(url):
    s = setup_session()
    with s.get(url, stream=True) as r:
        r.raise_for_status()
        with open(update_filename, 'wb') as f:
            for chunk in r.iter_content(chunk_size=8192): 
                if chunk:
                    f.write(chunk)
    return update_filename

After it has downloaded the new SO file, I manually typed the commands I listed above.

Apanage

I suspect the issue (that you won't beat) is that anything that holds a different reference to your module won't be updated. If you access m it should be the new version, but you'll have lots of references to the old version scattered about. Could you show how you're determining that it isn't reloaded? – Azaleeazan

@Azaleeazan I'm running dir(my_module) before and after the module is reloaded. dir(my_module) shows the same properties that were present in the older module and the properties of the new module isn't shown – Apanage

I'll edit my question to add some more code that shows how it's trying to update the SO file – Apanage

@Azaleeazan BTW, download_update() overwrites the old SO module present in the current directory and replaces it with the new one – Apanage

related, with explanation what is going on: https://mcmap.net/q/541123/-reload-module-with-pyximport In your case, as you are don't build the extension from pyx-file only solution B (or a similar approach) can work. – Nucleonics

I think this may just be that you have to do m = reload(m)? i.e. reload can't change m in place, but can return a new module. – Azaleeazan

@Nucleonics I'll try that out. What worked for me was to have a separate Python script that simply imports the Cython module and I can call that script from my main program using subprocess.call(). It's not the best solution but it works. I'll try out your approach tonight and see if that works – Apanage

@Azaleeazan I've tried that (see my_module = reload(my_module)) but after doing that the 'new' m still hadn't changed from the old m – Apanage

@Apanage Just to confirm, your third test where you do my_module = reload(my_module) is with reload_support=True? – Azaleeazan

@Azaleeazan Yes. I did from importlib import reload; import pyximport; pyximport.install(reload_support=True) first after which I imported the module as import module as m and then m = reload(m) – Apanage

@Nucleonics I didn't fully understand Solution B. Are you saying I'll need to build the module with a different name then import module_suffix as module? – Apanage

After the updated SO file is downloaded, if I quite and relaunch my program with python3 program.py it uses the updated module. I'm okay with this but is there a way to auto close and relaunch my python program in this case? – Apanage

@Apanage I'm afraid I don't know. Thanks for clarifying – Azaleeazan

@Apanage the problem is that once a shared object (*.so) is loaded it cannot be reloaded into the same process - that is just how dlopen works. So you either has to load it from a different path or kill process and start anew (but then it would not be on the fly). – Nucleonics

Answer 1

So I never found the right way to do this but got around the problem by splitting my program into two separate parts:

1. An unchanging 'runner' part which simply imports the .SO file (or .PYD file on Windows) and runs a function within it
2. The actual .SO (or .PYD) file with the core logic in it

The unchanging script checks for updates to the .SO/.PYD file (using it's SHA256 hash + module version) and when an updated version is found, it downloads it and replaces the existing .SO/.PYD file and restarts itself, thus loading the updated module.

When no update is found, it imports the local .SO/.PYD file and runs a function within it. This approach worked for me on both Windows and OSX.

The runner script (run.py)

import requests, os, sys
from pathlib import Path
from shutil import move

original_filename = "my_module.cp38-win32.pyd" # the filename of the Cython module to load
update_filename = f"{original_filename}.update"
UPDATE_SERVER = "https://example.com/PROD/update-check"

def check_for_update():
    replace_current_pyd_with_previously_downloaded_update() # Actually perform the update
    # Add your own update check logic here
    # This one checks with {UPDATE_SERVER} for updates to {original_filename} and returns the direct link to the updated PYD file if an update exists
    s = requests.Session()
    data = {
        "hash": sha256(original_filename),
        "type": "sha256",
        "current_version": get_daemon_version(),
        "platform": platform.system()
    }
    response = s.post(UPDATE_SERVER, json=data)
    return response.text # direct link to newer version of PYD file if update exists

def download_update(url):
    # Download updated PYD file from update server and write/replace {update_filename}

def replace_current_pyd_with_previously_downloaded_update():
    print("Checking for previously downloaded update file")
    update_file_path = Path(update_filename)
    if update_file_path.is_file():
        print(f"Update file found! Performing update by replacing {original_filename} with the updated version and deleting {update_filename}")
        move(update_filename, original_filename)
    else:
        print("No previously downloaded update file found. Checking with update server for new versions")

def get_daemon_version():
    from my_module import get_version
    return get_version() # my_module.__version__.lower().strip()

def restart():
    print ("Restarting to apply update...\r\n")
    python = sys.executable
    os.execl(python, python, *sys.argv)

def apply_update():
    restart()

def start_daemon():
    import my_module
    my_module.initiate()
    my_module.start()

if __name__ == "__main__":
    response = None
    print ("Checking to see if an update is available...")
    try:
        response = check_for_update()
    except Exception as ex:
        print ("Unable to check for updates")
        pass
    if response is None:
        print ("Unable to check for software updates. Using locally available version.")
        start_daemon()
    elif response != "No new version available!" and response != '':
        print ("Newer version available. Updating...")
        print ("Update downloaded: {}".format(download_update(response)))
        apply_update()
        start_daemon()
    else:
        print ("Response from update check API: {}\r\n".format(response))
        start_daemon()

The .SO/.PYD file

The actual .SO file (in this case, .PYD file) should contain a method called get_version which should return the version of the module and your update server should contain logic to determine if an update is available for the combination of (SHA256 + module_version).

You could of course implement the update check in a totally different way altogether.

Apanage