Hot-swapping of Python running program

Question

The following code allows you to modify the contents of runtime.py at run time. In other words, you don't have to interrupt runner.py.

#runner.py
import time
import imp

def main():
    while True:
        mod = imp.load_source("runtime", "./runtime.py")
        mod.function()
        time.sleep(1)

if __name__ == "__main__":
    main()

The module imported at runtime is:

# runtime.py
def function():
    print("I am version one of runtime.py")

This primitive mechanism allows you to "how-swap" Python code (a la Erlang). Is there a better alternative?

Please notice that this is a merely academic question, as I don't have the necessity to do anything like this. However, I am interested in learning more about the Python runtime.

Edit:

I created the following solution: an Engine object provides an interface to the functions contained in a module (in this case the module is called engine.py). The Engine object also spawns a thread which monitors for changes in the source file and, if changes are detected, it calls the notify() method on the engine, which reloads the source file.

In my implementation, the change detection is based on polling every frequency seconds checking the SHA1 checksum of the file, but other implementations are possible.

In this example every change detected is logged to a file called hotswap.log, where the checksum is registered.

Other mechanisms for detecting changes could be a server or the use of inotify in the Monitor thread.

import imp
import time
import hashlib
import threading
import logging

logger = logging.getLogger("")

class MonitorThread(threading.Thread):
    def __init__(self, engine, frequency=1):
        super(MonitorThread, self).__init__()
        self.engine = engine
        self.frequency = frequency
        # daemonize the thread so that it ends with the master program
        self.daemon = True 

    def run(self):
        while True:
            with open(self.engine.source, "rb") as fp:
                fingerprint = hashlib.sha1(fp.read()).hexdigest()
            if not fingerprint == self.engine.fingerprint:
                self.engine.notify(fingerprint)
            time.sleep(self.frequency)

class Engine(object):
    def __init__(self, source):
        # store the path to the engine source
        self.source = source        
        # load the module for the first time and create a fingerprint
        # for the file
        self.mod = imp.load_source("source", self.source)
        with open(self.source, "rb") as fp:
            self.fingerprint = hashlib.sha1(fp.read()).hexdigest()
        # turn on monitoring thread
        monitor = MonitorThread(self)
        monitor.start()

    def notify(self, fingerprint):
        logger.info("received notification of fingerprint change ({0})".\
                        format(fingerprint))
        self.fingerprint = fingerprint
        self.mod = imp.load_source("source", self.source)

    def __getattr__(self, attr):
        return getattr(self.mod, attr)

def main():
    logging.basicConfig(level=logging.INFO, 
                        filename="hotswap.log")
    engine = Engine("engine.py")
    # this silly loop is a sample of how the program can be running in
    # one thread and the monitoring is performed in another.
    while True:
        engine.f1()
        engine.f2()
        time.sleep(1)

if __name__ == "__main__":
    main()

The engine.py file:

# this is "engine.py"
def f1():
    print("call to f1")

def f2():
    print("call to f2")

Log sample:

INFO:root:received notification of fingerprint change (be1c56097992e2a414e94c98cd6a88d162c96956)
INFO:root:received notification of fingerprint change (dcb434869aa94897529d365803bf2b48be665897)
INFO:root:received notification of fingerprint change (36a0a4b20ee9ca6901842a30aab5eb52796649bd)
INFO:root:received notification of fingerprint change (2e96b05bbb8dbe8716c4dd37b74e9f58c6a925f2)
INFO:root:received notification of fingerprint change (baac96c2d37f169536c8c20fe5935c197425ed40)
INFO:root:received notification of fingerprint change (be1c56097992e2a414e94c98cd6a88d162c96956)
INFO:root:received notification of fingerprint change (dcb434869aa94897529d365803bf2b48be665897)

Again - this is an academic discussion because I have no need at this moment of hot-swapping Python code. However, I like being able to understand a little bit the runtime and realize what is possible and what is not. Notice that the loading mechanism could add a lock, in case it is using resources, and exception handling, in case the module is not loaded successfully.

Comments?

Kostroma

Don't edit an answer into your quesiton. Add it as an answer. – Protrusive

Possible duplicate of How do I unload (reload) a Python module? – Prichard

Answer 1

You could poll the runtime.py file, waiting for it to change. Once it changes, just call

reload(runtime)

Any time I'm debugging a python module, I use this approach in the interactive python command prompt (except I manually call reload(), I don't poll anything).

EDIT: To detect changes in a file, check out this SO question. Polling may be the most reliable option, but I would only reload the file if the modified time is updated, rather than reloading it on every poll. You should also consider catching exceptions when you reload, especially syntax errors. And you may or may not encounter problems with thread safety.

Forewent

In fact, the function reload and imp.load_source are equivalent. I guess I am more interested in the "polling" or "listener" mechanism that brokers the reloading mechanism. Thanks! – Kostroma

Note that for Python 3.4+ you need to use "from importlib import reload" while for earlier versions "from imp import reload" – Grantley

Answer 2

globe = __import__('copy').copy(globals())
while True:
    with open('runtime.py', 'r') as mod:
        exec mod in globe
    __import__('time').sleep(1)

Will repeatedly read and run runtime.py with a nearly unpolluted globals() and no locals(), and won't pollute the global scope, but all of runtime's namespace will be available in globe

Protrusive

This is interesting, but wouldn't I loose the namespacing? – Kostroma

Yes, you would. I assumed runner.py is just a helper to repeatedly reload and re-execute code from runtime.py so you didn't care. You'd need to make a copy of globals() instead, editing to show that. – Protrusive

Answer 3

If you want hot-swap code that found as when you use import out of functions etc, you need overwrite a global var of module, if for example you use:

import mylib

You need when load module in code asign to mylib the new module. Other cuestion is try this in program that use threads for know if is secure with threads and, when use multiprocessing this only found in one process, for change code in all process all need load new code, is necesary try if is secure in multiproces.

And, is interessting check first if have new code or not for not load same code. And think in Python only you can load a new module and replace variable name of module, but if you really need a good hot change code see Erlang language and OTP, it's very good.

Dissolve

The suggest of use Erlang/OTP I think is the best solution when need hot-swap of code good, fault-tolerant software, and concurrent or distribute execution. Whatsapp run the servers in Erlang, Facebook same for the Facebook chat, Tuenti too, Call of Duty servers, etc. – Prichard