How to make a copy of a python module at runtime?

Question

I need to make a copy of a socket module to be able to use it and to have one more socket module monkey-patched and use it differently.

Is this possible?

I mean to really copy a module, namely to get the same result at runtime as if I've copied socketmodule.c, changed the initsocket() function to initmy_socket(), and installed it as my_socket extension.

Tact

Couldn't you simply do: 1) import the base module 2) monkey patch it 3) use the patched module 4) when you are done using the patched module call reload(socket) ? – Lickspittle

I need to use them both at the same time. – Tact

@Dmitry: Why do you think you need to monkey-patch? That's not the best approach in most cases. Can't you just use a wrapper-module for that which you can adapt to your needs? – Hibernate

I'll clear up a bit: I'm writing a tracer for a python code that uses mankey-patched socket. The tracer need to use socket module also, but monkey-patching spoil things. So I need to have a copy of socket in tracer. Also as tracer is C-extension it could use C-implementation of socket from Python, but I don't know how to do that also. – Tact

Fix the code that uses the monkey-patched socket module (because this is clearly broken design). Make it use a monkey-patched wrapper around socket instead. – Hibernate

Unfortunately that is not possible in general case. – Tact

@Dmitry: In what case is it not possible? I can't imagine one. – Hibernate

The code being traced can be any code. For example if it is a code using gevent, it is impossible to fix it not monkey-patch socket, as gevent monkey-patches just works that way. – Tact

Impossible to use a subclass and the base class simultaneusly? – Ezara

You did not even read my answer, did you? :P – 3d

I did, but too late ) Thanks! – Tact

Answer 1

You can always do tricks like importing a module then deleting it from sys.modules or trying to copy a module. However, Python already provides what you want in its Standard Library.

import imp # Standard module to do such things you want to.

# We can import any module including standard ones:
os1=imp.load_module('os1', *imp.find_module('os'))

# Here is another one:
os2=imp.load_module('os2', *imp.find_module('os'))

# This returns True:
id(os1)!=id(os2)

Python3.3+

imp.load_module is deprecated in python3.3+, and recommends the use of importlib

#!/usr/bin/env python3

import sys
import importlib.util

SPEC_OS = importlib.util.find_spec('os')
os1 = importlib.util.module_from_spec(SPEC_OS)
SPEC_OS.loader.exec_module(os1)
sys.modules['os1'] = os1

os2 = importlib.util.module_from_spec(SPEC_OS)
SPEC_OS.loader.exec_module(os2)
sys.modules['os2'] = os2
del SPEC_OS

assert os1 is not os2, \
    "Module `os` instancing failed"

Here, we import the same module twice but as completely different module objects. If you check sys.modules, you can see two names you entered as first parameters to load_module calls. Take a look at the documentation for details.

UPDATE:

To make the main difference of this approach obvious, I want to make this clearer: When you import the same module this way, you will have both versions globally accessible for every other module you import in runtime, which is exactly what the questioner needs as I understood.

Below is another example to emphasize this point.

These two statements do exactly the same thing:

import my_socket_module as socket_imported

socket_imported = imp.load_module('my_socket_module',
    *imp.find_module('my_socket_module')
)

On second line, we repeat 'my_socket_module' string twice and that is how import statement works; but these two strings are, in fact, used for two different reasons.

Second occurrence as we passed it to find_module is used as the file name that will be found on the system. The first occurrence of the string as we passed it to load_module method is used as system-wide identifier of the loaded module.

So, we can use different names for these which means we can make it work exactly like we copied the python source file for the module and loaded it.

socket = imp.load_module('socket_original', *imp.find_module('my_socket_module'))
socket_monkey = imp.load_module('socket_patched',*imp.find_module('my_socket_module'))

def alternative_implementation(blah, blah):
    return 'Happiness'

socket_monkey.original_function = alternative_implementation

import my_sub_module

Then in my_sub_module, I can import 'socket_patched' which does not exist on system! Here we are in my_sub_module.py.

import socket_patched
socket_patched.original_function('foo', 'bar')
# This call brings us 'Happiness'

3d

Awesome! This helped me solve a problem with mock patching module methods for unit testing, while also calling those modules in parallel for real operations. The two occasionally conflicted. With this, I was able to clone the module for testing and avoid the conflict! – Benedictbenedicta

Answer 2

This is pretty disgusting, but this might suffice:

import sys

# if socket was already imported, get rid of it and save a copy
save = sys.modules.pop('socket', None)

# import socket again (it's not in sys.modules, so it will be reimported)
import socket as mysock

if save is None:
    # if we didn't have a saved copy, remove my version of 'socket'
    del sys.modules['socket']
else:
    # if we did have a saved copy overwrite my socket with the original
    sys.modules['socket'] = save

Asthmatic

This has a side effect as it executes the module upon the import instead of a deepcopy it without execution as states in the question. – Cleotildeclepe

Answer 3

Here's some code that creates a new module with the functions and variables of the old:

def copymodule(old):
    new = type(old)(old.__name__, old.__doc__)
    new.__dict__.update(old.__dict__)
    return new

Note that this does a fairly shallow copy of the module. The dictionary is newly created, so basic monkey patching will work, but any mutables in the original module will be shared between the two.

Edit: According to the comment, a deep copy is needed. I tried messing around with monkey-patching the copy module to support deep copies of modules, but that didn't work. Next I tried importing the module twice, but since modules are cached in sys.modules, that gave me the same module twice. Finally, the solution I hit upon was removing the modules from sys.modules after importing it the first time, then importing it again.

from imp import find_module, load_module
from sys import modules

def loadtwice(name, path=None):
    """Import two copies of a module.

    The name and path arguments are as for `find_module` in the `imp` module.
    Note that future imports of the module will return the same object as
    the second of the two returned by this function.
    """
    startingmods = modules.copy()
    foundmod = find_module(name, path)
    mod1 = load_module(name, *foundmod)
    newmods = set(modules) - set(startingmods)
    for m in newmods:
        del modules[m]
    mod2 = load_module(name, *foundmod)
    return mod1, mod2

Falsity

I need full copy of a module with copies of mutables etc – Tact

Edited for a version with a full, deep copy. – Falsity

The problem with shallow copying a module like illustrated by your first snippet is that bindings are unaltered, owing to the same unchanged global scope. Example: if socket defines an x that is bound to X and the copy overrides x using e.g. copymodule(socket).x = Y, an f defined in the module as def f(): return x, will return X, not Y [when called on the copied module]. In summary, such copying does not always suffice, depending on the use case. – Brynnbrynna

Answer 4

Physically copy the socket module to socket_monkey and go from there? I don't feel you need any "clever" work-around... but I might well be over simplifying!

Shana

What do you mean by 'Physically copy'? – Tact

Copy the file using cp or copy (depending on OS) – Shana

I suspected that I would end copy-pasting here. :) That is not very nice, but solves the problem. I think I'll try that way while there is no other.... – Tact

Unfortunaltely that is also complicated as socketmodule.c differs in Python 2 and Python 3. So that is not a solution neither. – Tact

I agree it's not ideal - but the most practical to get stuff done – Shana

@DmitryTrofimov Any reason it's not a solution just because you've suddenly introduced multiple Python versions? There's an amount of compat. problems between 2.x and 3.x. And if you look at the socketmodule.c you'll see that can change even between the same major releases of Python... so I'm really puzzled why you don't think is a workable solution. – Shana

That isn't a workable solution as there are changes between versions of Python - thats very hard to maintain such a code. I'm looking the solution wich allows to use bundled working socketmodule, not copy-pasted one working only for one version of python. As for now I ended with copy paste but still hope to find other solution. – Tact

@Dmitry: copy in runtime, import, monkey-patch, use it, and then delete. Nobody will notice :) – Mccarron