Python dynamic function creation with custom names

Question

Apologies if this question has already been raised and answered. What I need to do is very simple in concept, but unfortunately I have not been able to find an answer for it online.

I need to create dynamic functions in Python (Python2.7) with custom names at runtime. The body of each function also needs to be constructed at runtime but it is (almost) the same for all functions.

I start off with a list of names.

func_names = ["func1", "func2", "func3"]

Note that the func_name list can hold a list of arbitrary names, so the names will NOT simply be func1, func2, func3, ....

I want the outcome to be :

    def func1(*args):
        ...

    def func2(*args):
        ...

    def func3(*args):
        ...

The reason I need to do this is that each function name corresponds to a test case which is then called from the outside world.

update: There is no user input. I'm tying two ends of a much bigger module. One end determines what the test cases are and among other things, populates a list of the test cases' names. The other end is the functions themselves, which must have 1:1 mapping with the name of the test case. So I have the name of the test cases, I know what I want to do with each test case, I just need to create the functions that have the name of the test cases. Since the name of the test cases are determined at runtime, the function creation based on those test cases must be at runtime as well.

update: I can also wrap this custom named functions in a class if that would make things easier.

I can hard-code the content of the functions (since they are almost the same) in a string, or I can base it off of a base class previously defined. Just need to know how to populate the functions with this content.

For example:

    func_content = """
                   for arg in args:
                       print arg
                   """

Thanks in advance,

Mahdi

Grope

Answer 1

For what you describe, I don't think you need to descend into eval or macros — creating function instances by closure should work just fine. Example:

def bindFunction1(name):
    def func1(*args):
        for arg in args:
            print arg
        return 42 # ...
    func1.__name__ = name
    return func1

def bindFunction2(name):
    def func2(*args):
        for arg in args:
            print arg
        return 2142 # ...
    func2.__name__ = name
    return func2

However, you will likely want to add those functions by name to some scope so that you can access them by name.

>>> print bindFunction1('neat')
<function neat at 0x00000000629099E8>
>>> print bindFunction2('keen')
<function keen at 0x0000000072C93DD8>

Stover

This solution is somewhat hacky. Doing this would not overwrite other outputs, where function name can be found. For example __qualname__ will still carry original name. – Pessimist

__qualname__ was introduced in Python 3.3 and should be also edited, if it is ever used in addition to __name__. They are basically the same thing but constructed differently. https://mcmap.net/q/45789/-what-is-__qualname__-in-python/3839249 – Rimmer

Answer 2

Extending on Shane's answer since I just found this question when looking for a solution to a similar problem. Take care with the scope of the variables. You can avoid scope problems by using a generator function to define the scope. Here is an example that defines methods on a class:

class A(object):
    pass

def make_method(name):
    def _method(self):
        print("method {0} in {1}".format(name, self))
    return _method

for name in ('one', 'two', 'three'):
    _method = make_method(name)
    setattr(A, name, _method)

In use:

In [4]: o = A()

In [5]: o.one()
method one in <__main__.A object at 0x1c0ac90>

In [6]: o1 = A()

In [7]: o1.one()
method one in <__main__.A object at 0x1c0ad10>

In [8]: o.two()
method two in <__main__.A object at 0x1c0ac90>

In [9]: o1.two()
method two in <__main__.A object at 0x1c0ad10>

Ferrara

How to add this inside class __init__? Not outside of class. – Dandy

@AlexeiMarinichenko Adding it to init__would be making it instance specific and is not a good idea. If you need instance specific behaviours I'd look towards specializing __getattr or using some distinct protocol defined in the class. Avoid esoteric stuff like this unless you really need it and if you really need it take extra care to make it as clear as possible or it will come back and bite you. – Ferrara

@AlexeiMarinichenko That said, it is 'untidy' adding the methods after the class specification and if that is what concerns you I'd just keep the class to its own module. If doing this for more than one class in your application I'd look to using a metaclass to specialize the class instantiation so that you can move the dynamic method specification to a class variable. – Ferrara

Answer 3

There probably is a sort of introspection to do this kind of thing, but I don't think it would be the pythonic approach to the problem.

I think you should take advantage of the nature of functions in python as first level citizens. Use closures as Shane Holloway pointed, to customize the function contents as you like. Then for the dynamic name binding, use a dictionary whose keys are the names defined dynamically, and the values will be the functions itself.

def function_builder(args):
    def function(more_args):
       #do stuff based on the values of args
    return function

my_dynamic_functions = {}
my_dynamic_functions[dynamic_name] = function_builder(some_dynamic_args)

#then use it somewhere else
my_dynamic_functions[dynamic_name](the_args)

Hope it makes sense to your use case.

Kowtko

Answer 4

To truly create functions dynamically, you can use makefun, I have developed it just for that. It supports three ways to define the signature to generate:

• a string representation, for example 'foo(a, b=1)'
• a Signature object, either handcrafted or derived by using inspect.signature on another function
• a reference function. In that case the exposed signature will be identical to this function's signature.

Moreover you can tell it to inject the created function's reference as the first argument in your implementation, so as to have minor behavior modifications depending on where the call comes from (which facade). For example:

# generic core implementation
def generic_impl(f, *args, **kwargs):
    print("This is generic impl called by %s" % f.__name__)
    # here you could use f.__name__ in a if statement to determine what to do
    if f.__name__ == "func1":
        print("called from func1 !")
    return args, kwargs

my_module = getmodule(generic_impl)

# generate 3 facade functions with various signatures
for f_name, f_params in [("func1", "b, *, a"),
                         ("func2", "*args, **kwargs"),
                         ("func3", "c, *, a, d=None")]:
    # the signature to generate
    f_sig = "%s(%s)" % (f_name, f_params)

    # create the function dynamically
    f = create_function(f_sig, generic_impl, inject_as_first_arg=True)

    # assign the symbol somewhere (local context, module...)
    setattr(my_module, f_name, f)

# grab each function and use it
func1 = getattr(my_module, 'func1')
assert func1(25, a=12) == ((), dict(b=25, a=12))

func2 = getattr(my_module, 'func2')
assert func2(25, a=12) == ((25,), dict(a=12))

func3 = getattr(my_module, 'func3')
assert func3(25, a=12) == ((), dict(c=25, a=12, d=None))

As you can see in the documentation, arguments are always redirected to the kwargs except when it is absolutely not possible (var-positional signatures such as in func2).

Sexennial

Answer 5

You may want to use eval; you'll build the string containing the Python definition of each function (i.e. a multiline string starting with def func1 ....) and you'll then eval it.

But I would generate a unique name for each such function (e.g. genfun345). Don't use some unchecked user input for such names. Because if the name is the same as some known name in Python, you are going into a difficult to debug disaster.

Do you trust the inputs from which these functions are made? Do you care about malware or abuse?

Read about hygenic macros on wikipedia.

Empiric

There is no user input. I'm tying two ends of a much bigger module. One end determines what the test cases are and among other things, populates a list of the test cases' names. The other end is the functions themselves, which must have 1:1 mapping with the name of the test case. So I have the name of the test cases, I know what I want to do with each test case, I just need to create the functions that have the name of the test cases. Since the name of the test cases are determined at runtime, the function creation based on those test cases must be at runtime as well. – Grope

Answer 6

If I'm understanding your requirements correctly, it sounds like you just want to dynamically assign existing functions new or alternative names -- in which case something along the following lines ought to do the job:

import sys

testcases = []
def testcase(f):
    """ testcase function decorator """
    testcases.append(f)
    return f

@testcase
def testcase0(*args):
    print 'testcase0 called, args:', args

@testcase
def testcase1(*args):
    print 'testcase1 called, args:', args

@testcase
def testcase2(*args):
    print 'testcase2 called, args:', args

def assign_function_names(func_names, namespace=None):
    if namespace is None:
        namespace = sys._getframe(1).f_globals  # default to caller's globals
    for name, func in zip(func_names, testcases):
        func.__name__ = name  # optional
        namespace[name] = func

assign_function_names(["funcA", "funcB", "funcC"])

funcA(1, 2, 3)
funcB(4, 5)
funcC(42)

Kimberleykimberli

Thanks for your response. But That is not the case. They are not existing functions, and furthermore, the number of functions that are to be dynamically created is also unknown (only known at runtime). – Grope

If you can "hard-code the content of the functions" you might as well precede that content with a def xxx(yyy): in a .py file and make it an existent function -- what do you think you're gaining by putting it in string and dynamically creating a function out of it? – Kimberleykimberli

I think you've misunderstood what I'm trying to do here and did not really answer my question but keep telling to do something else. Thanks nonetheless. I've already solved my problem. – Grope

@mahdiolfat: Apparently I'm not the only one who hasn't understood. You didn't answer my question about why you need to dynamically create the functions if you know their content in advance. Dynamically changing their name and making them methods is a more understandable need and relatively easily done. – Kimberleykimberli

My question was how to create dynamic functions. It wasn't on why one would need to do one. If you know how to do it, let me know. If not, don't waste your time. – Grope

@mahdiolfat: Well, from what you have written, it's not clear you need to do it, doesn't provide much motivation to help you perfect or improve something seemingly unnecessary -- so explaining more about why you think you need to do so might be worthwhile. – Kimberleykimberli

Answer 7

Since nobody really gave a proper answer, I'll add mine (works in Python 3).

dynamic_func.py

from copy import deepcopy

__all__ = []  # For safer * imports
FUNC_NAMES = ["func1", "func2", "func3"]


def generate_func(func_name):
    def __dynamic_func():
        return func_name  # This is for testing
    return __dynamic_func

# Loop through, construct function body and add 
# it to the global scope of the module.
for __func_name in FUNC_NAMES:
    __func = generate_func(__func_name)
    globals()[__func_name] = deepcopy(__func)
    __all__.append(__func_name)


# Clean up
del __func_name
del __func
__all__ = tuple(__all__)

What comes below is a test.

test_dynamic_func.py

from unittest import TestCase
from dynamic_func import func1, func2, func3


class DynamicFuncTestCase(TestCase):
    def test_dynamic_func(self):
        self.assertEqual(func1(), "func1")
        self.assertEqual(func2(), "func2")
        self.assertEqual(func3(), "func3")

Refurbish