Is it possible to call a function from within a list comprehension without the overhead of calling the function?

Question

In this trivial example, I want to factor out the i < 5 condition of a list comprehension into it's own function. I also want to eat my cake and have it too, and avoid the overhead of the CALL_FUNCTION bytecode/creating a new frame in the python virtual machine.

Is there any way to factor out the conditions inside of a list comprehension into a new function but somehow get a disassembled result that avoids the large overhead of CALL_FUNCTION?

import dis
import sys
import timeit

def my_filter(n):
    return n < 5

def a():
    # list comprehension with function call
    return [i for i in range(10) if my_filter(i)]

def b():
    # list comprehension without function call
    return [i for i in range(10) if i < 5]

assert a() == b()

>>> sys.version_info[:]
(3, 6, 5, 'final', 0)

>>> timeit.timeit(a)
1.2616060493517098
>>> timeit.timeit(b)
0.685117881097812

>>> dis.dis(a)
  3           0 LOAD_CONST               1 (<code object <listcomp> at 0x0000020F4890B660, file "<stdin>", line 3>)
  # ...

>>> dis.dis(b)
  3           0 LOAD_CONST               1 (<code object <listcomp> at 0x0000020F48A42270, file "<stdin>", line 3>)
  # ...

# list comprehension with function call
# big overhead with that CALL_FUNCTION at address 12
>>> dis.dis(a.__code__.co_consts[1])
3         0 BUILD_LIST               0
          2 LOAD_FAST                0 (.0)
    >>    4 FOR_ITER                16 (to 22)
          6 STORE_FAST               1 (i)
          8 LOAD_GLOBAL              0 (my_filter)
         10 LOAD_FAST                1 (i)
         12 CALL_FUNCTION            1
         14 POP_JUMP_IF_FALSE        4
         16 LOAD_FAST                1 (i)
         18 LIST_APPEND              2
         20 JUMP_ABSOLUTE            4
    >>   22 RETURN_VALUE

# list comprehension without function call
>>> dis.dis(b.__code__.co_consts[1])
3         0 BUILD_LIST               0
          2 LOAD_FAST                0 (.0)
    >>    4 FOR_ITER                16 (to 22)
          6 STORE_FAST               1 (i)
          8 LOAD_FAST                1 (i)
         10 LOAD_CONST               0 (5)
         12 COMPARE_OP               0 (<)
         14 POP_JUMP_IF_FALSE        4
         16 LOAD_FAST                1 (i)
         18 LIST_APPEND              2
         20 JUMP_ABSOLUTE            4
    >>   22 RETURN_VALUE

I'm willing to take a hacky solution that I would never use in production, like somehow replacing the bytecode at run time.

In other words, is it possible to replace a's addresses 8, 10, and 12 with b's 8, 10, and 12 at runtime?

Noleta

I'm willing to take a hacky solution that I would never use in production - is it worth considering? – Bouncing

@Bouncing for educational purposes - sure. – Noleta

Feels like your time would be better spent learning something like Cython, to be honest. – Implicatory

I imagine your best option is to generate the code for the listcomp at run-time and then eval it. – Eulogy

It sounds to me like you're looking for a way to inline a function... there is no such thing in CPython, although attempts have been made: bugs.python.org/issue10399 – Successive

Answer 1

Consolidating all of the excellent answers in the comments into one.

As georg says, this sounds like you are looking for a way to inline a function or an expression, and there is no such thing in CPython attempts have been made: https://bugs.python.org/issue10399

Therefore, along the lines of "metaprogramming", you can build the lambda's inline and eval:

from typing import Callable
import dis

def b():
    # list comprehension without function call
    return [i for i in range(10) if i < 5]

def gen_list_comprehension(expr: str) -> Callable:
    return eval(f"lambda: [i for i in range(10) if {expr}]")

a = gen_list_comprehension("i < 5")
dis.dis(a.__code__.co_consts[1])
print("=" * 10)
dis.dis(b.__code__.co_consts[1])

which when run under 3.7.6 gives:

 6           0 BUILD_LIST               0
              2 LOAD_FAST                0 (.0)
        >>    4 FOR_ITER                16 (to 22)
              6 STORE_FAST               1 (i)
              8 LOAD_FAST                1 (i)
             10 LOAD_CONST               0 (5)
             12 COMPARE_OP               0 (<)
             14 POP_JUMP_IF_FALSE        4
             16 LOAD_FAST                1 (i)
             18 LIST_APPEND              2
             20 JUMP_ABSOLUTE            4
        >>   22 RETURN_VALUE
==========
  1           0 BUILD_LIST               0
              2 LOAD_FAST                0 (.0)
        >>    4 FOR_ITER                16 (to 22)
              6 STORE_FAST               1 (i)
              8 LOAD_FAST                1 (i)
             10 LOAD_CONST               0 (5)
             12 COMPARE_OP               0 (<)
             14 POP_JUMP_IF_FALSE        4
             16 LOAD_FAST                1 (i)
             18 LIST_APPEND              2
             20 JUMP_ABSOLUTE            4
        >>   22 RETURN_VALUE

From a security standpoint "eval" is dangerous, athough here it is less so because what you can do inside a lambda. And what can be done in an IfExp expression is even more limited, but still dangerous like call a function that does evil things.

However, if you want the same effect that is more secure, instead of working with strings you can modify AST's. I find that a lot more cumbersome though.

A hybrid approach would be the call ast.parse() and check the result. For example:

import ast
def is_cond_str(s: str) -> bool:
    try:
        mod_ast = ast.parse(s)
        expr_ast = isinstance(mod_ast.body[0])
        if not isinstance(expr_ast, ast.Expr):
            return False
        compare_ast = expr_ast.value
        if not isinstance(compare_ast, ast.Compare):
            return False
        return True
    except:
        return False

This is a little more secure, but there still may be evil functions in the condition so you could keep going. Again, I find this a little tedious.

Coming from the other direction of starting off with bytecode, there is my cross-version assembler; see https://pypi.org/project/xasm/

Fitz