Lifetime of object in lambda connected to pyqtSignal

Question

Suppose I have an object and want one of its methods to be executed when a PyQt signal is emitted. And suppose I want it to do so with a parameter that is not passed by the signal. So I create a lambda as the signal's slot:

class MyClass(object):
    def __init__(self, model):
        model.model_changed_signal.connect(lambda: self.set_x(model.x(), silent=True))

Now, normally with PyQt signals and slots, signal connections don't prevent garbage collection. When a connected slot's object is garbage collected, the slot will no longer be called when the corresponding signal is emitted.

However, how does this work when using lambdas? I don't store a reference to the lambda, yet the signal-slot connection does keep working. So the lambda is not garbage collected.

If I now set the instance of MyClass to None, that instance is not garbage collected either: emitting the model_changed_signal still executes the lambda succesfully. So apparently, a reference to the instance of MyClass is kept around somewhere (maybe in the context of the lambda?) - which I don't want.

Why does this happen?

Humpy

Answer 1

The lambda in your example forms a closure. That is, it is a nested function that references objects available in its enclosing scope. Every function that creates a closure keeps a cell object for every item it needs to maintain a reference to.

In your example, the lambda creates a closure with references to the local self and model variables inside the scope of the __init__ method. If you keep a reference to the lambda somewhere, you can examine all the cell objects of its closure via its __closure__ attribute. In your example, it would display something like this:

>>> print(func.__closure__)
(<cell at 0x7f99c16c5138: MyModel object at 0x7f99bbbf0948>, <cell at 0x7f99c16c5168: MyClass object at 0x7f99bbb81390>)

If you deleted all other references to the MyModel and MyClass objects shown here, the ones kept by the cells would still remain. So when it comes to object cleanup, you should always explicitly disconnect all signals connected to functions that may form closures over the relevant objects.

---

Note that when it comes to signal/slot connections, PyQt treats wrapped C++ slots and Python instance methods differently. The reference counts of these types of callable are not increased when they are connected to signals, whereas lambdas, defined functions, partial objects and static methods are. This means that if all other references to the latter types of callable are deleted, any remaining signal connections will keep them alive. Disconnecting the signals will allow such connected callables to be garbage-collected, if necessary.

The one exception to the above is class methods. PyQt creates a special wrapper when creating connections to these, so if all other references to them are deleted, and the signal is emitted, an exception will be raised, like this:

TypeError: 'managedbuffer' object is not callable

The above should apply to PyQt5 and most versions of PyQt4 (4.3 and greater).

Mana

That confirms and clarifies the keeping of a reference to MyClass and MyModel by the lambda, thanks! My remaining unclarity is why the garbage-collection is different for lambdas than for other function objects. If I would, say, instantiate a class and connect one of its methods as follows: model.model_changed_signal.connect(ModelListener().handle_signal), the instance of ModelListener is garbage collected, and handle_signal will never be called. – Humpy

@tjalling. Garbage colection is exactly the same for lambdas. The difference is entirely due the the closure, and any function can form one of those. It's the closure that keeps the extra references, not the function. There is no closure in the example given in your comment. – Mana

But doesn't there also need to be a reference to the closure? If I throw away my last reference to a lambda (e.g. by setting it to None), the lambda and corresponding closure will be garbage-collected, right? Why does connecting a lambda to a pyqtSignal prevent that lambda (and the closure) from being garbage-collected, while connecting a bound function to a signal doesn't prevent the object (to which the function is bound) from being garbage-collected? – Humpy

@tjalling. Wrong. Connect a global function (with or without a closure) to a signal, then delete the global reference to the function - the signal will still work. The same is not true for instance methods - if the instance is deleted, the signal will be automatically disconnected. If pyqt didn't keep a reference to an unbound function used as a slot, the signal could never work at all, because the connection would be broken as soon as the function object went out of scope. – Mana

@ekhumorao: Exactly. I'd expect the lambda to be immediately garbage-collected and therefore never work at all. And I was wondering whether the reason that it does work is (a) something about lambdas/lifetime in Python that I'm not familiar with, or (b) PyQt treats lambdas/unbound functions differently from bound functions. From your comment, I gather that it's (b). Is that correct? – Humpy

apologies for the misspelling, only noticed it after the editing window had closed. – Humpy

@tjalling. I have updated my answer with the relevant info. – Mana

Great addition! Any idea whether those differences in connection treatment in PyQt are documented anywhere? (Since not updating the reference count is a feature, documenting limitations of / requirements for using that feature would be useful.) – Humpy

@tjalling. It used to be documented for the old-style signals and slots, but the current pyqt5 docs don't mention it. – Mana

Ah, I hadn't checked the old style's documentation. Thanks. – Humpy