I don't understand this python __del__ behaviour

Question

Can someone explain why the following code behaves the way it does:

import types

class Dummy():
    def __init__(self, name):
        self.name = name
    def __del__(self):
        print "delete",self.name

d1 = Dummy("d1")
del d1
d1 = None
print "after d1"

d2 = Dummy("d2")
def func(self):
    print "func called"
d2.func = types.MethodType(func, d2)
d2.func()
del d2
d2 = None
print "after d2"

d3 = Dummy("d3")
def func(self):
    print "func called"
d3.func = types.MethodType(func, d3)
d3.func()
d3.func = None
del d3
d3 = None
print "after d3"

The output (note that the destructor for d2 is never called) is this (python 2.7)

delete d1
after d1
func called
after d2
func called
delete d3
after d3

Is there a way to "fix" the code so the destructor is called without deleting the method added? I mean, the best place to put the d2.func = None would be in the destructor!

Thanks

[edit] Based on the first few answers, I'd like to clarify that I'm not asking about the merits (or lack thereof) of using __del__. I tried to create the shortest function that would demonstrate what I consider to be non-intuitive behavior. I'm assuming a circular reference has been created, but I'm not sure why. If possible, I'd like to know how to avoid the circular reference....

Zoophilia

Your code for __del__ doesn't actually delete anything. Regardless, using __del__ is not safe. – Astromancy

Yeah, types.MethodType(func, d2) has a reference to d2 and you put that on d2, so you have a circular reference. Nothing surprising about it, but why do you do that if that's not what you want? – Baler

@Jochen Ritzel - I DO want a dynamically bound method. I've created a class that adds methods (getattr) as needed. This is a long running class, so using with isn't a good option, so __ del__ seems like a great place to clean up. I was hoping there was a simple way to remove the circular dependency. The code above is simplified to show the issue. I think I'm inclined to go with an atexit solution. – Zoophilia

Here's the link to a similar question and an atexit answer: link – Zoophilia

"Is there a way to "fix" the code so the destructor is called without deleting the method added?" I think we're all confused by this statement. If your deleting an instance of a specific object, that you've added a specific method to, why do you need that method to continue to exist? If you want it to be a part of the Dummy object then you can add the method to it: Dummy.func = types.MethodType(func, Dummy). This will then add the method to all instances of Dummy. (Which may not be the desired behavior, but that's what you'd get) – Virge

@Virge - sorry for the confusion. d2/d3 show that the destructor is called "as expected" only if the added function is first removed from the instance. My goal is to have the class destructor called when the object goes out of scope, without my having to call any cleanup function manually. d3 isn't feasible because it requires a cleanup function before __ del__ runs, which doesn't make sense. I'm wondering if there is some sort of fix (like weakref?) that will make d2's destructor get called. Make more sense? – Zoophilia

@Brett: You're doing something weird (adding functions to instances) and it's causing problems, so you're looking for more weird things to fix that. It would be much more reasonable to take a step back and ask for a way to solve your original problem in a way that doesn't cause more problems. You could fix this problem with descriptors, but I'm hesitant to explain how - it is kind of complicated and the only thing worse than a complicated solution is a complicated solution to a avoidable problem. – Baler

"You could fix this problem with descriptors, but I'm hesitant to explain how". Didn't expect to see that type of comment on Stackoverflow. I guess I will have to look up descriptors myself. – Zoophilia

@JochenRitzel Descriptors won't help here, since all the descriptor would do is instantiate the MethodType object behind the scenes. This would still create a circular reference. – Sidwell

Answer 1

I'm providing my own answer because, while I appreciate the advice to avoid __del__, my question was how to get it to work properly for the code sample provided.

Short version: The following code uses weakref to avoid the circular reference. I thought I'd tried this before posting the question, but I guess I must have done something wrong.

import types, weakref

class Dummy():
    def __init__(self, name):
        self.name = name
    def __del__(self):
        print "delete",self.name

d2 = Dummy("d2")
def func(self):
    print "func called"
d2.func = types.MethodType(func, weakref.ref(d2)) #This works
#d2.func = func.__get__(weakref.ref(d2), Dummy) #This works too
d2.func()
del d2
d2 = None
print "after d2"

Longer version: When I posted the question, I did search for similar questions. I know you can use with instead, and that the prevailing sentiment is that __del__ is BAD.

Using with makes sense, but only in certain situations. Opening a file, reading it, and closing it is a good example where with is a perfectly good solution. You've gone a specific block of code where the object is needed, and you want to clean up the object and the end of the block.

A database connection seems to be used often as an example that doesn't work well using with, since you usually need to leave the section of code that creates the connection and have the connection closed in a more event-driven (rather than sequential) timeframe.

If with is not the right solution, I see two alternatives:

1. You make sure __del__ works (see this blog for a better description of weakref usage)
2. You use the atexit module to run a callback when your program closes. See this topic for example.

While I tried to provide simplified code, my real problem is more event-driven, so with is not an appropriate solution (with is fine for the simplified code). I also wanted to avoid atexit, as my program can be long-running, and I want to be able to perform the cleanup as soon as possible.

So, in this specific case, I find it to be the best solution to use weakref and prevent circular references that would prevent __del__ from working.

This may be an exception to the rule, but there are use-cases where using weakref and __del__ is the right implementation, IMHO.

Zoophilia

It's fantastically important to remember that there is just no guarantee on whether __del__ will be called, when it gets called, or how many times it gets called. There are all sorts of ways which, out of your control, garbage collection happens who knows when. In particular, cPython is the only python that does reference counting and eagerly collects such garbage. Jython, PyPy (and IronPython, I think) collect garbage only during sweeps. Depending on __del__ is the surest path to disaster. – Concordance

Just to follow up on TokenMacGuy's comment - the null garbage collector is a perfectly valid implementation if you never run out of memory, and it will result in __del__ never being called. GC's are not aware of non-memory resources, and shouldn't be relied upon to manage them. – Elsaelsbeth

@Nick - GC's are not aware of non-memory resources? Of course they are. They are aware of anything they count references from. If you think of the reference as memory (in which case every object is memory and works with the GC) your statement is meaningless. If you don't count the reference as memory, your statement is incorrect. – Zoophilia

@BrettStottlemyer: No, the Python GC is not specifically aware of any non-memory resource - the only thing it knows how to do is find loose objects and delete them (from memory). Because Python doesn't have destructors that the GC will trigger, this means the GC has no side effects, and won't close database connections, etc. Yes, yes, of course you could write a database connection GC, but the fact is that Python doesn't have one (in any implementation), and the Python GC implementations merely clean up the only resource they know about - memory. – Elsaelsbeth

weakref is the right solution to the problem you presented; I'm curious, though, why you don't add the method to the class instead of the instance? That would avoid the problem all together. – Cowardly

@EthanFurman - Because, in my case, adding the functions is dynamic (at runtime) and not known in advance. – Zoophilia

@BrettStottlemyer: Sorry, I phrased that poorly. It is possible to add to the class just as easily as to the instance; the question is whether all instances of that class need the additional methods. Also, if you assign to the class instead of the instance you don't have to mess with MethodType or circular references -- it's just Dummy.func = func, or d2.__class__.func = func. – Cowardly

"It's fantastically important to remember that there is just no guarantee on whether del will be called". There are strict, explicit conditions on whether del will be called - please see this reference. – Appose

Explicit is better than implicit. If you need to open a connection in one event handler and close it in another, call close() explicitly! If not, wrap your whole event loop in a with statement. Use contextlib.closing if need be. If you don't know where to close your connections, then you've a much bigger problem. Instead of hacking at the language so your sloppy magic code will work, figure out where to put those close() statements. You'll be glad you did. – Sidwell

-1. You have to know how to ask a question, or at least how to take an answer. See this meta question and this one on the XY Problem. You asked "Why doesn't __del__ seem to work," which is a legitimate question about python internals, to which you got a legitimate answer. Your real problem was "How do I close a resource in one event handler that I've opened in another," the answer to which has nothing to do with __del__. – Sidwell

@Aryeh - I disagree. My question was how to get a feature of python to work, and I did provide a correct answer. If the real answer is to not use __ del__, then __ del__ should be removed from python. I believe that the upvotes give people an idea of what is commonly accepted, which my answer isn't. That's fine. But how can you say my answer is wrong, when it DOES answer the question (as asked, not as you incorrectly restated it)? – Zoophilia

Regarding a database connection as an example that doesn't work well using with, I strongly disagree. It is generally very simple to acquire a database at some high scope of your program, pass it as an argument into other portions of code, and then release it once they return. It might be executing a lot of code between passing the argument and returning, but it's still a very clearly defined scope. Favor those kinds of code over using globals for database connections, to solve exactly this sort of problem. – Metritis

@BrettStottlemyer I'm very grateful that you answered your own question, while others complained about del without a helpful explanation. I found your solution to be very illuminating about del's functionality. I upvoted both your question and answer without hesitation. – Courson

It might be helpful to have specific details on the precise problem your example demonstrates. I found it here: eli.thegreenplace.net/2009/06/12/… It refers to this documentation: docs.python.org/2/reference/datamodel.html Which states: "Circular references which are garbage are detected when the option cycle detector is enabled (it’s on by default), but can only be cleaned up if there are no Python-level __del__() methods involved" – Daffodil

@AinzTitor If so, then you upvoted a misguided answer that taught you the wrong way to write the code. – Bechtel

Answer 2

You cannot assume that __del__ will ever be called - it is not a place to hope that resources are automagically deallocated. If you want to make sure that a (non-memory) resource is released, you should make a release() or similar method and then call that explicitly (or use it in a context manager as pointed out by Thanatos in comments below).

At the very least you should read the __del__ documentation very closely, and then you should probably not try to use __del__. (Also refer to the gc.garbage documentation for other bad things about __del__)

Elsaelsbeth

Exactly right. __del__ is usually a bad idea. Being explicit is always a good idea. – Astromancy

Additionally, the with statement can be used in conjunction with a release()-like function to write simpler, cleaner code. – Sisely

If "being explicit" is always a good idea, why have garbage collection at all? "Remembering to call" a release function is fragile. You might forget to call it - or an exception might be raised at an unexpected point and you might never reach the release function. This article explains how you can guarantee that your del function is called. If possible, a context manager is even better, but much of the time it isn't possible, as your object's lifespan transcends one method or function. – Appose

@TomSwirly: You may be able to guarantee your object will qualify for garbage collection, but you cannot guarantee when it will be collected. If the garbage collector is free to collect an object, it is also free to not collect it. This blog post from Raymond Chen is always worth remembering. Assuming too much about a given GC implementation might take today's working code and mysteriously break it tomorrow. – Elsaelsbeth

@TomSwirly If you can't identify a single scope in which a particular resource is held and then released, then you should reconsider how your code is organized. Creating a single scope where you can acquire and release a resource is the best plan, even when it requires significant refactoring. You can then pass your resource object into a function call or what have you and keep passing it down the stack as needed. Often, you can find a way to hold a resource for only a short section of the code (generate content before opening a file for writing, close connection as soon as data is read). – Metritis

@jpmc26: That is certainly good advice, although what I say above still applies - qualifying for garbage collection is not the same thing as guaranteeing that you will be collected. If your resource isn't memory (the only thing the garbage collector is designed to do reasonable things with), you need to take a more proactive role (like using a context manager, which is within the scope (no pun intended.. :-)) of what you are suggesting). – Elsaelsbeth

@NickBastin My apologies for not being clear enough. I was trying to say that when you find yourself "unable" to use a context manager, reorganize your code so you can use one. =) I thought my mentions of "acquire" and "release" would be understood as the functions a context manager typically performs. I agree with you 110%. – Metritis

In my opinion, since Python has context managers and __del is essentially useless it should have been removed in Python 3. – Slowmoving

How do you use a context manager that returns something? Unlike a unique-ptr in C++, the CM "finally" part runs on the return, not when the object is deleted as it would in the RAII model. For example, consider a function that creates a temporary directory, and you want it to be deleted unconditionally once "done" - you'd have to put a CM at the level where you call the function, but then if you have a function that calls it... it has to be CMs all the way down? – Dingdong

Answer 3

Instead of __del__, you can use the with operator.

https://web.archive.org/web/20201111211102/http://effbot.org/zone/python-with-statement.htm

just like with filetype objects, you could do something like

with Dummy('d1') as d:
    #stuff
# d's __exit__ method is guaranteed to have been called

Wolffish

d is not out of scope. the only guarantee is that the __exit__ method was called on object created by Dummy('d1') – Concordance

@TokenMacGuy: ok, you're right. It's not technically out of scope, I guess that's worth mentioning. However, to use the syntax, the object MUST implement then __enter__ and __exit__ methods. Hence, for all intents and purposes, it goes out of scope after the with statement, as one would presumably write their __exit__ method as if it were a destructor. I didn't, however, realize that it didn't ACTUALLY go out of scope, which I imagine can be useful for all sorts of hacking – Wolffish

Answer 4

del doesn't call __del__

del in the way you are using removes a local variable. __del__ is called when the object is destroyed. Python as a language makes no guarantees as to when it will destroy an object.

CPython as the most common implementation of Python, uses reference counting. As a result del will often work as you expect. However it will not work in the case that you have a reference cycle.

d3 -> d3.func -> d3

Python doesn't detect this and so won't clean it up right away. And its not just reference cycles. If an exception is throw you probably want to still call your destructor. However, Python will typically hold onto to the local variables as part of its traceback.

The solution is not to depend on the __del__ method. Rather, use a context manager.

class Dummy:
   def __enter__(self):
       return self

   def __exit__(self, type, value, traceback):
       print "Destroying", self

with Dummy() as dummy:
    # Do whatever you want with dummy in here
# __exit__ will be called before you get here

This is guaranteed to work, and you can even check the parameters to see whether you are handling an exception and do something different in that case.

Cunning

Answer 5

It seems to me the real heart of the matter is here:

adding the functions is dynamic (at runtime) and not known in advance

I sense that what you are really after is a flexible way to bind different functionality to an object representing program state, also known as polymorphism. Python does that quite well, not by attaching/detaching methods, but by instantiating different classes. I suggest you look again at your class organization. Perhaps you need to separate a core, persistent data object from transient state objects. Use the has-a paradigm rather than is-a: each time state changes, you either wrap the core data in a state object, or you assign the new state object to an attribute of the core.

If you're sure you can't use that kind of pythonic OOP, you could still work around your problem another way by defining all your functions in the class to begin with and subsequently binding them to additional instance attributes (unless you're compiling these functions on the fly from user input):

class LongRunning(object):
    def bark_loudly(self):
        print("WOOF WOOF")

    def bark_softly(self):
        print("woof woof")


while True:
    d = LongRunning()
    d.bark = d.bark_loudly
    d.bark()

    d.bark = d.bark_softly
    d.bark()

Sidwell

Answer 6

A full example of a context manager.

class Dummy(object):
    def __init__(self, name):
        self.name = name
    def __enter__(self):
        return self
    def __exit__(self, exct_type, exce_value, traceback):
        print 'cleanup:', d
    def __repr__(self):
        return 'Dummy(%r)' % (self.name,)

with Dummy("foo") as d:
    print 'using:', d

print 'later:', d

Concordance

Answer 7

An alternative solution to using weakref is to dynamically bind the function to the instance only when it is called by overriding __getattr__ or __getattribute__ on the class to return func.__get__(self, type(self)) instead of just func for functions bound to the instance. This is how functions defined on the class behave. Unfortunately (for some use cases) python doesn't perform the same logic for functions attached to the instance itself, but you can modify it to do this. I've had similar problems with descriptors bound to instances. Performance here probably isn't as good as using weakref, but it is an option that will work transparently for any dynamically assigned function with the use of only python builtins.

If you find yourself doing this often, you might want a custom metaclass that does dynamic binding of instance-level functions.

Another alternative is to add the function directly to the class, which will then properly perform the binding when it's called. For a lot of use cases, this would have some headaches involved: namely, properly namespacing the functions so they don't collide. The instance id could be used for this, though, since the id in cPython isn't guaranteed unique over the life of the program, you'd need to ponder this a bit to make sure it works for your use case... in particular, you probably need to make sure you delete the class function when an object goes out of scope, and thus its id/memory address is available again. __del__ is perfect for this :). Alternatively, you could clear out all methods namespaced to the instance on object creation (in __init__ or __new__).

Another alternative (rather than messing with python magic methods) is to explicitly add a method for calling your dynamically bound functions. This has the downside that your users can't call your function using normal python syntax:

class MyClass(object):
    def dynamic_func(self, func_name):
        return getattr(self, func_name).__get__(self, type(self))

    def call_dynamic_func(self, func_name, *args, **kwargs):
        return getattr(self, func_name).__get__(self, type(self))(*args, **kwargs)

    """
    Alternate without using descriptor functionality:
    def call_dynamic_func(self, func_name, *args, **kwargs):
        return getattr(self, func_name)(self, *args, **kwargs)
    """

Just to make this post complete, I'll show your weakref option as well:

import weakref
inst = MyClass()
def func(self):
    print 'My func'
#  You could also use the types modules, but the descriptor method is cleaner IMO
inst.func = func.__get__(weakref.ref(inst), type(inst))

Plug

There are, of course, a million other ways to get similar functionality without the circular reference. – Plug

Answer 8

use eval()

In [1]: int('25.0')                                                                                                                                                                                                                                                               
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-1-67d52e3d0c17> in <module>
----> 1 int('25.0')

ValueError: invalid literal for int() with base 10: '25.0'

In [2]: int(float('25.0'))                                                                                                                                                                                                                                                        
Out[2]: 25

In [3]: eval('25.0')                                                                                                                                                                                                                                                              
Out[3]: 25.0

Sympathy