Does deleting a dictionary close the file descriptors inside the dict?

Question

Clarified

There are two questions indeed. Updated to make this clearer.

I have:

t = {
    'fd': open("filename", 'r')
}

I understand that del t['fd'] removes the key and closes the file. Is that correct?

Does del t call del on contained objects (fd in this case)?

Idette

It will eventually, but that may not be until the script ends — so it's not a good idea to count on it. – Thirza

Generally, it's better to close the file explicitly (either using close or a with statement). Deleting the variable (directly or as part of a dict) isn't really the right way to do it – Arbe

Per the documentation: "Warning Calling f.write() without using the with keyword or calling f.close() might result in the arguments of f.write() not being completely written to the disk, even if the program exits successfully." docs.python.org/3/tutorial/… – Arbe

Related question: Does Python GC close files too? – Frediafredie

Answer 1

The two parts of your question have completely different answers.

• Deleting a variable to close file is not reliable; sometimes it works, sometimes it doesn't, or it works but in a surprising way. Occasionally it may lose data. It will definitely fail to report file errors in a useful way.

  The correct ways to close a file are (a) using a with statement, or (b) using the .close() method.

• Deleting an object indeed deletes all contained objects, with a couple of caveats:

  • if (some of) those objects are also in another variable, they will continue to exist until that other variable is also deleted;

  • if those objects refer to each other, they may continue to exist for some time afterwards and get deleted later; and

  • for immutable objects (strings, integers), Python may decide to keep them around as an optimisation, but this mostly won't be visible to us and will in any case differ between versions.

Strohl

NB: when you use .close() instead a context manager (i.e. with), you should do that using try-finally to make sure the file is actually closed, even when some exception occurred. – Frediafredie

From all the answers I get that del t['fd'] does not deterministically and immediately close the file, even if that is the last reference to the file object. However, this conclusion is always hinted at, never stated explicitly with a reference, only recommendations. This sounds strange to me. I suppose a file object has a destructor, and this destructor closes the file. Is that true? If true, calling del on the last reference to the file object should not close the file immediately and reliably? (should I make this into a different question?) – Towns

That's what it says in the python manual; "might" lose data. See the red box at docs.python.org/3/tutorial/… – Arbe

There are a couple of circumstances where she fine might not be closed properly at all; especially if the handle ends up in a global variable or a reference cycle. The other thing is that if the file close has an error, there's no way to report it; nowhere to raise an exception. – Arbe

The main situation where it doesn't matter is if the file is opened read-only and the script is short-lived; in that case, it doesn't matter at all whether or not it's closed. It's still a good habit to do it properly, though. – Arbe

Answer 2

The behavior of what you want to do is nondeterministic. Depending on the implementation and internal functioning of Python (CPython, PyPi, ...), this can be work or not:

Working example:

t = {
    'fd': open('data.txt', 'r')
}

def hook_close_fd():
    print('del dictionary, close the file')

t['fd'].close = hook_close_fd

del t

Output:

del dictionary, close the file

In this case, the close function is called on delete

Non working example:

t = {
    'fd': open('data.txt', 'r')
}

def hook_close_fd():
    print('del dictionary, close the file')

t['fd'].close = hook_close_fd

3 / 0

Output of 1st run:

---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)
<ipython-input-211-2d81419599a9> in <module>
     10 t['fd'].close = hook_close_fd
     11 
---> 12 3 / 0

ZeroDivisionError: division by zero

Output of 2nd run:

del dictionary, close the file
---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)
<ipython-input-212-2d81419599a9> in <module>
     10 t['fd'].close = hook_close_fd
     11 
---> 12 3 / 0

ZeroDivisionError: division by zero

As you can see, when an exception is raised, you can't be sure if your file descriptor will be closed properly (especially if you don't catch exception yourself).

Sublieutenant

Ok, I think your answer and mine complement each other, I didn't know that close was called directly when deleting the descriptor. – Woadwaxen

Per the documentation: "Warning Calling f.write() without using the with keyword or calling f.close() might result in the arguments of f.write() not being completely written to the disk, even if the program exits successfully." – Arbe

There's no way to tell if it's being called because the script is ending, or when the dict is deleted. – Thirza

@martineau, that is simply tested by adding a sleep after the delete. However, that still proofs nothing. It is clearly not guaranteed but implementation-defined. – Frediafredie

Apparently PyPy does it differently, for one – Arbe

There are also circumstances where the last reference lasts longer than you'd expect; eg if an exception is raised, it'll keep a reference to all local variables – Arbe

In any case, it's documented to be unsafe; if it happens to work under some circumstances, that's an implementation detail or even just luck – Arbe

One real problem is that if closing the file has an error, there's nowhere for the exception to be raised, much less handled. A lot of file errors can happen on close... – Arbe

@JiříBaum. I tested with an exception and the behavior is different between two runs. Nice point. – Sublieutenant

I think that may not be random, just later than expected; if that's IPython / Jupyter, it's quite possible that the "del dictionary, close the file" in the second run actually belongs to the first run, triggered when the variable t and/or the "last unhandled exception" variable got new values (thereby deleting the values from the first run). You can test that by using a different message on each run. Not that it changes much - it's still not the desired, reliable behaviour. – Arbe

Answer 3

No. You must close the files you open (see note).

Or, but it may not fit your project, in a more safer way open it in a context manager.

with open("filename", 'r') as fd:
    ...

If you are opening simultaneously a unknown number of files, you may need to write you own context manager or more simply use contextlib.ExitStack

Note: From : "3.10.0 Documentation / The Python Tutorial / 7. Input and Output / 7.2. Reading and Writing Files https://docs.python.org/3/tutorial/inputoutput.html#reading-and-writing-files": Warning Calling f.write() without using the with keyword or calling f.close() might result in the arguments of f.write() not being completely written to the disk, even if the program exits successfully.

Keefer

Answer 4

Your question is less trivial than you think. But deep down it all comes down to understanding what a pointer is and how the garbage collector works.

Indeed, by doing t['fd'], you delete that entry in the dictionary. What you do is delete the pointer that points to that entry. If you have a dictionary as follows:

t = { 'a':3, 'b':4 }

Then when you do del t you delete the pointer to the dictionary t. As there is therefore no further reference to the dictionary keys, the garbage collector deletes these as well, thus freeing up all the dictionary memory.

However, as long as you have a file descriptor, you can delete it but it is not desirable to do so since the file descriptor (fd, pointer to a file and much more ) is the way the programmer interacts with a file, if the descriptor is deleted abruptly, the state of the file can get corrupted by being in an inconsistent state.

Therefore, it is a good idea to call the close function before you stop working with a file. This function takes care of all those things for you.

Woadwaxen

Answer 5

Yes, deleting the variable t closes the file (using psutil to show open files per process):

import psutil


def find_open_files():
    for proc in psutil.process_iter():
        if proc.name() == 'python3.9':
            print(proc.open_files())


filename = '/tmp/test.txt'
t = {'fd': open(filename, 'r')}
find_open_files()
del t
find_open_files()

Out:

[popenfile(path='/private/tmp/test.txt', fd=3)]
[]

Jordonjorey

As discussed in various other answers and comments, this is not reliable and may lose data in some circumstances – Arbe

The time I answered, the question was about: Does deleting an object delete all contained objects? Not, if there are any side effects. – Jordonjorey

If that's the question, then "yes, deleting the variable t closes the file" is a strange answer, don't you think? (Since the question is not about closing a file.) Although this is a common problem when the question is based on an incorrect premise and contains multiple questions in one... – Frediafredie