Why can't my file handler's file within a temporary directory be cleaned up on Windows?

Question

I have some code which runs fine on unix systems, but not on Windows. I'd like to make it cross-platform, but am banging my head against the wall. The minimal repro is as follows:

File 1: foo.py

import os
import sys
import logging

logging.basicConfig(level=logging.INFO, stream=sys.stdout)
logger = logging.getLogger('foo')

def main(dir):
    logger.addHandler(logging.FileHandler(
        os.path.join(dir, 'temporary.log')
    ))

    logger.info("Hello, world!")

File 2: main.py

from foo import main

import tempfile

if __name__ == "__main__":
    with tempfile.TemporaryDirectory("test") as tmp:
        main(tmp)

What I'd expect is that the temporary directory would be created, a file would be created within that to which logs would be emitted, and then both would be cleaned up when tmp goes out of scope.

Instead, Windows provides an error:

PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: '...'

I've tried changing the mode of the FileHandler away from append mode, I've tried manually cleaning up the files and directories, I've tried delaying the file from being created until its logged to and cranked up the log level, and I've even tried instantiating the logger inside foo.main in hopes that no references to the handler would be persist -- regardless, I still see this error.

How can I fix this?

Amyotonia

"temporary.log" is opened without delete sharing. Even if it did share delete access, the 'deleted' file would actually remain linked in the directory as long as the logging handler has a reference to the file. (In the current release of Windows 10, it would be unlinked POSIX-style, but you can't assume this.) This in turn would cause cleaning up the temp directory to fail because it wouldn't be empty. – Colenecoleopteran

The main function could keep a reference to the handler to call logger.removeHandler(handler) in a finally block. This would ensure that the file is closed before main returns. – Colenecoleopteran

Answer 1

You need to close the handler, which closes the file. Then the deletion of the temporary directory should work. I made changes as follows:

# foo.py
import os
import sys
import logging

logging.basicConfig(level=logging.INFO, stream=sys.stdout)
logger = logging.getLogger('foo')

def main(dir):
    h = logging.FileHandler(os.path.join(dir, 'temporary.log'))
    logger.addHandler(h)
    logger.info("Hello, world!")
    logger.removeHandler(h)
    return h

and

# main.py
from foo import main

import tempfile

if __name__ == "__main__":
    with tempfile.TemporaryDirectory("test") as tmp:
        print('Using temp dir %s' % tmp)
        h = main(tmp)
        h.close()

Following which, it seems to work:

~> python3 c:\temp\main.py
Using temp dir C:\Users\Vinay\AppData\Local\Temp\tmp60qirkhutest
INFO:foo:Hello, world!

~> dir AppData\Local\Temp\tmp60qirkhutest
 Volume in drive C has no label.
 Volume Serial Number is D195-0C0D

 Directory of C:\Users\Vinay\AppData\Local\Temp

File Not Found

If you comment out the h.close() line, it fails as before.

Downspout

When the handler is removed in main and goes out of scope (assuming the OP's code that doesn't return the handler), the file is automatically closed in CPython -- at least it was in my test, though I haven't checked the code. You can close it explicitly to be certain. But removing the handler definitely needs to be in a finally block in order to ensure that cleaning up the temp directory won't fail if there's an unhandled exception. – Colenecoleopteran

@ErykSun I actually tried not returning the handler and just removing it and closing it in the finally block, but that didn't work for some reason... I somehow thought scoping would handle this for me. – Amyotonia

@erip, it worked fine for me. Closed is closed. I don't see any reason to have to return the handler, other than as a general means to pass control back to the caller. – Colenecoleopteran

Just telling you what I saw. :-) – Amyotonia

There's no reason to return the handler particularly, I was just using it to illustrate one way of resolving the issue. It's not the best way, necessarily, but I took OP's script to be just a means of illustrating the issue and addressed my changes in that spirit. Also, open handlers are held in an internal list and not necessarily closed when they go out of scope. – Downspout

I recently learned that there's also logging.shutdown() which can be called at the end of main. – Amyotonia