Why does the Python filelock library delete lockfiles on Windows but not UNIX?

Question

I'm using the filelock module for Python.

• On Windows, when a lock is released, the file that's backing it is deleted.

• On UNIX, lock files still exist on the filesystem even after the locks are released.

Is there a reason this is different between operating systems? If there isn't a reason for it to differ, which of these behaviors is more correct?

Dipietro

I've edited this into a more clear and specific question; hopefully it'll be reopened. (Feel free to @-notify me if/when this happens, and I'll write up an answer with an extended version of the content in my comments above). – Anarchist

@CharlesDuffy: The question is open, at this moment. – Brat

@user000001, thank you; I've added an answer merging the details from the now-deleted comments with an expansion on the content of the previously-referenced comment thread. – Anarchist

@CharlesDuffy, this question is being discussed on Meta. – Sauder

Answer 1

Speaking Specifically To py-filelock

The filelock library used to delete lockfiles on UNIX; this behavior was removed as of benediktschmitt/py-filelock#31, which refers to flock(): removing locked file without race condition? -- which discusses the same race condition described in a later section of this answer.

---

Why General Practices Differ

The operating-system semantics are different, so different approaches are appropriate in each case. In UNIX, you can delete a file even while there's an open handle, so lockfiles must not be deleted or else two programs can both think they hold the same lock, when in fact they hold locks on completely different inodes which were, at different points in time, referenced under the same filename.

By contrast, default filesystem semantics on Windows make it impossible to delete a file while any program has it open (even though NTFS is powerful enough to support it, it's artificially prevented for backwards compatibility with programs designed around FAT limitations), so on Windows, it's safe to delete a lockfile: If the deletion goes through, that proves that nobody held the lock (or was even in the process of opening the file to later grab a lock on it).

---

A Specific Race Example

To provide an example of how allowing open files to be unlinked on UNIX makes deleting lockfiles dangerous, consider the following illustration of a common race condition:

• Program 1 creates and opens file A1 (under the name "A"), receiving a file handle attached to the inode (the object reflecting the actual file itself, not the directory entry it's attached to) for that newly-created file.
• Program 1 requests an exclusive advisory lock on that handle. No other processes have a handle on that same file, so its request for a lock is granted.
• Program 2 opens file A1, receiving a second file handle on same.
• Program 2 requests an exclusive advisory lock on that handle. However, because Program A already holds a lock, the request blocks -- which is to say, the program waits for the operating system to pass control back to it later, when the lock can be granted.
• Program 1 finishes the processes it needed the lock for.
• Program 1 uses the unlink() syscall to delete the lockfile. (To be safe on UNIX, just leave this step out). This doesn't delete the file itself (the "inode") until no programs have it open, but it does immediately delete the link to that file from the directory that previously contained it.
• Program 1 closes its handle on the file, thus releasing its lock. The inode is not deleted, because Program 2 still holds a handle.
• Program 2 is granted the lock it's been waiting for on the file handle (on the now-unlinked file A1) it opened back before the deletion happened, and is able to resume execution.
• Program 3 creates and opens a new file A2 (with a new and distinct inode) under the same name "A" (which is available because A1's inode is no longer linked to that name), getting a file handle on same.
• Program 3 requests a lock on the file handle it owns. This is immediately granted, because A2 is a different file from A1, and the still-running Program 2 holds a lock on A1, not A2. Thus, we end up with two programs -- Program 2 and Program 3 -- thinking they hold the same lock.

Consequently, the above illustrates how on UNIX, deleting lockfiles allows race conditions wherein a lock can appear to be held by two programs at once.

Anarchist

Sorry if this is dumb, but it feels like this can be easily fixed? If the issue is that Program 2 acquires the same A1 handle as Program 1, preventing the inode for A1 from being deleted, can't Programs 2 periodically poll A1 so that the inode gets the chance to be deleted? – Actual

@crypdick, ...if you switch to a polling model where the directory entry and not the inode is what "locked" status attaches to, you might as well give up on the inode-attached advisory lock approach altogether. Which is to say -- you'd no longer be able to let the OS inform you when a lock frees up but would be obligated to poll. That's a workable model, but it's an entirely different kind of locking than what we're discussing here. – Anarchist

@crypdick, ...I feel like there's probably an assumption somewhere in your comment that doesn't quite hold, but it's too short an explanation for me to really tease out what the expectations are (even "so that the inode gets a chance to be deleted" is a little bit off -- since we're attaching the lock to the inode, used the conventional way we don't want the inode to be deleted; even just detachment of the inode from the directory entry is sufficient to break the ability of other tools to grab the same lock) – Anarchist

Yes, I was suggesting an alternative mechanism that would permit lockfiles to be deleted. It seems absurd that the status quo on Linux is to abandon lockfiles even after the program has finished executing. – Actual

I disagree that it's absurd in any way, but to really argue the point I'd want to be in person with a whiteboard. Anyhow -- if you want something that allows deletion, poll and use mkdir/rmdir. You lose various advantages (like automatic cleanup when an application crashes, or notification from the OS with no need to poll), but if not leaving anything behind is more important to you, that's where you end up. – Anarchist