python lockf and flock behaviour

Question

I have read enough posts on stackoverflow regarding the difference between flock/lockf/fcntl but I am unable to answer the below observation:

>>> import fcntl
>>> a = open('/tmp/locktest', 'w')
>>> b = open('/tmp/locktest', 'w')
>>> fcntl.lockf(a, fcntl.LOCK_EX | fcntl.LOCK_NB)
>>> fcntl.lockf(a, fcntl.LOCK_EX | fcntl.LOCK_NB)
>>> fcntl.lockf(b, fcntl.LOCK_EX | fcntl.LOCK_NB)
>>>
>>> a.close()
>>> b.close()

>>> a = open('/tmp/locktest', 'w')
>>> b = open('/tmp/locktest', 'w')
>>> fcntl.flock(a, fcntl.LOCK_EX | fcntl.LOCK_NB)
>>> fcntl.flock(a, fcntl.LOCK_EX | fcntl.LOCK_NB)
>>> fcntl.flock(b, fcntl.LOCK_EX | fcntl.LOCK_NB)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IOError: [Errno 35] Resource temporarily unavailable

Why is the behaviour different in the two cases? I know the obvious answer that these are two different locking mechanisms. I am looking for:

1. What actually lockf() or flock() does to file (inode/fd)?
2. As per the demo, are we allowed taking the same lock recursively?

I understand the basics of fds and stuff so I would prefer to have a technical answer with more insights to operating system level details.

OSX 10.9.3, Python: 2.7.5

Eulaeulachon

I've taken the liberty to edit the question to actually include a question. Please review and edit if that's not what you're asking. Thanks. – Angi

@NPE: It was part of my first sentence though its better to have it explicitly put there. Thanks :) – Eulaeulachon

Answer 1

A nice article about this: On the Brokenness of File Locking

In short:

• POSIX locks:

  lockf() most of the time implemented as just an interface of fcntl()

  fcntl() locks are bound to processes, not file descriptors. If a process has multiple open file descriptors for a particular file, any one of these file descriptors used for acquiring locking will RESET the lock.

• BSD lock:

  flock() locks are bound to file descriptors, not processes.

Furthermore

A good analysis with tests: Advisory File Locking – My take on POSIX and BSD locks

Excerpt of Summary:

• fcntl and flock style locks are completely orthogonal to each other. Any system providing both(Linux does) will be treating locks obtained through each one of them independently.
• Both POSIX and BSD locks are automatically released when the process exits or is aborted.
• Both POSIX and BSD locks are preserved across execve calls except when the process has set FD_CLOEXEC flag forcing the file descriptor to be closed, and not inherited by the new process.

Ochs

A fair assessment. It should be emphasized that POSIX locks silently fail under multithreading scenarios and should (arguably) be considered harmful for use in modern software stacks – GIL notwithstanding. More critically, it should be emphasized that both POSIX and BSD locks on group- or world-readable files pose an extreme security risk. Why? Because any user with read permissions for the file to be locked can permanently deadlock your Python application by preemptively acquiring that lock first and then never releasing it. – Toothless

tl;dr: (A) in-memory locks (e.g., multiprocessing module synchronization primitives) are strongly preferable to both BSD- and POSIX-style locks, (B) BSD-style flock locks are strongly preferable to POSIX-style fcntl locks, and (C) neither BSD- nor POSIX-style locks should be used unless permissions for the file(s) to be locked are strictly constrained to at most 0600. – Toothless

The only thing I would add to this answer is to note that you're using the fcntl.LOCK_NB flag. If you remove this flag, then your second case will block instead of raising an exception. – Paquito

Also, if you run another python session, in another window, and use lockf() on the file and leave the file open, then you'll find the exception occurs in the first call to fcntl.lockf(), in your second session (if we're back to using fcntl.LOCK_NB). This confirms that the state of fcntl.lockf() process-global, while the state of fcntl.flock() is scoped by the file descriptor. – Paquito

Regarding @CecilCurry's link, notice that Python multiprocessing module is only applicable to synchronize Python subprocesses spawned by a common parent, and if one of the process crashes this leaks semaphores and shared memory segments and corrupts Queue objects and pipes. – Lenten

I would hope that opening a file with O_EXCL is not relying on these locks, while my orientation into navigating linux source code is very lacking ... – Woehick