Is it safe to use flock on AWS EFS to emulate a critical section?

Question

According to the docs, AWS EFS (Amazon Elastic File System) supports file locking:

Amazon EFS provides a file system interface and file system access semantics (such as strong data consistency and file locking).

On a local file system (e.g., ext4), flock can be used in shell scripts to create a critical section. For example, this answer describe a pattern that I used in the past:

#!/bin/bash
(
  # Wait for lock on /var/lock/.myscript.exclusivelock (fd 200) for 10 seconds
  flock -x -w 10 200 || exit 1

  # Do stuff

) 200>/var/lock/.myscript.exclusivelock

Can the same pattern be applied on EFS? Amazon mentions that they are using the NFSv4 protocol, but does it provide the same guarantees as flock on ext4?

If not, how can you enforce that an operation runs exclusively across all EC2 instances that are attached to the same EFS volume? It is sufficient if it works for processes, as I'm not planning to run multiple threads.

Or did I misunderstood the locking support provided in NFSv4? Unfortunately, I don't know the details of the protocol, but providing atomicity in a distributed system is a much harder problem than on a local machine.

Update: small scale experiment

Not a proof, of course, but in my tests it works across multiple instances. For now, I assume the pattern is safe to use. Still, would be nice to know if it is theoretically sound.

Fiducial

I'm testing this out myself too. So far I haven't found any issues. – Dang

@ScottTalbert Yes, I was thinking of writing an answer myself. From my understanding, the pattern above should be supported with NFS, which Amazon's EFS implements. I have also not encountered any problems so far. – Damico

Answer 1

It should work.

The flock command as used in the pattern in the question should work on all NFS file systems. That means, it will also work on EFS, which implements the NFSv4 protocol. In practice, I also did not encounter any problems so far when using it to synchronize shell scripts on different EC2 instances.

---

Depending on your use case, you have to aware of the gotchas of file locking on Linux, although most of it is not NFS specific. For instance, the pattern above operates on the process level, and cannot be used if want to synchronize multiple threads.

While reading, I came across old issues. In kernels prior to 2.6.12, there seemed to be problems with NFS and the flock system call (e.g., see flock vs lockf on Linux).

It should not apply here, as it has been improved in newer kernels. Looking the source code of the flock command, you can confirm that it still uses the flock system call, but it could be potentially implemented by the safe fcntl system call:

while (flock(fd, type | block)) {
  ...
  case EBADF:       /* since Linux 3.4 (commit 55725513) */
        /* Probably NFSv4 where flock() is emulated by fcntl().
         * Let's try to reopen in read-write mode.
         */

Note: the workaround refers to this commit in the Linux kernel can be found:

Since we may be simulating flock() locks using NFS byte range locks, we can't rely on the VFS having checked the file open mode for us.

Fiducial

Unfortunately, I may have just found a case where this does not work. I left two EC2 instances running over the weekend, one holding a flock and the other waiting to acquire it. It seems that at a certain point, the machine holding the flock lost connectivity to the NFS server (131090ms. Dec 2 05:04:32 ip-172-31-30-242 kernel: [105059.038603] nfs: server fs-e05dfc48 .efs.us-west-2.amazonaws.com not responding, timed out). At this point, it seems that the other system was able to acquire the lock, while the original system still seems to thing that it has it. Two systems with the lock. – Dang

@ScottTalbert Interesting. In my setup, I'm using flock with a timeout of 200 seconds. Most locks are held for a very short period (<1 sec). I'm not using it to synchronize long-running batch jobs. – Damico

Yes, in my case, I was trying to use it to prevent multiple nodes from assuming a 'leader' role, which will persist for a long time. It doesn't seem to be suitable for this use case, as it seems that NFS can reclaim locks, as in the case I saw where one node lost connectivity to the server briefly. – Dang

Answer 2

If the use-case is just to make sure that another process (or instance/container) doesn't "take over" the job, I'd use a simpler lock file instead. I/it's called a lock file, but it's really just a simple file.

Something like

while true; do
    printf "Aquiring lock: "
    if [ ! -e "some_lock_file_somewhere" ]; then
        echo "done."
        touch some_lock_file_somewhere
        echo "doing stuff"
        sleep 60 # just because I couldn't come up with something that takes a while :D 
    else
        echo "waiting 60s for lock"
        sleep 60
    fi
done

You'd have to remove that lock file manually, or write the logic to do it, but that can then be run in multiple shells and only the first one will do the actual work.

Yulan

This looks like a race to me, since it is not an atomic test-and-set. If you run two scripts at the same time, it may happen that both don't see the file at first and both ending up executing the block. For some use cases, that may be OK, but it is not my understanding of a critical section (which should grant exclusive access). – Damico

Yes, that is true. But the likelihood of that happening is REALLY small. If you want to minimise the risk of that happening, if you have multiple instances coming up at the same time (at creation of infrastructure for example which triggers X number of instances from an ASG), then I'd put a random (between 1 and 10 seconds?) sleep in front of that section. Btw, that can actually still happen with flock! – Yulan

Wow, this is horrible, the risk is not small at all, especially on remote filesystems where there is a delay from creation to visibility on other nodes. And no, it cannot happen with flock() if used correctly. – Mandarin