Python : Locking text file on NFS

Question

I have a file results.txt on a server which is accessed by multiple VMs through NFS. A process runs on each of these VMs which reads the results.txt file and modifies it. If two processes, A and B, read the file at the same time, then modification of either A or B would be present in results.txt based on the order in which the processes write to the file.

If process A has a write lock over the file then process B would have to wait till the lock is released to read the results.txt file.

I have tried implementing this using Python:

import fcntl


f = open("/path/result.txt")
fcntl.flock(f,fcntl.LOCK_EX)
#code

It works as expected for files on the local disk.

but when I run try to lock a file on the mounted path, I get the following error:

Traceback (most recent call last):
  File "lock.py", line 12, in <module>
    fcntl.flock(f,fcntl.LOCK_EX)
IOError: [Errno 45] Operation not supported 

I tried fcntl.fcntl and fcntl.flock but got the same error. Is this an issue with the way I am using fcntl? Is any configuration required on the server where file is stored?

Edit:

This is how I am using fcntl.fcntl:

f= open("results.txt")
lockdata = struct.pack('hhllhh', fcntl.F_RDLCK,0,0,0,0,0)
rv = fcntl.fcntl(f, fcntl.F_SETLKW, lockdata)

The NFS server version is 3.

Needham

Could you paste the way you use fcntl.fcntl()? – Taddeo

@vmonteco, I have added the code snippet. – Needham

Does fcntl.fcntl(f, fcntl.LOCK_EX) work? – Taddeo

@Taddeo It doesn't give any error. But I don't think lock is being applied. I am able to run the script on the same file in two different sessions and none of them wait. – Needham

Answer 1

I found flufl.lock most suited for my requirement.

Quoting the author from project page:

[...] O_EXCL is broken on NFS file systems, programs which rely on it for performing locking tasks will contain a race condition. The solution for performing atomic file locking using a lockfile is to create a unique file on the same fs (e.g., incorporating hostname and pid), use link(2) to make a link to the lockfile. If link() returns 0, the lock is successful. Otherwise, use stat(2) on the unique file to check if its link count has increased to 2, in which case the lock is also successful.

Since it is not part of the standard library I couldn't use it. Also, my requirement was only a subset of all the features offered by this module.

The following functions were written based on the modules. Please make changes based on the requirements.

def lockfile(target,link,timeout=300):                                             
        global lock_owner                                                          
        poll_time=10                                                               
        while timeout > 0:                                                         
                try:                                                               
                        os.link(target,link)                                       
                        print("Lock acquired")                                      
                        lock_owner=True                                            
                        break                                                      
                except OSError as err:                                             
                        if err.errno == errno.EEXIST:                              
                                print("Lock unavailable. Waiting for 10 seconds...")
                                time.sleep(poll_time)                              
                                timeout-=poll_time                                 
                        else:                                                      
                                raise err                                          
        else:                                                                      
                print("Timed out waiting for the lock.") 

def releaselock(link):                          
        try:                                    
                if lock_owner:                  
                        os.unlink(link)         
                        print("File unlocked")   
        except OSError:                         
                print("Error:didn't possess lock.")

This is a crude implementation that works for me. I have been using it and haven't faced any issues. There are many things that can be improved though. Hope this helps.

Needham