inotify with NFS

Question

I've recently created a dropbox system using inotify, watching for files created in a particular directory. The directory I'm watching is mounted from an NFS server, and inotify is behaving differently than I'd expect. Consider the following scenario in which an inotify script is run on machine A, watching /some/nfs/dir/also/visible/to/B.

-Using machine A to create a file in /some/nfs/dir/also/visible/to/B, the script behaves as expected. Using machine B to carry out the same action, the script is not notified about a new file dropped in the directory.
-When the script is run on the NFS server, it gets notified when files are created from both machine A and machine B.

Is this a bug in the bug in the package I'm using to access inotofy, or is this expected behaviour?

Geter

So you're saying that only the script on the server that owns the filesystem or the computer that made the change gets notified? That is the behavior I would expect. – Refugee

I reported a feature request here. – Pictor

I think this question could be moved (migrated) to serverfault.com – Villalpando

Workaround script for mounted drives. Somewhat hacky, but to execute-after-save while-developing it's good enough! – Wellbeloved

Answer 1

inotify requires support from the kernel to work. When an application tracks a directory, it asks the kernel to inform it when those changes occur. When the change occurs, in addition to writing those changes to disk, the kernel also notifies the watching process.

On a remote NFS machine, the change is not visible to the kernel; it happens entirely remotely. NFS predates inotify and there is no network level support for it in NFS, or anything equivalent.

If you want to get around this, You can run a service on the storage server (since that kernel will always see changes to the filesystem) that brokers inotify requests for remote machines, and forward the data to the remote clients.

Edit: It seems odd to me that NFS should be blamed for its lack of support for inotify.

Network File System (NFS) is a distributed file system protocol originally developed by Sun Microsystems in 1984, wikipedia article

However:

Inotify (inode notify) is a Linux kernel subsystem that acts to extend filesystems to notice changes to the filesystem. [...] It has been included in the mainline Linux kernel from release 2.6.13 (June 18, 2005 ) [...]. wikipedia article

It's hard to expect a portable network protocol/application to support a specific kernel feature developed for a different operating system and that appeared more than twenty years later. Even if it did include extensions for it, they would not be available or useful on other operating systems.

_{*emphasis mine in all cases}

---

Another problem with this; Lets suppose we are not using a network at all, but rather, a local filesystem with good inotify support: ext3 (suppose its mounted at /mnt/foo). But instead of a real disk, the filesystem is mounted from a loopback device ; and the underlying file is in turn accessible at a different location in the vfs (say, /var/images/foo.img).

Now, you're not supposed to modify mounted ext3 filesystems, But it's still reasonably safe to do so if the change is to file contents instead of metadata.

So suppose a clever user modifies the file system image (/var/images/foo.img) in a hex editor, replacing a file's contents with some other data, while at the same time an inotify watch is observing the same file on the mounted filesystem.

There's no reasonable way one can arrange for inotify to always inform the watching process of this sort of change. Although there are probably some gyrations that could be take to make ext3 notice and honor the change, none of that would apply to, say, the xfs drtiver, which is otherwise quite similar.

Nor should it. You're cheating!. inotify can only inform you of changes that occured through the vfs at the actual mountpoint being watched. If the changes occured outside that VFS, because of a change to the underlying data, inotify can't help you and isn't designed to solve that problem.

Have you considered using a message queue for network notification?

Insectivorous

This just seems to a bug. Yes the change is made elsewhere, but the fact is NFS clients do eventually discover newly added files. Even if the event were deferred, it should be possible for a NFS client to notify inotify listeners when it discovers newly added / removed resources. – Starshaped

@JamesBlackburn : this is likely for performance reasons. NFS clients probably only discover changes lazily, upon request. – Mccabe

Of course, NFS can not be blamed for not supporting inotify. But, you example is totally flawed. An NFS implementation could support something like inotify and it could map its (NFS internal protocol) stuff to inotify stuff. It does not. So it is – Derrik

I need to implement something like this, but I'm confused on how I would go about using message queues to solve the problem. Do you mind elaborating? I have about 13-14 instances of different (VMs) that each access a mounted NAS, and since inotify is per kernel/VM, I run into this dilemma. – Bandage

@meder: that sounds an awful lot like a question! – Insectivorous

The suggestion of proxying inotify events is very sensible and there are implementations out there: github.com/sillypog/inotify-proxy – Dissolution

Can this work on NAS Storage if it is mounted like a directory on a server? – Barbirolli

Your last part comparing nfs to loopback is not really relavent or an accurate anology. NFS shares are designed to be modified by multiple systems. The protocol is written with that in mind. Loopback devices do not support writing to the underlying file directly. There is zero code designed to check for stale pages of cache having been updated on disk. – Teleprinter

Answer 2

To anyone who has come across this question in the search for an answer of why bind mounting on Docker will not detect file changes from host directory (for hot reloading of an app), it's because the propagation of file changes between host and container is not communicated to the container kernel.

Only changes from the container itself is communicated to the kernel. Solution for this is to have your live reload utility turn on "polling mode" instead of using fsnotify.

Dinka

Yes and if u came from java world here is an example how to do it easily with apache-commons -> github.com/eugenp/tutorials/blob/master/core-java-io/src/main/… – Candytuft

Can this work on NAS Storage if it is mounted like a directory on a server? – Barbirolli

Actually I just tried this on a CentOS 7 host and container guest with Docker 18, and it worked, even on an NFS host volume. – Allow

In Docker there is no such thing as a container kernel: There is only one kernel which is shared between the host and the container. Thus inotify works inside the container if used on bind mounts from host local filesystems. But it does not work if the bind mount is of a NFS filesystem where the docker host is the NFS client and the filesystem is changed by some other NFS client or directly on the NFS server. – Pederson

Answer 3

I found an SGI FAM using an supervisor daemon to monitor file modification. It supports NFS and you can see some description on wiki

Trichina

It seems that FAM requires a FAM server running on the remote FS in order to work. Given that FAM is even older than inotify, I wonder why the later doesn't support remote FS. – Biped

Broken link, maybe go here instead: linuxfromscratch.org/blfs/view/5.1/general/fam.html – Ahrens

Answer 4

the problem with notify-forwarder is that it does not trigger an inotify event. It uses utime to update the timestamp for the file on the remote system but inotify fails to see this.

AFAIK, the timestamp already gets updated when using an NFS mount. I have verified this myself between a Synology NAS NFS server and a Raspbian NFS mount (client).

Here's my solution / hack on the client:

#!/bin/bash 
path=$1
firstmd5=`ls -laR $path | md5sum | awk ' { print $1 }'`

    while true
    do 
       lastmd5=`ls -laR $path | md5sum | awk ' { print $1 }'`
       if [ $firstmd5 !=  $lastmd5 ]
       then
          firstmd5=$lastmd5
          echo files changed
       fi
       sleep 1
    done

Granted, this doesn't report on the specific file being changed, but does provide a general notification hook that something's changed.

It's annoying / kludgy but if I needed more details I would do some additional hacking to isolate the actual files changed.

Diffuser

Answer 5

The question is old, but continues to be at the top of search results and deserves a more correct answer.

From the inotify docs:

Inotify monitoring is inode-based: when monitoring a file (but not when monitoring the directory containing a file), an event can be generated for activity on any link to the file (in the same or a different directory).

"Inotify monitoring is inode-based" means inotify is only capable of monitoring file system changes coming from the same machine as the process that is watching the files for changes.

So if you have Machine A and Machine B and B makes a change, only processes watching changes on B can see the change. If a process on A is watching the files, it will not see any changes that B makes.

This is a fundamental limitation of how inotify works: it's watching each mounted filesystem's inode table for calls coming into the kernel wishing to operate on those inodes. It's not a historical artifact as implied by @SingleNegationElimination in his answer, not is it something that "can be fixed" given enough time, as implied by @JamesBlackburn's comment.

The FAM route suggested by @cmchao seems interesting, but the link is broken to the original FAM website. There is this page I found, but it still seems like libfam is very old and unmaintained. I wouldn't feel comfortable using it at this point.

Ahrens

Answer 6

I agree with SingleNegationElimination's explanation, and would like to add that iSCSI targets will work, since they alert the kernel.

So things on "real" file systems (relative to the system, that is) will trigger Inotify to alert. Like Rsync'ing, net-catting something into a mounted partition.

If you have to get notifications via inotify (or have to use inotify) you can make a cron to rsync -avz over to the file system. Drawbacks of course are that you are using real system hdd space.

Dermot

All targets work across all filesystems, just not across machines. See my answer: https://mcmap.net/q/278494/-inotify-with-nfs – Ahrens

Answer 7

I second @SingleNegationElimination.

Also, you can try notify-forwarder.

• Machine A watches for local inotify events, then forwards them to Machine B (via UDP).
• Machine B doesn't (can't?) replay the events, but fires an ATTRIB event for the changed file.

If you use vagrant, use vagrant-notify-forwarder.

Cida

Can this work on NAS Storage if it is mounted like a directory on a server? – Barbirolli

@JaskaranSinghPuri inotify works at the VFS layer of the kernel, so I think you will see file operations performed on the machine running inotifywait, but not operations performed on a different remote client. – Cida

Answer 8

improved the script with action on click and icon

#!/bin/bash
DAT=$(date +%Y%m%d)
CAM="cam1 "
CHEMIN=/mnt/cams/cam1/$DAT/

first="$CHEMIN"
if [ -d "$CHEMIN" ];then
    first=`ls -1rt $CHEMIN | tail -n 1`
fi

echo $first
    while true
    do 
     if [ -d "$CHEMIN" ];then
            last=`ls -1rt $CHEMIN | tail -n 1`
            if [ $first !=  $last ]
            then
               first=$last
               echo $last created
                         #notify-send -h string:desktop-entry:nautilus -c "transfer.complete" -u critical -i $PWD../QtVsPlayer.png $CAM $last"\n\r"$CHEMIN 
      reply=$(dunstify -a QtVsPlayer -A 'open,ouvrir' -i "QtVsPlayer" "$CAM $last"\n\r"$CHEMIN")
        if [[ "$reply" == "open" ]]; then
            QtVsPlayer -s $CHEMIN$last
        fi
             fi
      fi
      sleep 5m
    done

Atalaya