Why there is a delay in detecting a file change using WatchService? (Java)

Question

I've an application in which when a file is added to the directory, WatchService detects the file and the file is added to a file list for further processing. This is my code

 public void run() {

    /*
     * Goes in an infinite loop
     */
     while(!finished) {

     /*
      *  Get a watch key, poll() returns a queued key 
      *  if no queued key, this method waits until the specified time.
      */
     WatchKey key;
     try {
             key = watcher.poll(eofDelay,TimeUnit.MILLISECONDS);
      } catch (InterruptedException x) {
          return;
      }

     Path dir = keys.get(key);

     if (dir == null) {
         continue;
      }

     Path child=null;

         /*
          * Fetching the list of watch events from
          * pollEvents(). There are four possible events
          */

         for (WatchEvent<?> event: key.pollEvents()) {
            WatchEvent.Kind kind = event.kind();

            /*
             * Overflow indicates that events 
             * might have been lost or discarded
             */
             if (kind == OVERFLOW) {
                 continue;
             }


             WatchEvent<Path> ev = cast(event);

             /*
              * Filename is the context of the event
              */
             Path name = ev.context();

             /*
              * Resolves the name of the file to a path
              */
              child = dir.resolve(name);

             /*
              *  If directory is created, and watching recursively, then
              * register it and its sub-directories
              */
             if (nested && (kind == ENTRY_CREATE)) {
                 try {
                     if (Files.isDirectory(child, NOFOLLOW_LINKS)) {
                         registerAll(child);
                     }
                 } catch (IOException x) {

                 }
             }
         }

         File file = child.toFile();

         /*
          * Only add the file if there is no wild card 
          * or it matches the specified wild card 
          */
         if (matcher == null || matcher.matches(file.toPath().getFileName())) {
             fileList.add(file);
         }
     /*
      * Key is reset so that it can receive further
      * events 
      */

         boolean valid = key.reset();
         if (!valid) {
             keys.remove(key);

            /*
             * If key is no longer valid and empty,
             * exit the loop
             */
             if (keys.isEmpty()) {
                continue;
             }
         }

     }
 }

This code works as expected but I'm designing a high performance application, which processes data in the files at very high speed. So the problem here is inconsistency in time taken to detect a file. For instance initially there are some files in the directory and they're processed by the application, now when a new file is added it takes 4-5 sec to detect the file or sometimes it takes 2 sec or 20ms and so. My eofDelay value is 10ms. What is the reason for this inconsistency? Is there a way to enhance this implementation? Or any other efficient library available for directory changes? I want the time taken to detect a file to be minimal and consistent, taking more than a second is very expensive. Any help in this regard will be appreciated. :)

Asthenopia

I would suggest (where applicable) you are at the mercy of the underlying OS and file system. The delay may occur because the functionality to "batching" the update calls and is waiting for a suitable time to dispatch the event - This conjecture, I've only used JNI in the past this type of solution and there was always some kind of delay. – Dyak

@Dyak Oh I understand, may be I should try JNI and use the one with a consistent delay. – Asthenopia

I think you'll still end up with the same problem - IMHO – Dyak

Are you using OSX? We see that it delay works as expected in Ubuntu but on OSX there are up to 5 seconds delay. – Pangenesis

@Pangenesis Yes I'm using OS X only. – Asthenopia

#9589237 --> second answer works for me, delay is noticable reduced, even if it is not as good as on by Ubuntu system – Pangenesis

Possible duplicate of stackoverflow.com/questions/9588737 – Celestina

Answer 1

You may be able to get faster results by adding a sensitivity flag to the folder (see below).

// copied from https://mcmap.net/q/338645/-is-java-7-watchservice-slow-for-anyone-else
folder.register(watcher, new WatchEvent.Kind[]{StandardWatchEventKinds.ENTRY_MODIFY}, SensitivityWatchEventModifier.HIGH); 

However, you will still be at the mercy of the underlying OS. Most file watching applications I've seen have a few second delay between when the file is added and when it is picked up. You're seeing normal lag times in your application.

If your application must respond to a few file being added in a few milliseconds, you should not use Java (NIO or otherwise), but C/C++. This will add significant complexity to your code.

Stapes

The WatchServices are highly optimized. On Windows it uses an ReadDirectoyChangesW which is as efficient as it gets. On Linux it tries to use iNotify. – Pronounced

Thanks for the link. I've used NIO on windows and saw the same thing as the original poster. Do you have first hand accounts that differ significantly? – Stapes

Did you try to use a local or shared directory? (shared requires SMB3 Windows 8) to be efficient. – Pronounced

I used a local FS. Sometimes I'd get a message almost immediately, sometimes it would be a second or two before I'd get a message. – Stapes