Why does ReadDirectoryChangesW omit events?

Question

I use ReadDirectoryChangesW to watch a specified directory and update indexing structures whenever a change is detected. I use the following code (roughly)

var
  InfoPointer : PFileNotifyInformation;
  NextOffset : DWORD;
...
while (not Terminated) do begin
  if ReadDirectoryChangesW (FDirHandle, FBuffer, FBufferLength, True,
                            FFilter, @BytesRead, @FOverlap, nil) then
    begin
    WaitResult := WaitForMultipleObjects (2, @FEventArray, False, INFINITE);
    if (WaitResult = waitFileChange) then
      begin 
      InfoPointer := FBuffer;
      repeat
        NextOffset := InfoPointer.NextEntryOffset;
        ...
        PByte (InfoPointer) := PByte (InfoPointer) + NextOffset;
      until NextOffset = 0;
      end;
    end;
end;  

Filter is

FFilter :=  FILE_NOTIFY_CHANGE_FILE_NAME or
            FILE_NOTIFY_CHANGE_DIR_NAME or
            FILE_NOTIFY_CHANGE_SIZE or
            FILE_NOTIFY_CHANGE_LAST_WRITE;

and the directory handle is obtained like this:

FDirHandle := CreateFile (PChar (FDirectoryWatch.WatchedDirectory),
                          FILE_LIST_DIRECTORY or GENERIC_READ,
                          FILE_SHARE_READ or FILE_SHARE_WRITE or FILE_SHARE_DELETE,
                          nil, OPEN_EXISTING, FILE_FLAG_BACKUP_SEMANTICS or   
                          FILE_FLAG_OVERLAPPED, 0);         

When I delete multiple files I get only one event and NextOffset is 0! And when I delete a directory I get only one event for the directory. What if I want one event for each file in the directory?

Any help would be appreciated.

Photochromy

Answer 1

It seems to me that you are mixing the various ways to use ReadDirectoryChangesW(), you do both specify the FILE_FLAG_OVERLAPPED flag when opening the directory and provide a pointer to the lpOverlapped parameter, meaning you want to wait on the event in the structure and handle the asynchronous I/O; and at the same time you call ReadDirectoryChangesW() in a loop in a worker thread. I would first try again with lpOverlapped set to nil, as you have a dedicated thread and can use the synchronous mode.

In the documentation of the ReadDirectoryChangesW() API function the different ways to use it are described. Note that it is also possible that the buffer overflows, so change events can be lost anyway. Maybe you should rethink your strategy of relying solely on this function, comparing snapshots of directory contents could work as well.

Edit:

Your edited code looks better. In my tests however ReadDirectoryChangesW() did work as advertised, there were either several data entries in the returned buffer, or there were more than one buffer to process. This depends on timing, after hitting a breakpoint in Delphi I get several entries in one buffer.

For completeness I attach the test code, implemented using Delphi 5:

type
  TWatcherThread = class(TThread)
  private
    fChangeHandle: THandle;
    fDirHandle: THandle;
    fShutdownHandle: THandle;
  protected
    procedure Execute; override;
  public
    constructor Create(ADirectoryToWatch: string);
    destructor Destroy; override;

    procedure Shutdown;
  end;

constructor TWatcherThread.Create(ADirectoryToWatch: string);
const
  FILE_LIST_DIRECTORY = 1;
begin
  inherited Create(TRUE);
  fChangeHandle := CreateEvent(nil, FALSE, FALSE, nil);
  fDirHandle := CreateFile(PChar(ADirectoryToWatch),
    FILE_LIST_DIRECTORY or GENERIC_READ,
    FILE_SHARE_READ or FILE_SHARE_WRITE or FILE_SHARE_DELETE,
    nil, OPEN_EXISTING, FILE_FLAG_BACKUP_SEMANTICS or FILE_FLAG_OVERLAPPED, 0);
  fShutdownHandle := CreateEvent(nil, FALSE, FALSE, nil);
  Resume;
end;

destructor TWatcherThread.Destroy;
begin
  if fDirHandle <> INVALID_HANDLE_VALUE then
    CloseHandle(fDirHandle);
  if fChangeHandle <> 0 then
    CloseHandle(fChangeHandle);
  if fShutdownHandle <> 0 then
    CloseHandle(fShutdownHandle);
  inherited Destroy;
end;

procedure TWatcherThread.Execute;
type
  PFileNotifyInformation = ^TFileNotifyInformation;
  TFileNotifyInformation = record
    NextEntryOffset: DWORD;
    Action: DWORD;
    FileNameLength: DWORD;
    FileName: WideChar;
  end;
const
  BufferLength = 65536;
var
  Filter, BytesRead: DWORD;
  InfoPointer: PFileNotifyInformation;
  Offset, NextOffset: DWORD;
  Buffer: array[0..BufferLength - 1] of byte;
  Overlap: TOverlapped;
  Events: array[0..1] of THandle;
  WaitResult: DWORD;
  FileName, s: string;
begin
  if fDirHandle <> INVALID_HANDLE_VALUE then begin
    Filter := FILE_NOTIFY_CHANGE_FILE_NAME or FILE_NOTIFY_CHANGE_DIR_NAME
      or FILE_NOTIFY_CHANGE_SIZE or FILE_NOTIFY_CHANGE_LAST_WRITE;

    FillChar(Overlap, SizeOf(TOverlapped), 0);
    Overlap.hEvent := fChangeHandle;

    Events[0] := fChangeHandle;
    Events[1] := fShutdownHandle;

    while not Terminated do begin
      if ReadDirectoryChangesW (fDirHandle, @Buffer[0], BufferLength, TRUE,
        Filter, @BytesRead, @Overlap, nil)
      then begin
        WaitResult := WaitForMultipleObjects(2, @Events[0], FALSE, INFINITE);
        if WaitResult = WAIT_OBJECT_0 then begin
          InfoPointer := @Buffer[0];
          Offset := 0;
          repeat
            NextOffset := InfoPointer.NextEntryOffset;
            FileName := WideCharLenToString(@InfoPointer.FileName,
              InfoPointer.FileNameLength);
            SetLength(FileName, StrLen(PChar(FileName)));
            s := Format('[%d] Action: %.8xh, File: "%s"',
               [Offset, InfoPointer.Action, FileName]);
            OutputDebugString(PChar(s));
            PByte(InfoPointer) := PByte(DWORD(InfoPointer) + NextOffset);
            Offset := Offset + NextOffset;
          until NextOffset = 0;
        end;
      end;
    end;
  end;
end;

procedure TWatcherThread.Shutdown;
begin
  Terminate;
  if fShutdownHandle <> 0 then
    SetEvent(fShutdownHandle);
end;

////////////////////////////////////////////////////////////////////////////////

procedure TForm1.FormCreate(Sender: TObject);
begin
  fThread := TWatcherThread.Create('D:\Temp');
end;

procedure TForm1.FormDestroy(Sender: TObject);
begin
  if fThread <> nil then begin
    TWatcherThread(fThread).Shutdown;
    fThread.Free;
  end;
end;

Deleting a directory does indeed only return one change for it, nothing for the files contained in it. But it does make sense, as you are watching the handle of the parent directory only. If you need notifications for subdirectories you probably need to watch them as well.

Koerner

Sorry for my delayed comment. I used the synchronous version before (with the exact same problems) but switched to the asynchronous version because I didn't found a way to cleanly terminate the thread. I missed one important line in the example code though (the blocking call to WaitForMultipleObjects, which can either be termineted by a file change event or by a terminate event). I edit the question accordingly. (...) – Photochromy

What do you mean by snapshot? If you mean iterating over all files using FindFirst, FindNext: I previously used such an approach but I wanted to avoid (1) delayed change detection times when using large directories and (2) constant I/O load for the indexing thread slowing down all other I/O operations. – Photochromy

Agreed with your second comment, but as the MSDN documentation states you need to be prepared for overflows of the internal buffer, and in that case a full (re-)scan of the directory is necessary. – Koerner

Wow thanks a lot mghie for the effort of such a detailed answer! I have to compare this to my solution and check what I'm doing wrong. I'm probably gonna write some feedback later. – Photochromy

Answer 2

We've had the same problem with losing events, especially if a lot of changes happens at the same time, ie. 500 files are copied to the monitored directory.

In the end we found Cromis and use the Directory watch. We have never looked back again.

Creature

directory watch is good indeed. for 64 bit compatibility you need to replace getWindowLong with getWindowLongPtr (also for set...) – Dispersoid