Asynchronous ReadDirectoryChangesW - GetQueuedCompletionStatus always times out

Question

Exactly as it sounds, I'm attempting asynchronous ReadDirectoryChangesW with IO Completion and it isn't working, specifically, GetLastError repeatedly returns 258 (GetQueuedCompletionStatus timeout).

I have structs:

typedef struct dirinfo_struct
{
    HANDLE hDirFH;           // directory handle
    OVERLAPPED Overlapped;   // overlapped storage
    int len_buffer;          // buffer length
    wchar_t* buffer;         // buffer itself
    wchar_t* directory_name; // target name
} dirinfo_t;

typedef struct dirmon_struct
{
    HANDLE hDirOPPort;       // handle to the IO port.
    dirinfo_t* dirinfo;      // pointer to the struct above.
} dirmon_t;

for storing the relevant information. This is initialised:

dirinfo_t* t = malloc(1*sizeof(dirinfo_t));
dirmon_t* d = malloc(1*sizeof(dirmon_t));
dirinfo_init(t); // does t->buffer = malloc(8192*sizeof(wchar_t));

Then I create my Directory Handle and com port:

t->hDirFH = CreateFile(L"C:\\test",
                        FILE_LIST_DIRECTORY,
                        FILE_SHARE_READ|FILE_SHARE_WRITE,
                        NULL,
                        OPEN_EXISTING,
                        FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OVERLAPPED,
                        NULL); 
d->dirinfo = t;
d->hDirOPPort = CreateIoCompletionPort(d->dirinfo->hDirFH, 
                                       NULL,       
                                       (ULONG_PTR)(d->dirinfo), 
                                       1); 

Then I pass this information via d to a new thread. Now on said new thread I have:

bResultQ = GetQueuedCompletionStatus(d->hDirOPPort, lpBytes, 
                                     (ULONG_PTR*)d->dirinfo,    
                                     lpOverlapped, 1000);

if ( bResultQ )
{
    bResultR = ReadDirectoryChangesW(d->dirinfo->hDirFH, 
                                     (void*)d->dirinfo->buffer, 
                                     8192, TRUE,
                                     FILE_NOTIFY_CHANGE_FILE_NAME | 
                                     FILE_NOTIFY_CHANGE_DIR_NAME |
                                     FILE_NOTIFY_CHANGE_ATTRIBUTES | 
                                     FILE_NOTIFY_CHANGE_SIZE |
                                     FILE_NOTIFY_CHANGE_LAST_WRITE | 
                                     FILE_NOTIFY_CHANGE_LAST_ACCESS | 
                                     FILE_NOTIFY_CHANGE_CREATION | 
                                     FILE_NOTIFY_CHANGE_SECURITY,
                                     lpReadDirBytes,
                                     &d->dirinfo->Overlapped,
                                     NULL );
} 
else
{
    printf("GetQueuedCompletionStatus(): Failed, ");
    errorcode = GetLastError();
    printf("Error Code %d\n", errorcode);
    Sleep(500);
}

So I set this off running and I merrily get timeouts (258 errors) as I should since the directory hasn't changed. However, even if I alter the directory, I'm still getting error messages; in other words those alterations are not being picked up. Which leads me to believe I've got this set up incorrectly.

Any ideas?

Caveats:

• Ironically, this will eventually be converted to Python via pywin32. However, until I understand how this is supposed to work in C, I'm not going there.
• Synchronous ReadDirectoryChangesW is not an option. If no events are fired, I need the thread this is on to timeout so it can check if it should still be running.
• I am writing in C specifically, not C++.
• FindFirstChangeNotification etc not an option either - I don't want to continually be comparing directory listings to work out what has changed.

Other notes:

• The Directory exists, that handle is not NULL. Likewise for the comport handle.
• Everything's being passed to the thread

I've taken a look at CDirectoryChangeWatcher from code project, but use of C++ and many more threads aside, I can't see what I'm doing differently. Feel free to point it out if I'm missing something though!

Output, if it helps, is basically on repeat, no matter how much I alter the directory in question.

GetQueuedCompletionStatus(): Failed, Error Code 258

Pansir

Getting this to work well is non-trivial (to put it mildly). If you haven't yet, I'd advise reading Jim Beveridge's blog on it: qualapps.blogspot.com/2010/05/…. Warning: his sample code is in C++, but at least it's better than most. Warning2: I'm reasonably certain that no matter what you do, ReadDirectoryChangesW will never really work entirely correctly. I usually advise using change journals instead. msdn.microsoft.com/en-us/library/aa363798.aspx. They're not trivial either, but at least they do work. – Verdin

@Jeffy thanks, I had a look at his source, but he doesn't (as far as I can see) use GetQueue... or CreateIo... His readme implies more files than are actually there so I'm thinking he's removed them. I am slowly beginning to come to this realisation... unfortunately change journals require admin privileges which we may well not have. I could always set off a windows service to handle that I suppose. – Pansir

Answer 1

I realise posting walls of code is generally considered horrendous, but here's how I got this working:

New structs:

BOOL runthread;

typedef struct overlapped_struct
{
    OVERLAPPED overlapped;
    wchar_t* buffer;
} overlapped_t;

typedef struct dirinfo_struct
{

    HANDLE hDirOPPort;
    HANDLE hDirFH;
    overlapped_t* o;
    int len_buffer;
    wchar_t* buffer;
    wchar_t* directory_name;
    ULONG_PTR CompletionKey;
} dirinfo_t;

int somekey = 1;

Allocation methods:

void dirinfo_init(dirinfo_t* t)
{
    t->buffer = malloc(16777216*sizeof(wchar_t));
    t->len_buffer = 16777216;
    t->o = calloc(1, sizeof(overlapped_t));
    t->o->buffer = calloc(16777216, sizeof(wchar_t));
    memset(t->o->buffer, 0, 16777216);
    memset(t->o, 0, sizeof(OVERLAPPED));
}

void dirinfo_free(dirinfo_t* t)
{
    free(t->buffer);
    free(t->o->buffer);
    free(t->o);
    free(t);
}

The important stuff from main() does this:

dirinfo_t* d = malloc(1*sizeof(dirinfo_t));
d->CompletionKey = (ULONG_PTR)&somekey;
dirinfo_init(d);

/* set up */
runthread = TRUE;
d->hDirFH = CreateFile(L"C:\\hydratest",
                FILE_LIST_DIRECTORY,
                FILE_SHARE_READ|FILE_SHARE_WRITE,
                NULL,
                OPEN_EXISTING,
                FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OVERLAPPED,
                NULL); 

d->hDirOPPort = CreateIoCompletionPort(d->hDirFH, NULL, 
                      (ULONG_PTR)d->CompletionKey, 1);  

Then finally my waiting thread. Here's the key: I'm not passing an overlapped structure in. I'm passing in a structure containing an OVERLAPPED plus a fair amount of wchar_t based storage. For reasons I don't fully understand, this works. Edit see this answer. I believe the data region here acts as the overlapped buffer.

DWORD WINAPI WaitingThread(void* args)
{
    DWORD errorcode = 0;    // an error code
    BOOL bResultQ = FALSE;  // obvios=us
    BOOL bResultR = FALSE;
    DWORD NumBytes = 0; 
    FILE_NOTIFY_INFORMATION* pInfo = NULL; // the data incoming is a pointer
                                           // to this struct.
    int i = 0;
    dirinfo_t* d = (dirinfo_t*) args;      // rescue struct from thread arg.

Then we get onto the main thread itself. Trial and error suggests you're supposed to call both ReadDirectoryW AND GetQueueCompletionStatus. I think what this means is that we're supposed to not touch the buffer from ReadDirectoryChangeW **unless* we're told we can by GetQueue. Corrections on that hypothesis welcome however.

    while ( runthread )
    {
        bResultR = ReadDirectoryChangesW(d->hDirFH, (void*)d->buffer, 
                                          16777216, TRUE,
               FILE_NOTIFY_CHANGE_FILE_NAME  | FILE_NOTIFY_CHANGE_DIR_NAME |
               FILE_NOTIFY_CHANGE_ATTRIBUTES | FILE_NOTIFY_CHANGE_SIZE |
               FILE_NOTIFY_CHANGE_LAST_WRITE | FILE_NOTIFY_CHANGE_LAST_ACCESS | 
               FILE_NOTIFY_CHANGE_CREATION   | FILE_NOTIFY_CHANGE_SECURITY,
                                          NULL,
                                          &d->o->overlapped,
                                          NULL );
        bResultQ = GetQueuedCompletionStatus(d->hDirOPPort, 
                                             &NumBytes, &(d->CompletionKey), 
                                             (LPOVERLAPPED*)(d->o), 1000);

So, now we've called those functions, we then test that they both returned true. big ugly warning if you've got your parameters set up right bResultR always returns true, or so it seems to me. bResultQ however varies depending on whether new data is on the port.

        if ( bResultQ && bResultR )
        {

So here we cast that buffer from ReadDirectoryChangesW and access the info from the struct.

            wprintf(L"\n");
            pInfo = (FILE_NOTIFY_INFORMATION*) d->buffer;
            wprintf(L"File %s", pInfo->FileName);
            wprintf(L" changes %d\n", pInfo->Action);
            memset(d->buffer, 0, 16777216);
        }

Otherwise, and thanks to Tony for this, you can safely ignore WAIT_TIMEOUT errors, but anything else probably means you're in trouble.

        else
        {
            errorcode = GetLastError();

            if ( errorcode == WAIT_TIMEOUT )
            {
                printf("GetQueuedCompletionStatus(): Timeout\n");
            }
            else
            {
                printf("GetQueuedCompletionStatus(): Failed\n");
                printf("Error Code %d\n", errorcode);
            }
            Sleep(500);
        }
    }   

    return 0;
}

And that completes what I think is a working example.

Some notes:

• I've set the buffer size to be huge. I noticed copying 100 files or so that the buffer ran out of space set to 8192 and missed off an item or two, here and there. So I don't expect this will always pick up everything. My solution would be to say every 100 events, verify the file tree is what you think it is if using this method. An infinitely better solution, however, to constantly enumerating the potentially large tree.

Pansir

hmmm interesting. Definitely an adventure this IO stuff. :) Thanks for posting. – Rainy

@Ninefingers: Hi, am working on similar project(but in c++) and my problem is to separate watching directory thread from my application thread , if you don't mind let me know when you called your WaitingThread I append it to your main code but I still have the same issue , thanks in advance. – Muslim

@Antony please give explain: malloc(16777216*sizeof(wchar_t)) as I guess 16777216 = 8192 * 2048 => what is this 2048? Thank you! – Carolyncarolyne

Hey Antony, is there any example of above code in C#? – Reconciliatory

@NadeemUllah Hello, not that I am aware of - however I don't know C# very well at all so there could well be. – Pansir

Answer 2

Note: To catch errors from GetQueuedCompletionStatus properly, as it is difficult to determine that this function actually returned, should be done as follows:

EXAMPLE:

DWORD dwNumBytes;
ULONG_PTR CompletionKey;
OVERLAPPED* pOverlapped;

//hIOCP is initialized somewhere else in the program
BOOL bOK = GetQueuedCompletionStatus(hIOCP, &dwNumBytes, &CompletionKey, &pOverlapped, 1000);

DWORD dwError = GetLastError();

if(bOK)
{
// Process a successfully completed I/O event.
}
else
{
  if (pOverlapped != NULL)
  {
    // Process a failed completed I/O request
    //dwError contains the reason for failure
  }
  else {
      if (dwError == WAIT_TIMEOUT)
      {
         //Time-out while waiting for completed I/O entry.
      }
      else {
          //Bad call to GetQueuedCompletionStatus
          //dwError contains the reason for the bad call.
      }
}

Example taken from the book (Windows via C/C++) Please try to implement this error handling in your code.

Also "... threads that call GetQueuedCompletionStatus are awakened in a last-in first-out (LIFO) fashion."

OVERLAPPED Structure:

When performing asynchronous device I/O, you must pass the address to an initialized OVERLAPPED structure via the pOverlapped parameter. The word "overlapped" in this context means that the time spent performing I/O requests overlaps the time your thread spends performing other tasks.

It's talking about the parameter when you call ReadFile or WriteFile, just as a note to the above, which requires this structure to be initialized.

It looks as follows:

typedef struct _OVERLAPPED {
  ULONG_PTR Internal;
  ULONG_PTR InternalHigh;
  union {
    struct {
      DWORD Offset;
      DWORD OffsetHigh;
    };
    PVOID  Pointer;
  };
  HANDLE    hEvent;
} OVERLAPPED, *LPOVERLAPPED;

NOTE: You are passing a pointer to a struct to your dwCompletionKey parameter of your CreateIoCompletionPort function. In the reference I am looking at they merely pass a constant (#define CK_FILE 1) to this. It does say you can pass whatever you like, as the OS doesn't care. Just wanted to point it out however.

Rainy

thanks a lot for this, I now know to expect a failure to actually potentially contain data...! I've modified my Overlapped struct to be ZeroMemory'd, I've pointed CompletionKey at a constant value, etc. Still, every time I get a null overlapped structure, false return value and a timeout error code. – Pansir

From MSDN: "If a call to GetQueuedCompletionStatus fails because the completion port handle associated with it is closed while the call is outstanding, the function returns FALSE, *lpOverlapped will be NULL, and GetLastError will return ERROR_ABANDONED_WAIT_0." – Rainy

What about NumBytes value, is that filled up when it returns? – Rainy

I might have it working. I missed changing the CreateIo.. completion key to the constant. Now I can see the data in the buffer, it's just a case of making the correct call to ReadDirectoryChangesW to get it out. – Pansir