c# FileSystemWatcher Created event not firing for all files created

Question

I want to insert file information into a database for all files created in a directory. My program will do that if I drop small numbers of files in the directory but it doesn't get them all if I drop larger amounts in the directory all at once (I didn't do a whole lot of testing but it only inserted about 200 names into my database when I tried to drop 800 files in the directory at once).

Here's my code:

static void Main(string[] args)
{
    // Create a new FileSystemWatcher and set its properties.
    FileSystemWatcher watcher = new FileSystemWatcher();
    watcher.Path = @"C:\dropDirectory";

    // Add event handlers.
    watcher.Created += new FileSystemEventHandler(OnChanged);

    // Begin watching.
    watcher.EnableRaisingEvents = true;

    while(DateTime.Now.Hour < 10);
}

private static void OnChanged(object source, FileSystemEventArgs e)
{
    string strInsert = "INSERT INTO  Files (Filename) VALUES ('" + e.Name + "')";
    string strConnection = "Server = server_name; Database = database_name; User Id = user_id; Password = password;";
    using (SqlConnection con = new SqlConnection(strConnection))
    {
        using (SqlCommand cmd = new SqlCommand(strInsert, con))
        {   
            con.Open();
            cmd.ExecuteScalar();
        }
    }
}

What changes do I need to make so my OnChanged method will be called for every file dropped in the destination directory regardless of the number of files dropped at once? Thanks in advance.

Nappie

Hi Gharbad, I know its been a while you posted this question . I have faced similar issue but in my case one of my customer is trying to copy 70 odd files but only around 20-30 events used to come to FSW . Customer PC RAM Size is 7GB . I have not set buffer size . so it takes default of 4KB . What was your PC RAM Size ? – Drought

@PraveenM I don't remember, that was at an old job on their machine – Nappie

Answer 1

Your problem is the buffer size, and its well known and documented issue

FileSystemWatcher uses ReadDirectoryChangesW WinApi call with a few relevant flags

When you first call ReadDirectoryChangesW, the system allocates a buffer to store change information. This buffer is associated with the directory handle until it is closed and its size does not change during its lifetime. Directory changes that occur between calls to this function are added to the buffer and then returned with the next call. If the buffer overflows, the entire contents of the buffer are discarded

The analogue in FileSystemWatcher is the FileSystemWatcher.InternalBufferSize property

Remarks You can set the buffer to 4 KB or larger, but it must not exceed 64 KB. If you try to set the InternalBufferSize property to less than 4096 bytes, your value is discarded and the InternalBufferSize property is set to 4096 bytes. For best performance, use a multiple of 4 KB on Intel-based computers.

The system notifies the component of file changes, and it stores those changes in a buffer the component creates and passes to the APIs. Each event can use up to 16 bytes of memory, not including the file name. If there are many changes in a short time, the buffer can overflow. This causes the component to lose track of changes in the directory, and it will only provide blanket notification. Increasing the size of the buffer can prevent missing file system change events. However, increasing buffer size is expensive, because it comes from non-paged memory that cannot be swapped out to disk, so keep the buffer as small as possible. To avoid a buffer overflow, use the NotifyFilter and IncludeSubdirectories properties to filter out unwanted change notifications.

If worse comes to worse you can use a mix of polling and tracking, it has helped me out of trouble a few times.

Antacid

Thanks for the answer, exactly what I needed. Accepted as answer and tried to upvote but my reputation is too low for it to show. Thanks again. – Nappie