FileSystemWatcher - is File ready to use

Question

When a file is being copied to the file watcher folder, how can I identify whether the file is completely copied and ready to use? Because I am getting multiple events during file copy. (The file is copied via another program using File.Copy.)

Lunchroom

Does this answer your question? A robust solution for FileSystemWatcher firing events multiple times – Diarrhea

Answer 1

When I ran into this problem, the best solution I came up with was to continually try to get an exclusive lock on the file; while the file is being written, the locking attempt will fail, essentially the method in this answer. Once the file isn't being written to any more, the lock will succeed.

Unfortunately, the only way to do that is to wrap a try/catch around opening the file, which makes me cringe - having to use try/catch is always painful. There just doesn't seem to be any way around that, though, so it's what I ended up using.

Modifying the code in that answer does the trick, so I ended up using something like this:

private void WaitForFile(FileInfo file)
{
    FileStream stream = null;
    bool FileReady = false;
    while(!FileReady)
    {
        try
        {
            using(stream = file.Open(FileMode.Open, FileAccess.ReadWrite, FileShare.None)) 
            { 
                FileReady = true; 
            }
        }
        catch (IOException)
        {
            //File isn't ready yet, so we need to keep on waiting until it is.
        }
        //We'll want to wait a bit between polls, if the file isn't ready.
        if(!FileReady) Thread.Sleep(1000);
    }
}

Anuska

You might want to add a count and quit trying after x failed attempts. – Yapon

This doesn't return the lock. So it could report TRUE and then you still throw an error on access because it got locked between when this method ends and the actual work begins. It is better to follow a similar pattern but with the actual code you are using. Why open it twice? Either have your WaitForOpen return the stream or just add the wait logic to your operation. – Diarrhea

Answer 2

Here is a method that will retry file access up to X number of times, with a Sleep between tries. If it never gets access, the application moves on:

private static bool GetIdleFile(string path)
{
    var fileIdle = false;
    const int MaximumAttemptsAllowed = 30;
    var attemptsMade = 0;

    while (!fileIdle && attemptsMade <= MaximumAttemptsAllowed)
    {
        try
        {
            using (File.Open(path, FileMode.Open, FileAccess.ReadWrite))
            {
                fileIdle = true;
            }
        }
        catch
        {
            attemptsMade++;
            Thread.Sleep(100);
        }
    }

    return fileIdle;
}

It can be used like this:

private void WatcherOnCreated(object sender, FileSystemEventArgs e)
{
    if (GetIdleFile(e.FullPath))
    {
        // Do something like...
        foreach (var line in File.ReadAllLines(e.FullPath))
        {
            // Do more...
        }
    }
}

Backpack

Answer 3

I had this problem when writing a file. I got events before the file was fully written and closed.

The solution is to use a temporary filename and rename the file once finished. Then watch for the file rename event instead of file creation or change event.

Trudytrue

This doesn't work if someone else is pushing the file to you (like over FTP or something like that) - you have to know the file is ready to use before you can rename it and say it's ready to use :) – Anuska

Answer 4

Note: this problem is not solvable in generic case. Without prior knowledge about file usage you can't know if other program(s) finished operation with the file.

In your particular case you should be able to figure out what operations File.Copy consist of.

Most likely destination file is locked during whole operation. In this case you should be able to simply try to open file and handle "sharing mode violation" exception.

You can also wait for some time... - very unreliable option, but if you know size range of files you may be able to have reasonable delay to let Copy to finish.

You can also "invent" some sort of transaction system - i.e. create another file like "destination_file_name.COPYLOCK" which program that copies file would create before copying "destination_file_name" and delete afterward.

Americanism

Answer 5

    private Stream ReadWhenAvailable(FileInfo finfo, TimeSpan? ts = null) => Task.Run(() =>
    {
        ts = ts == null ? new TimeSpan(long.MaxValue) : ts;
        var start = DateTime.Now;
        while (DateTime.Now - start < ts)
        {
            Thread.Sleep(200);
            try
            {
                return new FileStream(finfo.FullName, FileMode.Open);
            }
            catch { }
        }
        return null;
    })
    .Result;

...of course, you can modify aspects of this to suit your needs.

Jackinthebox

is ts the timeout? If so you should name it so it's more clear what you are doing. – Diarrhea

Answer 6

One possible solution (It worked in my case) is to use the Change event. You can log in the create event the name of the file just created and then catch the change event and verify if the file was just created. When I manipulated the file in the change event it didn't throw me the error "File is in use"

Herniotomy

Answer 7

If you are doing some sort of inter-process communication, as I do, you may want to consider this solution:

1. App A writes the file you are interested in, eg "Data.csv"
2. When done, app A writes a 2nd file, eg. "Data.confirmed"
3. In your C# app B make the FileWatcher listen to "*.confirmed" files. When you get this event you can safely read "Data.csv", as it is already completed by app A.
4. (Edit: inspired by commets) Delete the *.confirmed filed with app B when done processing the "Data.csv" file.

Satsuma

Already mentioned in existing answer – Szabo

@HereticMonkey this is not exactly the same with Alexei Levenkov's proposal. The idea is similar, but the implementation is different. – Snap

@TheodorZoulias I didn’t say it was exactly anything. But the idea is the same; use another file as a transaction mechanism. Alexei didn’t describe an implementation, just an outline of an idea, which is repeated in this answer. I didn’t say whether it was right or wrong. I think it would be nice to credit the guy in the answer, but that’s me. – Szabo

@HereticMonkey Alexei Levenkov uses the ".COPYLOCK" files as a transaction mechanism, while Xcessity uses the "*.confirmed" files as a notification mechanism. I think that Xcessity's approach is more robust because someone could visually inspect the folder, and be alerted that something has gone wrong by the presence of old ".confirmed" files in the folder. That's because the consumer is responsible for deleting the ".confirmed" files. Alexei Levenkov's approach is to have the producer create and then delete the ".COPYLOCK" files, leaving no visual trace in case the consumer malfunctions. – Snap

@TheodorZoulias I’m not really interested In debating the relative merits of the specific implementations of the core idea of “create another file when the file under observation is written”. Nor, to be honest, your feelings about my initial statement. It is a comment; a clarification of the answer’s worth in my opinion. Take it as that. – Szabo

Answer 8

I have solved this issue with two features:

1. Implement the MemoryCache pattern seen in this question: A robust solution for FileSystemWatcher firing events multiple times
2. Implement a try\catch loop with a timeout for access

You need to collect average copy times in your environment and set the memory cache timeout to be at least as long as the shortest lock time on a new file. This eliminates duplicates in your processing directive and allows some time for the copy to finish. You will have much better success on first try, which means less time spent in the try\catch loop.

Here is an example of the try\catch loop:

public static IEnumerable<string> GetFileLines(string theFile)
{
    DateTime startTime = DateTime.Now;
    TimeSpan timeOut = TimeSpan.FromSeconds(TimeoutSeconds);
    TimeSpan timePassed;
    do
    {
        try
        {
            return File.ReadLines(theFile);
        }
        catch (FileNotFoundException ex)
        {
            EventLog.WriteEntry(ProgramName, "File not found: " + theFile, EventLogEntryType.Warning, ex.HResult);
            return null;
        }
        catch (PathTooLongException ex)
        {
            EventLog.WriteEntry(ProgramName, "Path too long: " + theFile, EventLogEntryType.Warning, ex.HResult);
            return null;
        }
        catch (DirectoryNotFoundException ex)
        {
            EventLog.WriteEntry(ProgramName, "Directory not found: " + theFile, EventLogEntryType.Warning, ex.HResult);
            return null;
        }
        catch (Exception ex)
        {
            // We swallow all other exceptions here so we can try again
            EventLog.WriteEntry(ProgramName, ex.Message, EventLogEntryType.Warning, ex.HResult);
        }

        Task.Delay(777).Wait();
        timePassed = DateTime.Now.Subtract(startTime);
    }
    while (timePassed < timeOut);

    EventLog.WriteEntry(ProgramName, "Timeout after waiting " + timePassed.ToString() + " seconds to read " + theFile, EventLogEntryType.Warning, 258);
    return null;
}

Where TimeoutSeconds is a setting that you can put wherever you hold your settings. This can be tuned for your environment.

Diarrhea