FileStream and a FileSystemWatcher in C#, Weird Issue "process cannot access the file"

Question

I have this complicated code-base that is listening for FileCreated events on a certain folder. When the file is Created (which also includes moving a file to that folder), I want to read in that file and do something with it. It works for the first file, but throws an exception after for all other attempts. In Debug-mode (With VisualStudio) the error would be thrown, but if i just click "continue".. it would then work (without an error).

I have posted simplified code, which demonstrates the issue.

For example, you start the application, click the "start" button, and then "create a new text file"

The output is:

Working

If you then create a second file in the exact same manner, the output is:

Broken: The process cannot access the file 'C:\TestFolder\New Text Document (2).txt' because it is being used by another process.
Working, after breaking

After looking at my code, you will see that the above set of print-outs implies that first there was an "cannot access the file" exception thrown, but doing the same call in the catch-statement suddenly works.

This makes no sense to me, since the file is clearly not being used by anything else (i just created it).. and it works a second later anyhow....

Below is my code

XAML:

<Window x:Class="WpfApplication1.MainWindow"
        xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
        Title="MainWindow" >
    <StackPanel>
        <Button Click="Button_Click"  Content="Start"/>
    </StackPanel>
</Window>

Code Behind:

using System;
using System.Diagnostics;
using System.IO;
using System.Threading;
using System.Windows;
using System.Windows.Controls;
using System.Windows.Input;


namespace WpfApplication1
{ 
    public partial class MainWindow : Window
    {
        public MainWindow()
        {
            InitializeComponent();
        }

        private void Button_Click(object sender, RoutedEventArgs e)
        {
            test();
        }


        String Folder = @"C:\TestFolder";

        private void test()
        {
            FileSystemWatcher watch = new FileSystemWatcher(Folder);
            watch.Created += new FileSystemEventHandler(FileCreated);
            watch.EnableRaisingEvents = true;

            Process.Start(Folder);
        }

        private void FileCreated(object sender, FileSystemEventArgs fsEvent)
        {

            if (File.Exists(fsEvent.FullPath))
            {

                // Thread.Sleep(1000);// Sleeping for 1 second seems to prevent the error from happening...?
                // If i am debugging, and pause on exceptions... then it also suddenly works (similar to the Sleep above)
                try
                {

                    FileStream fs = new FileStream(fsEvent.FullPath, FileMode.Open); 
                    Console.WriteLine("Working");
                    fs.Close();
                }
                catch (IOException ex)
                {
                    Console.WriteLine("Broken: " + ex.Message);
                    try
                    {                        
                        FileStream fs = new FileStream(fsEvent.FullPath, FileMode.Open);
                        Console.WriteLine("Working, after breaking");
                        fs.Close();

                    }
                    catch(IOException ex2)
                    {                        
                        FileStream fs = new FileStream(fsEvent.FullPath, FileMode.Open);
                        Console.WriteLine("really broken: " + ex2.Message);
                        fs.Close();
                    }
                }


            }
        }
    }
}

Morril

Answer 1

I've seen the behavior you describe since .NET 1.0 and never bothered finding out why it happens. It seems like the OS or .NET sometimes(?) keep a lock on the file for a short time after you called close and dispose.

I made a workaround - or hack if you like - that has proven to be very robust for us. We're processing millions of files daily in a server farm, and all files detected by filewatchers pass through this method before they are handed over for further processing.

What it does is to place an exclusive lock on the file. If it fails it will optionally wait up to 10 seconds for the file to be closed before giving up.

    public static bool IsFileClosed(string filepath, bool wait)
    {
        bool        fileClosed = false;
        int         retries = 20;
        const int   delay = 500; // Max time spent here = retries*delay milliseconds

        if (!File.Exists(filepath))
            return false;

        do
        {
            try 
            {
                // Attempts to open then close the file in RW mode, denying other users to place any locks.
                FileStream fs = File.Open(filepath, FileMode.Open, FileAccess.ReadWrite, FileShare.None);
                fs.Close();
                fileClosed = true; // success
            }
            catch (IOException) {}

            if (!wait) break;

            retries --;

            if (!fileClosed)
                Thread.Sleep( delay );
        }
        while (!fileClosed && retries > 0);

        return fileClosed;
    }

Triiodomethane

+1 I might try this... my simple "sleep(1000)" seems to work fine right now. – Morril

Answer 2

Most likely what is happening here is that the FileCreated event is being raised and tries to process the file before is has been completely written to disk.

See Wait Until File Is Completely Written for a technique to avoid this problem.

Contented

+1. But any idea why it would work for the first file created, and not for other files afterwards? – Morril

Not sure... The answer would depend on what mechanism is being used to create the files. – Contented

Answer 3

Try disabling any antivirus/anti-malware software. Most are configured to scan files on create by default.

Hime

Very good point, but even with exception rules (disabling is seldom an option in production), I've seen these locks occur if you access the file right after it is closed. – Triiodomethane

Maybe, but it still doesn't explain why it works once, and then fails. – Morril

+1 for an interesting idea anyhow. (they also don't like us to turn off antivirus at work :P ) – Morril

nearly 8 years later, a coworker mentioned that they believe this is something being done too on all newly created files. No way to prove it, and probably no way to avoid it either. But just wanted to give you the shout-out. – Morril

I appreciate your commitment to this thread ;-) If you're still struggling with this, check if anything is replicating the filesystem or if the underlying storage is network backed – Hime

Answer 4

Ty to EventHorizon for the code above, worked quite well for my needs and seems to capture the essence of what was in the link Brendan shared.

A slight rework of this into an async returning a tuple (both for getting back the file lock status at the end of the do/while and getting ms passed to see what kind of exit occurred). The only thing I'm not comfortable with, and I'm not sure why this is the case, even dropping the delay down to 2ms I still haven't been able to trip the catch IOException condition (evidence of more than one do loop) so I don't have any direct visibility of the IO failure case, someone else might have larger files they can verify this with:

public async Task<(bool, int)> IsFileClosed(string filepath)
{
  bool fileClosed = false;
  int baseretries = 40;
  int retries = 40;
  const int delay = 250;

  if (!File.Exists(filepath))
    return (false,0);

  Task<bool> FileCheck = Task.Run(() =>
  {
    do
    {
      try
      {
        FileStream fs = File.Open(filepath, FileMode.Open, FileAccess.ReadWrite, FileShare.None);
        fs.Close();
        fileClosed = true;
      }
      catch (IOException) { }
      retries--;

      if (!fileClosed)
        Thread.Sleep(delay);
    }
    while (!fileClosed && retries > 0);
    return fileClosed;
  });

  fileClosed = await FileCheck;
  return (fileClosed, (baseretries - retries) * delay);
}

Waylon