Wait until file is unlocked in .NET

Question

What's the simplest way of blocking a thread until a file has been unlocked and is accessible for reading and renaming? For example, is there a WaitOnFile() somewhere in the .NET Framework?

I have a service that uses a FileSystemWatcher to look for files that are to be transmitted to an FTP site, but the file created event fires before the other process has finished writing the file.

The ideal solution would have a timeout period so the thread doesn't hang forever before giving up.

Edit: After trying out some of the solutions below, I ended up changing the system so that all files wrote to Path.GetTempFileName(), then performed a File.Move() to the final location. As soon as the FileSystemWatcher event fired, the file was already complete.

Idealism

Since the release of .NET 4.0, is there a better way to solve this problem? – Goble

Answer 1

This was the answer I gave on a related question:

    /// <summary>
    /// Blocks until the file is not locked any more.
    /// </summary>
    /// <param name="fullPath"></param>
    bool WaitForFile(string fullPath)
    {
        int numTries = 0;
        while (true)
        {
            ++numTries;
            try
            {
                // Attempt to open the file exclusively.
                using (FileStream fs = new FileStream(fullPath,
                    FileMode.Open, FileAccess.ReadWrite, 
                    FileShare.None, 100))
                {
                    fs.ReadByte();

                    // If we got this far the file is ready
                    break;
                }
            }
            catch (Exception ex)
            {
                Log.LogWarning(
                   "WaitForFile {0} failed to get an exclusive lock: {1}", 
                    fullPath, ex.ToString());

                if (numTries > 10)
                {
                    Log.LogWarning(
                        "WaitForFile {0} giving up after 10 tries", 
                        fullPath);
                    return false;
                }

                // Wait for the lock to be released
                System.Threading.Thread.Sleep(500);
            }
        }

        Log.LogTrace("WaitForFile {0} returning true after {1} tries",
            fullPath, numTries);
        return true;
    }

Carpentaria

I find this ugly but the only possible solution – Nog

Is this really going to work in the general case? if you open the file in a using() clause, the file is closed and unlocked when the using scope ends. If there is a second process using the same strategy as this (retry repeatedly), then after exit of WaitForFile(), there is a race condition regarding whether the file will be openable or not. No? – Surovy

Are you talking about 2 threads in the same app calling WaitForFile on the same file? Hmm, not sure, since I mostly use this to wait for other processes to let go of the file. I've had this code in production for a long time and it has worked well for me. Should be pretty simple to write a an app to test your theory. – Carpentaria

Bad idea! While the concept is right, a better solution will be to return the FileStream instead of a bool. If the file is locked again before the user got a chance to get his lock on the file - he will get an exception even if the function returned "false" – Zielsdorf

I also think this is ugly, and find Fero's answer below to do everything that this one does, elegantly, and in a fraction of the code. Anyone care to comment on any pitfalls in Fero's method? I am about to implement something similar myself. – Potage

where is Fero's method ? – Opponent

Nissim's comment is exactly what I was thinking too but if you're going to use that seek then don't forget to reset it to 0 after reading the byte. fs.Seek(0, SeekOrigin.Begin); – Kellene

try catch, is a resource consuming process, if we somehow can avoid it, its best, as we write such a code, it mean we expect that this happen often, and it mean lots of exceptions, and it lead to lot of CPU overheat. – Mohandas

is there a way to convert to using a waithandle, instead of the thread.sleep? – Maracanda

wouldnt buffer size 1 make more sense? – Vigue

Answer 2

Starting from Eric's answer, I included some improvements to make the code far more compact and reusable. Hope it's useful.

FileStream WaitForFile (string fullPath, FileMode mode, FileAccess access, FileShare share)
{
    for (int numTries = 0; numTries < 10; numTries++) {
        FileStream fs = null;
        try {
            fs = new FileStream (fullPath, mode, access, share);
            return fs;
        }
        catch (IOException) {
            if (fs != null) {
                fs.Dispose ();
            }
            Thread.Sleep (50);
        }
    }

    return null;
}

Greiner

I've come from the future to say that this code still works like a charm. Thanks. – Kaiserism

Works very good. But you aren't closing the stream so you may have problems acessing those files later (if you plan to wipe them after reading, for example). – Martelle

@PabloCosta The caller should probably just wrap it in a using block. Of course you cannot just forget about the stream. – Greiner

@PabloCosta Exactly! It can't close it, because if it did, another thread might race in and open it, defeating the purpose. This implementation is correct because it keeps it open! Let the caller worry about that, it is safe to using on a null, just check for null inside the using block. – Activator

"FileStream fs = null;" should be declared outside the try but inside the for. Then assign and use fs inside the try. The catch block should do "if (fs != null) fs.Dispose();" (or just fs?.Dispose() in C#6) to ensure the FileStream that isn't being returned is cleaned up properly. – Bracteole

@BillMenees Thank you, well spotted. Fixed! – Greiner

Is it really necessary to read a byte? In my experience if you have opened the file for read access, you have it, don't have to test for it. Although with the design here you aren't forcing exclusive access so it's even possible you may be able to read the first byte, but no others (byte level locking). From the original question you are likely to open with read-only share level, so no other process can lock or modify the file. At any rate, I feel the fs.ReadByte() is either a complete waste, or not enough, depending on usage. – Anticipative

@Anticipative Good point. I forgot how I tested this back then. To add, it would also be strange to read a byte if opening in write-only mode. – Greiner

Right now I can't seem to to be able to come up with a scenario where the byte reading is really useful, so I removed it. If other find problems with it, please let me know. – Greiner

User which circumstance can fs be not null in the catch block? If the FileStream constructor throws, the variable will not be assigned a value, and there is nothing else inside the try that can throw an IOException. To me it seems like it should be ok to just do return new FileStream(...). – Cocoa

8 and a half years later this code still rocks! Thanks @Greiner :) – Daffy

Answer 3

Here is a generic code to do this, independant from the file operation itself. This is an example on how to use it:

WrapSharingViolations(() => File.Delete(myFile));

or

WrapSharingViolations(() => File.Copy(mySourceFile, myDestFile));

You can also define the retry count, and the wait time between retries.

NOTE: Unfortunately, the underlying Win32 error (ERROR_SHARING_VIOLATION) is not exposed with .NET, so I have added a small hack function (IsSharingViolation) based on reflection mechanisms to check this.

    /// <summary>
    /// Wraps sharing violations that could occur on a file IO operation.
    /// </summary>
    /// <param name="action">The action to execute. May not be null.</param>
    public static void WrapSharingViolations(WrapSharingViolationsCallback action)
    {
        WrapSharingViolations(action, null, 10, 100);
    }

    /// <summary>
    /// Wraps sharing violations that could occur on a file IO operation.
    /// </summary>
    /// <param name="action">The action to execute. May not be null.</param>
    /// <param name="exceptionsCallback">The exceptions callback. May be null.</param>
    /// <param name="retryCount">The retry count.</param>
    /// <param name="waitTime">The wait time in milliseconds.</param>
    public static void WrapSharingViolations(WrapSharingViolationsCallback action, WrapSharingViolationsExceptionsCallback exceptionsCallback, int retryCount, int waitTime)
    {
        if (action == null)
            throw new ArgumentNullException("action");

        for (int i = 0; i < retryCount; i++)
        {
            try
            {
                action();
                return;
            }
            catch (IOException ioe)
            {
                if ((IsSharingViolation(ioe)) && (i < (retryCount - 1)))
                {
                    bool wait = true;
                    if (exceptionsCallback != null)
                    {
                        wait = exceptionsCallback(ioe, i, retryCount, waitTime);
                    }
                    if (wait)
                    {
                        System.Threading.Thread.Sleep(waitTime);
                    }
                }
                else
                {
                    throw;
                }
            }
        }
    }

    /// <summary>
    /// Defines a sharing violation wrapper delegate.
    /// </summary>
    public delegate void WrapSharingViolationsCallback();

    /// <summary>
    /// Defines a sharing violation wrapper delegate for handling exception.
    /// </summary>
    public delegate bool WrapSharingViolationsExceptionsCallback(IOException ioe, int retry, int retryCount, int waitTime);

    /// <summary>
    /// Determines whether the specified exception is a sharing violation exception.
    /// </summary>
    /// <param name="exception">The exception. May not be null.</param>
    /// <returns>
    ///     <c>true</c> if the specified exception is a sharing violation exception; otherwise, <c>false</c>.
    /// </returns>
    public static bool IsSharingViolation(IOException exception)
    {
        if (exception == null)
            throw new ArgumentNullException("exception");

        int hr = GetHResult(exception, 0);
        return (hr == -2147024864); // 0x80070020 ERROR_SHARING_VIOLATION

    }

    /// <summary>
    /// Gets the HRESULT of the specified exception.
    /// </summary>
    /// <param name="exception">The exception to test. May not be null.</param>
    /// <param name="defaultValue">The default value in case of an error.</param>
    /// <returns>The HRESULT value.</returns>
    public static int GetHResult(IOException exception, int defaultValue)
    {
        if (exception == null)
            throw new ArgumentNullException("exception");

        try
        {
            const string name = "HResult";
            PropertyInfo pi = exception.GetType().GetProperty(name, BindingFlags.NonPublic | BindingFlags.Instance); // CLR2
            if (pi == null)
            {
                pi = exception.GetType().GetProperty(name, BindingFlags.Public | BindingFlags.Instance); // CLR4
            }
            if (pi != null)
                return (int)pi.GetValue(exception, null);
        }
        catch
        {
        }
        return defaultValue;
    }

Persson

They could really have provided a SharingViolationException. In fact, they still can, backwards-compatibly, as long as it descends from IOException. And they really, really should. – Incus

Marshal.GetHRForException msdn.microsoft.com/en-us/library/… – Morrissette

In .NET Framework 4.5, .NET Standard, and .NET Core, HResult is a public property on the Exception class. Reflection is no longer needed for this. From MSDN: Starting with the .NET Framework 4.5, the HResult property's setter is protected, whereas its getter is public. In previous versions of the .NET Framework, both getter and setter are protected. – Outlet

Answer 4

I threw together a helper class for these sorts of things. It will work if you have control over everything that would access the file. If you're expecting contention from a bunch of other things, then this is pretty worthless.

using System;
using System.IO;
using System.Threading;

/// <summary>
/// This is a wrapper aroung a FileStream.  While it is not a Stream itself, it can be cast to
/// one (keep in mind that this might throw an exception).
/// </summary>
public class SafeFileStream: IDisposable
{
    #region Private Members
    private Mutex m_mutex;
    private Stream m_stream;
    private string m_path;
    private FileMode m_fileMode;
    private FileAccess m_fileAccess;
    private FileShare m_fileShare;
    #endregion//Private Members

    #region Constructors
    public SafeFileStream(string path, FileMode mode, FileAccess access, FileShare share)
    {
        m_mutex = new Mutex(false, String.Format("Global\\{0}", path.Replace('\\', '/')));
        m_path = path;
        m_fileMode = mode;
        m_fileAccess = access;
        m_fileShare = share;
    }
    #endregion//Constructors

    #region Properties
    public Stream UnderlyingStream
    {
        get
        {
            if (!IsOpen)
                throw new InvalidOperationException("The underlying stream does not exist - try opening this stream.");
            return m_stream;
        }
    }

    public bool IsOpen
    {
        get { return m_stream != null; }
    }
    #endregion//Properties

    #region Functions
    /// <summary>
    /// Opens the stream when it is not locked.  If the file is locked, then
    /// </summary>
    public void Open()
    {
        if (m_stream != null)
            throw new InvalidOperationException(SafeFileResources.FileOpenExceptionMessage);
        m_mutex.WaitOne();
        m_stream = File.Open(m_path, m_fileMode, m_fileAccess, m_fileShare);
    }

    public bool TryOpen(TimeSpan span)
    {
        if (m_stream != null)
            throw new InvalidOperationException(SafeFileResources.FileOpenExceptionMessage);
        if (m_mutex.WaitOne(span))
        {
            m_stream = File.Open(m_path, m_fileMode, m_fileAccess, m_fileShare);
            return true;
        }
        else
            return false;
    }

    public void Close()
    {
        if (m_stream != null)
        {
            m_stream.Close();
            m_stream = null;
            m_mutex.ReleaseMutex();
        }
    }

    public void Dispose()
    {
        Close();
        GC.SuppressFinalize(this);
    }

    public static explicit operator Stream(SafeFileStream sfs)
    {
        return sfs.UnderlyingStream;
    }
    #endregion//Functions
}

It works using a named mutex. Those wishing to access the file attempt to acquire control of the named mutex, which shares the name of the file (with the '\'s turned into '/'s). You can either use Open(), which will stall until the mutex is accessible or you can use TryOpen(TimeSpan), which tries to acquire the mutex for the given duration and returns false if it cannot acquire within the time span. This should most likely be used inside a using block, to ensure that locks are released properly, and the stream (if open) will be properly disposed when this object is disposed.

I did a quick test with ~20 things to do various reads/writes of the file and saw no corruption. Obviously it's not very advanced, but it should work for the majority of simple cases.

Genovevagenre

Answer 5

For this particular application directly observing the file will inevitably lead to a hard to trace bug, especially when the file size increases. Here are two different strategies that will work.

• Ftp two files but only watch one. For example send the files important.txt and important.finish. Only watch for the finish file but process the txt.
• FTP one file but rename it when done. For example send important.wait and have the sender rename it to important.txt when finished.

Good luck!

Satang

That is the opposite of automatic. That's like manually getting the file, with more steps. – Blouin

Answer 6

One of the techniques I used some time back was to write my own function. Basically catch the exception and retry using a timer which you can fire for a specified duration. If there is a better way, please share.

Grefe

Answer 7

From MSDN:

The OnCreated event is raised as soon as a file is created. If a file is being copied or transferred into a watched directory, the OnCreated event will be raised immediately, followed by one or more OnChanged events.

Your FileSystemWatcher could be modified so that it doesn't do its read/rename during the "OnCreated" event, but rather:

1. Spanws a thread that polls the file status until it is not locked (using a FileInfo object)
2. Calls back into the service to process the file as soon as it determines the file is no longer locked and is ready to go

Germano

Spawning the thread of the filesystemwatcher can lead the underlying buffer to overflow, thus missing a lot of changed files. A better approach will be to create a consumer/producer queue. – Zielsdorf

Answer 8

In most cases simple approach like @harpo suggested will work. You can develop more sophisticated code using this approach:

• Find all opened handles for selected file using SystemHandleInformation\SystemProcessInformation
• Subclass WaitHandle class to gain access to it's internal handle
• Pass found handles wrapped in subclassed WaitHandle to WaitHandle.WaitAny method

Nickolasnickolaus

Answer 9

Ad to transfer process trigger file SameNameASTrasferedFile.trg that is created after file transmission is completed.

Then setup FileSystemWatcher that will fire event only on *.trg file.

Com

Answer 10

A possible solution would be, to combine a filesystemwatcher with some polling,

get Notified for every Change on a File, and when getting notified check if it is locked as stated in the currently accepted answer: https://mcmap.net/q/176664/-wait-until-file-is-unlocked-in-net The code for opening the filestream is copied from the answer and slightly modified:

public static void CheckFileLock(string directory, string filename, Func<Task> callBack)
{
    var watcher = new FileSystemWatcher(directory, filename);
    FileSystemEventHandler check = 
        async (sender, eArgs) =>
    {
        string fullPath = Path.Combine(directory, filename);
        try
        {
            // Attempt to open the file exclusively.
            using (FileStream fs = new FileStream(fullPath,
                    FileMode.Open, FileAccess.ReadWrite,
                    FileShare.None, 100))
            {
                fs.ReadByte();
                watcher.EnableRaisingEvents = false;
                // If we got this far the file is ready
            }
            watcher.Dispose();
            await callBack();
        }
        catch (IOException) { }
    };
    watcher.NotifyFilter = NotifyFilters.LastWrite;
    watcher.IncludeSubdirectories = false;
    watcher.EnableRaisingEvents = true;
    //Attach the checking to the changed method, 
    //on every change it gets checked once
    watcher.Changed += check;
    //Initially do a check for the case it is already released
    check(null, null);
}

With this way you can Check for a file if its locked and get notified when its closed over the specified callback, this way you avoid the overly aggressive polling and only do the work when it may be actually be closed

Obstetric

Answer 11

I don't know what you're using to determine the file's lock status, but something like this should do it.

while (true)
{
    try {
        stream = File.Open( fileName, fileMode );
        break;
    }
    catch( FileIOException ) {

        // check whether it's a lock problem

        Thread.Sleep( 100 );
    }
}

Burch

A bit late, but when the file is locked somehow you'll never exit your loop. You should add a counter (see 1st answer). – Mealie

Answer 12

Here is a similar answer to the above except I added a check to see if the file exists.

bool WaitForFile(string fullPath)
        {
            int numTries = 0;
            while (true)
            {
                //need to add this line to prevent infinite loop
                if (!File.Exists(fullPath))
                {
                    _logger.LogInformation("WaitForFile {0} returning true - file does not exist", fullPath);
                    break;
                }
                ++numTries;
                try
                {
                    // Attempt to open the file exclusively.
                    using (FileStream fs = new FileStream(fullPath,
                        FileMode.Open, FileAccess.ReadWrite,
                        FileShare.None, 100))
                    {
                        fs.ReadByte();

                        // If we got this far the file is ready
                        break;
                    }
                }
                catch (Exception ex)
                {
                    _logger.LogInformation(
                       "WaitForFile {0} failed to get an exclusive lock: {1}",
                        fullPath, ex.ToString());

                    if (numTries > 10)
                    {
                        _logger.LogInformation(
                            "WaitForFile {0} giving up after 10 tries",
                            fullPath);
                        return false;
                    }

                    // Wait for the lock to be released
                    System.Threading.Thread.Sleep(500);
                }
            }

            _logger.LogInformation("WaitForFile {0} returning true after {1} tries",
                fullPath, numTries);
            return true;
        }

Steradian

Answer 13

I do it the same way as Gulzar, just keep trying with a loop.

In fact I don't even bother with the file system watcher. Polling a network drive for new files once a minute is cheap.

Ptolemaeus

It may be cheap but once a minute is too long for lots of applications. Real time monitoring is essential sometimes. Instead of you having to implement something that will be listening for Filesystem messages in C#(not the most convenient language for these things) you use FSW. – Maddi

Answer 14

Simply use the Changed event with the NotifyFilter NotifyFilters.LastWrite:

var watcher = new FileSystemWatcher {
      Path = @"c:\temp\test",
      Filter = "*.xml",
      NotifyFilter = NotifyFilters.LastWrite
};
watcher.Changed += watcher_Changed; 
watcher.EnableRaisingEvents = true;

Isma

FileSystemWatcher doesn't only notify when a file is done being written to. It will often notify you several times for a "single" logical write, and if you try to open the file after receiving the first notification you will get an exception. – Destiny

Answer 15

I ran into a similar issue when adding an outlook attachment. "Using" saved the day.

string fileName = MessagingBLL.BuildPropertyAttachmentFileName(currProp);

                //create a temporary file to send as the attachment
                string pathString = Path.Combine(Path.GetTempPath(), fileName);

                //dirty trick to make sure locks are released on the file.
                using (System.IO.File.Create(pathString)) { }

                mailItem.Subject = MessagingBLL.PropertyAttachmentSubject;
                mailItem.Attachments.Add(pathString, Outlook.OlAttachmentType.olByValue, Type.Missing, Type.Missing);

Dvinsk

Answer 16

How about this as an option:

private void WaitOnFile(string fileName)
{
    FileInfo fileInfo = new FileInfo(fileName);
    for (long size = -1; size != fileInfo.Length; fileInfo.Refresh())
    {
        size = fileInfo.Length;
        System.Threading.Thread.Sleep(1000);
    }
}

Of course if the filesize is preallocated on the create you'd get a false positive.

Faience

If the process writing to the file pauses for more than a second, or buffers in memory for more than a second, then you will get another false positive. I don't think this is a good solution under any circumstance. – Idealism