Using Directory.Delete() and Directory.CreateDirectory() to overwrite a folder

Question

In my WebApi action method, I want to create/over-write a folder using this code:

string myDir = "...";
if(Directory.Exists(myDir)) 
{
    Directory.Delete(myDir, true);
}
Directory.CreateDirectory(myDir);

// 1 - Check the dir 
Debug.WriteLine("Double check if the Dir is created: " + Directory.Exists(myDir));

// Some other stuff here...

// 2 - Check the dir again
Debug.WriteLine("Check again if the Dir still exists: " + Directory.Exists(myDir));

Issue

Strangely, sometimes right after creating the directory, the directory does not exist!

Sometimes when checking the dir for the first time (where the number 1 is); Directory.Exist() returns true, other times false. Same happens when checking the dir for the second time (where the number 2 is).

Notes

• None of this part of code throw any exception.
• Only can reproduce this when publishing the website on server. (Windows server 2008)
• Happens when accessing the same folder.

Questions

• Is this a concurrency issue race condition?
• Doesn't WebApi or the Operating System handle the concurrency?
• Is this the correct way to overwrite a folder?
• Should I lock files manually when we have many API requests to the same file?

Or in General:

• What's the reason for this strange behavior?

UPDATE:

• Using DirectoryInfo and Refresh() instead of Directory does not solve the problem.

• Only happens when the recursive option of Delete is true. (and the directory is not empty).

Abhor

Is the directory a fileshare (i.e. is it the disk on a machine other than the web server)? – Seften

@Seften No it's on the same machine as the web server. – Abhor

Answer 1

Many filesystem operations are not synchonous on some filesystems (in case of windows - NTFS). Take for example RemoveDirectory call (which is called by Directory.DeleteDirectory at some point):

The RemoveDirectory function marks a directory for deletion on close. Therefore, the directory is not removed until the last handle to the directory is closed.

As you see, it will not really delete directory until all handles to it are closed, but Directory.DeleteDirectory will complete fine. In your case that is also most likely such concurrency problem - directory is not really created while you executing Directory.Exists.

So, just periodically check what you need and don't consider filesystem calls in .NET to be synchronous. You can also use FileSystemWatcher in some cases to avoid polling.

EDIT: I was thinking how to reproduce it, and here is the code:

internal class Program {
    private static void Main(string[] args) {
        const string path = "G:\\test_dir";
        while (true) {         
            if (Directory.Exists(path))
                Directory.Delete(path);       
            Directory.CreateDirectory(path);   
            if (!Directory.Exists(path))
                throw new Exception("Confirmed");                 
        }            
    }        
}

You see that if all filesystem calls were synchronous (in .NET), this code should run without problem. Now, before running that code, create empty directory at specified path (preferrably don't use SSD for that) and open it with windows explorer. Now run the code. For me it either throws Confirmed (which exactly reproduces your issue) or throws on Directory.Delete saying that directory does not exist (almost the same case). It does it 100% of the time for me.

Here is another code which when running on my machine confirms that it's certainly possible for File.Exists to return true directly after File.Delete call:

internal class Program {
    private static void Main(string[] args) {
        while (true) {
            const string path = @"G:\test_dir\test.txt";
            if (File.Exists(path))
                File.Delete(path);
            if (File.Exists(path))
                throw new Exception("Confirmed");
            File.Create(path).Dispose();
        }
    }        
 }

To do this, I opened G:\test_dir folder and during execution of this code tried to open constantly appearing and disappearing test.txt file. After couple of tries, Confirmed exception was thrown (while I didn't create or delete that file, and after exception is thrown, it's not present on filesystem already). So race conditions are possible in multiple cases and my answer is correct one.

Warhead

The only time a delete is not done when Delete returns is when there is a handle open that has file share delete. This is extremely rare. This is opt in behavior that almost no code uses. Otherwise, deletes are fully synchronous. I cannot reproduce this behavior either. I find this answer to be highly misleading. – Sugary

Likely, his problem is based on a race. Not on FILE_SHARE_DELETE. I don't know why this answer is accepted since it does not provide a solution either. It's multistage process, and windows cannot even know when it will be completed, so it will return from File.Delete before that. ... - directory is not really created while you executing Directory.Exists. both sentences are false. – Sugary

@Sugary File.Delete is just example - author does not delete files. You can try to run sample code above, which reproduces author's problem exactly. – Warhead

As I said I cannot reproduce it with this exact code and the instructions. And I see no reason why I should be able to reproduce it. Explorer does not use FILE_SHARE_DELETE for folders. Therefore, I get an access denied exception. – Sugary

Let me point out where exactly this answer is misleading: When Delete returns and there are no share-delete handles (and there almost never are) the file is gone. – Sugary

I deleted misleading part about files, since that is not related to question anyway. However, answer itself is correct, and I can reproduce behavior myself with code provided, so in that part you are certainly wrong - if CreateDirectory returned, Directory.Exists might and will return false in certain cases. – Warhead

@Warhead I'm deleting the folder and it's files: Directory.Delete(myDir, true) – Abhor

Well sure you delete folder with files, but problem is not related exactly to that. Maybe you reproduce the problem with code and comments above? – Warhead

@Sugary I've updated my answer with one more example which confirms my point. – Warhead

OK, I can reproduce the second behavior that you give code for. I used procmon to find out what happens. Indeed, explorer.exe uses FILE_SHARE_DELETE. Right after that, the linqpad code crashes. The crash happens before by notepad++ comes into play. My point stands: Only through file_share_delete usage this issue can happen. – Sugary

Run pastebin.com/z15Pgs2h in a second program concurrently. Now there is no async delete anymore. – Sugary

Not sure how it contradicts my point that this can happen this way, without any races in code, no matter if because of FILE_SHARE_DELETE (which as we find out is not extremely rare) or otherwise. – Warhead

Answer 2

I wrote myself a little C# method for synchronous folder deletion using Directory.Delete(). Feel free to copy:

private bool DeleteDirectorySync(string directory, int timeoutInMilliseconds = 5000)
{
    if (!Directory.Exists(directory))
    {
        return true;
    }

    var watcher = new FileSystemWatcher
    {
        Path = Path.Combine(directory, ".."),
        NotifyFilter = NotifyFilters.DirectoryName,
        Filter = directory,
    };
    var task = Task.Run(() => watcher.WaitForChanged(WatcherChangeTypes.Deleted, timeoutInMilliseconds));

    // we must not start deleting before the watcher is running
    while (task.Status != TaskStatus.Running)
    {
        Thread.Sleep(100);
    }

    try
    {
        Directory.Delete(directory, true);
    }
    catch
    {
        return false;
    }

    return !task.Result.TimedOut;
}

Note that getting task.Result will block the thread until the task is finished, keeping the CPU load of this thread idle. So that is the point where it gets synchronous.

Kassa

Answer 3

Sounds like race condition to me. Not sure why - you did not provide enough details - but what you can do is to wrap everything in lock() statement and see if the problem is gone. For sure this is not production-ready solution, it is only a quick way to check. If it's indeed a race condition - you need to rethink your approach of rewriting folders. May be create "GUID" folder and when done - update DB with the most recent GUID to point to the most recent folder?..

Kaki

Added this line to the code: Debug.WriteLine(System.Threading.Thread.CurrentThread.ManagedThreadId). Everything seems to happen in the same thread. Also tried locking too, still having the same issue. – Abhor

Plus, it's already a GUID folder, in some cases I need to overwrite this GUID folder, which this happens. – Abhor

Nothing in the world will allow you to lock stuff going on outside your program, e.g. the file system. – Waldrop