File Access Strategy in a Multi-Threaded Environment (Web App)

Question

I have a file which is an XML representation of some data that is taken from a Web service and cached locally within a Web Application. The idea being is that this data is very static, but just might change. So I have set it up to cache to a file, and stuck a monitor against it to check if it has been deleted. Once deleted, the file will be refreshed from its source and rebuilt.

I am now running in to problems though, because obviously in a multi-threaded environment it falls over as it is trying to access the data when it is still reading/writing the file.

This is confusing me, because I added a object to lock against, and this is always locked during read/write. It was my understanding that attempted access from other threads would be told to "wait" until the lock was released?

Just to let you know, I am real new to multi-threaded development, so I am totally willing to accept this is a screw up on my part :)

• Am I missing something?
• What is the best file access strategy in a multi-threaded environment?

  ---

Edit

Sorry - I should have said this is using ASP.NET 2.0 :)

Arzola

Answer 1

Here is the code that I use to make sure a file is not locked by another process. It's not 100% foolproof, but it gets the job done most of the time:

    /// <summary>
    /// Blocks until the file is not locked any more.
    /// </summary>
    /// <param name="fullPath"></param>
    bool WaitForFile(string fullPath)
    {
        int numTries = 0;
        while (true)
        {
            ++numTries;
            try
            {
                // Attempt to open the file exclusively.
                using (FileStream fs = new FileStream(fullPath,
                    FileMode.Open, FileAccess.ReadWrite, 
                    FileShare.None, 100))
                {
                    fs.ReadByte();

                    // If we got this far the file is ready
                    break;
                }
            }
            catch (Exception ex)
            {
                Log.LogWarning(
                   "WaitForFile {0} failed to get an exclusive lock: {1}", 
                    fullPath, ex.ToString());

                if (numTries > 10)
                {
                    Log.LogWarning(
                        "WaitForFile {0} giving up after 10 tries", 
                        fullPath);
                    return false;
                }

                // Wait for the lock to be released
                System.Threading.Thread.Sleep(500);
            }
        }

        Log.LogTrace("WaitForFile {0} returning true after {1} tries",
            fullPath, numTries);
        return true;
    }

Obviously you can tweak the timeouts and retries to suit your application. I use this to process huge FTP files that take a while to be written.

Homebrew

Answer 2

If you're locking on a object stored as a static then the lock should work for all threads in the same Application Domain, but perhaps you need to upload a code sample so we can have a look at the offending lines.

That said, one thought would be to check if IIS is configured to run in Web Garden mode (i.e. more than 1 process executing your application) which would break your locking logic. While you could fix such a situation with a mutex it'd be easier to reconfigure your application to execute in a single process, although you'd be wise to check the performance before and after messing with the web garden settings as it can potentially affect performance.

Snow

Answer 3

You could maybe create the file with a temporary name ("data.xml_TMP"), and when it's ready change the name to what it is supposed to be. That way, no other process will be accessing it before it is ready.

Infringement

Answer 4

OK, I have been working on this and ended up creating a stress-test module to basically hammer the crap out of my code from several threads (See Related Question).

It was much easier from this point on to find holes in my code. It turns out that my code wasn't actually far off, but there was a certain logic path that it could enter in to which basically caused read/write operations to stack up, meaning if they didn't get cleared in time, it would go boom!

Once I took that out, ran my stress test again, all worked fine!

So, I didn't really do anything special in my file access code, just ensured I used lock statements where appropriate (i.e. when reading or writing).

Arzola

Answer 5

How about using AutoResetEvent to communicate between threads? I created a console app which creates roughly 8 GB file in createfile method and then copy that file in main method

 static AutoResetEvent waitHandle = new AutoResetEvent(false);
    static string filePath=@"C:\Temp\test.txt";
    static string fileCopyPath=@"C:\Temp\test-copy.txt";
    static void Main(string[] args)
    {
        Console.WriteLine("in main method");
        Console.WriteLine();
        Thread thread = new Thread(createFile);
        thread.Start();

        Console.WriteLine("waiting for file to be processed ");
        Console.WriteLine();
        waitHandle.WaitOne();
        Console.WriteLine();

        File.Copy(filePath, fileCopyPath);
        Console.WriteLine("file copied ");

    }


    static void createFile()
    {

        FileStream fs= File.Create(filePath);            
        Console.WriteLine("start processing a file "+DateTime.Now);
        Console.WriteLine();
        using (StreamWriter sw = new StreamWriter(fs))
        {
            for (long i = 0; i < 300000000; i++)
            {
                sw.WriteLine("The value of i is " + i);

            }
        }
        Console.WriteLine("file processed " + DateTime.Now);
        Console.WriteLine();

        waitHandle.Set();
    }

Ctenidium