Copy a file without using the windows file cache

Question

Anybody know of a way to copy a file from path A to path B and suppressing the Windows file system cache?
Typical use is copying a large file from a USB drive, or server to your local machine. Windows seems to swap everything out if the file is really big, e.g. 2GiB. Prefer example in C#, but I'm guessing this would be a Win32 call of some sort if possible.

Elayneelazaro

If you really want to understand the mechanics of cache interactions with file copying, this is a must read. – Bissonnette

Answer 1

Even more important, there are FILE_FLAG_WRITE_THROUGH and FILE_FLAG_NO_BUFFERING.

MSDN has a nice article on them both: http://support.microsoft.com/kb/99794

Verisimilitude

This answer though correct isn't an answer to the original question. Can someone with Editing rights reword it? – Outshout

Can you please tell me where you see the discrepancy between the Q and A? I'll be happy to reword the answer. – Verisimilitude

@gabr: your reply references ("Even more important") another answer. Don't. Make your answer standalone. Think that only you answered, no one else. – Vevine

Answer 2

In C# I have found something like this to work, this can be changed to copy directly to destination file:

    public static byte[] ReadAllBytesUnbuffered(string filePath)
    {
        const FileOptions FileFlagNoBuffering = (FileOptions)0x20000000;
        var fileInfo = new FileInfo(filePath);
        long fileLength = fileInfo.Length;
        int bufferSize = (int)Math.Min(fileLength, int.MaxValue / 2);
        bufferSize += ((bufferSize + 1023) & ~1023) - bufferSize;
        using (var stream = new FileStream(filePath, FileMode.Open, FileAccess.Read, FileShare.None,
                                           bufferSize, FileFlagNoBuffering | FileOptions.SequentialScan))
        {
            long length = stream.Length;
            if (length > 0x7fffffffL)
            {
                throw new IOException("File too long over 2GB");
            }
            int offset = 0;
            int count = (int)length;
            var buffer = new byte[count];
            while (count > 0)
            {
                int bytesRead = stream.Read(buffer, offset, count);
                if (bytesRead == 0)
                {
                    throw new EndOfStreamException("Read beyond end of file EOF");
                }
                offset += bytesRead;
                count -= bytesRead;
            }
            return buffer;
        }
    }

Nanice

Answer 3

I am not sure if this helps, but take a look at Increased Performance Using FILE_FLAG_SEQUENTIAL_SCAN.

SUMMARY

There is a flag for CreateFile() called FILE_FLAG_SEQUENTIAL_SCAN which will direct the Cache Manager to access the file sequentially.

Anyone reading potentially large files with sequential access can specify this flag for increased performance. This flag is useful if you are reading files that are "mostly" sequential, but you occasionally skip over small ranges of bytes.

Youth

I use this flag for the same purposes, and I can testify that FILE_FLAG_SEQUENTIAL_SCAN is doing the job. – Vevine

Answer 4

If you dont mind using a tool, ESEUTIL worked great for me.

You can check out this blog entry comparing Buffered and NonBuffered IO functions and from where to get ESEUTIL.

copying some text from the technet blog:

So looking at the definition of buffered I/O above, we can see where the perceived performance problems lie - in the file system cache overhead. Unbuffered I/O (or a raw file copy) is preferred when attempting to copy a large file from one location to another when we do not intend to access the source file after the copy is complete. This will avoid the file system cache overhead and prevent the file system cache from being effectively flushed by the large file data. Many applications accomplish this by calling CreateFile() to create an empty destination file, then using the ReadFile() and WriteFile() functions to transfer the data. CreateFile() - The CreateFile function creates or opens a file, file stream, directory, physical disk, volume, console buffer, tape drive, communications resource, mailslot, or named pipe. The function returns a handle that can be used to access an object. ReadFile() - The ReadFile function reads data from a file, and starts at the position that the file pointer indicates. You can use this function for both synchronous and asynchronous operations. WriteFile() - The WriteFile function writes data to a file at the position specified by the file pointer. This function is designed for both synchronous and asynchronous operation. For copying files around the network that are very large, my copy utility of choice is ESEUTIL which is one of the database utilities provided with Exchange.

Crosscrosslet

Answer 5

Eseutil is a correct answer, also since Win7 / 2008 R2, you can use the /j switch in Xcopy, which has the same effect.

Cangue

Answer 6

I understand this question was 11 years ago, nowadays there is robocopy which is kind of replacement for xcopy.

you need to check /J option /J :: copy using unbuffered I/O (recommended for large files)

Plateau