Reading a File while it is being written by another process

Question

I'm trying to read binary data from a buffer file which is continuously written to by a different process (that I cannot modify). I'm using the following code in order to open the file:

fileH = CreateFileA((LPCSTR)filename,
                    GENERIC_READ,
                    FILE_SHARE_READ | FILE_SHARE_WRITE,
                    NULL,
                    OPEN_EXISTING,
                    FILE_ATTRIBUTE_NORMAL, NULL);

And it opens correctly with no error. However, when I read data from the file, it seems to block the other process from writing to the file since I loss data.

The buffer is circular, meaning that the file size is fixed, and new data is constantly written over older data in the buffer.

EDIT: Sometimes the most trivial solution works...

I've contacted the software company and told them about the bug, and within a day they posted a new version with a fix. Sorry this cannot work for everybody.

Malemute

Reading and writing from/to a file at the same time is a bad idea. Try to use Mutex or Events to handle the access. – Seaddon

@bash.d, really? Tell that to tail and similar utilities. Also you can't use mutexes and events to sync processes. Besides he is saying he can't modify the other application. – Puduns

@aleguna Hell, yes you can! Read the first line here. – Seaddon

The real question is why you're still using non-Unicode APIs in the year 2013. Don't you mean CreateFileW? – Candelabra

Do you know if that other process keeps the file open, or if it tries to open the file each time it needs to write? If it's trying to open file in exclusive mode, it will fail when you have the file open for read. Contrary to what somebody else said, it's perfectly okay in general to have multiple processes reading/writing the same file. – Quentin

"The buffer is circular, file size is fixed" are you sure the other process uses WriteFile and doesn't memory map the file then? That's what I'd do in that case anyway. And it might explain why you don't see changes. As @JimMischel said, reading and writing concurrently otherwise is no issue. Assuming the other process has already openend (and keeps open!) the file previously, there's nothing I see that could fail either. – Humberto

@JimMischel It's a good point. I just tried to only open the file, and it seems that it blocks the buffer in the same way. Is there a way to overcome this? – Malemute

@Humberto I agree with you that a memory map would have been better, unfortunately this is not the case. – Malemute

Too localized? Seriously? – Peplos

Answer 1

It's hard to say what your options are without knowing how the writing process is opening the file. Obviously, it's not opening the file for exclusive access and keeping it open. Otherwise you wouldn't be able to read it at all.

The behavior you describe indicates that the writing process opens the file for exclusive access, writes to it, and then closes the file. If that's the case, then you can't have your program open the file and keep it open. That would cause the writing process to fail whenever it tried to write.

If you can't modify the writing process, then your options are limited and not very attractive. Most likely, you'll have to make your program open the file, read a small chunk, close the file, and then wait for a bit before reading again. Even then, there's no guarantee that you won't have the file open when the writing process tries to write. Which, I think, you have already discovered.

Do you know if the writing process loses the data when it can't open the file, or if it just buffers the data and writes it the next time it can actually open the file? If that's the case, then my suggestion of stepping through the file a little at a time could work. Otherwise, you're going to lose data.

There is no open mode that I know of that is the equivalent of "Open the file for reading, but if somebody wants exclusive access, then let them have it."

Another possibility would be to have your program rename the file whenever you want to read, and then delete the renamed file after you've read it. This assumes, of course, that the writing process will create a new file if necessary. Even then, there might be a problem if the writing process tries to write while you're renaming. I don't think that'll be a problem (the rename could be atomic as far as the file system is concerned), but it's something you'd have to research.

Quentin

Data is really lost for the writing process, so even opening the file for a very short period, can cause this loss. Renaming the file sounds like a interesting possibility, I'll check it. – Malemute

Answer 2

I would recommend looking at the source code of the excellent Far Manager. Its internal viewer can handle multi-gigabyte files easily, has no problems showing files which are being written and can update the changed file contents almost in real-time. I've never noticed any blocking issues with the files being displayed.

The source code related to the question seems to be in the viewer.cpp file.

One interesting thing is that it does not use GENERIC_READ:

ViewFile.Open(strFileName, FILE_READ_DATA, FILE_SHARE_READ|FILE_SHARE_WRITE|FILE_SHARE_DELETE, nullptr, OPEN_EXISTING);

I suspect dropping SYNCHRONIZE might be important here.

The file change detection is in Viewer::ProcessKey, KEY_IDLE case:

// Smart file change check -- thanks Dzirt2005
//
bool changed = (
    ViewFindData.ftLastWriteTime.dwLowDateTime!=NewViewFindData.ftLastWriteTime.dwLowDateTime ||
    ViewFindData.ftLastWriteTime.dwHighDateTime!=NewViewFindData.ftLastWriteTime.dwHighDateTime ||
    ViewFindData.nFileSize != NewViewFindData.nFileSize
);
if ( changed )
    ViewFindData = NewViewFindData;
else {
    if ( !ViewFile.GetSize(NewViewFindData.nFileSize) || FileSize == static_cast<__int64>(NewViewFindData.nFileSize) )
        return TRUE;
    changed = FileSize > static_cast<__int64>(NewViewFindData.nFileSize); // true if file shrank
}

Cached file reading is implemented in cache.cpp. But there's nothing really earth-shattering there, just some Seek() and Read() (that eventually result in SetFilePointerEx and ReadFile API calls). OVERLAPPED is not used.

Arceliaarceneaux

I'd be pretty surprised if it could view a file that's opened in exclusive mode by another process. – Quentin

Of course it does not work with exclusively opened files. But I haven't found it to be common, especially for log files. – Arceliaarceneaux