How can I log low-level OS file transactions in C#?

Question

Since File/Process Monitor falls short in terms of filtering and unnecessary duplication when logging, I want to recreate what that program does and log all Windows file operations live in realtime.

I want to record various attributes such as the time, process name, source path, destination path, operation, result, and detail, just like Process Monitor does.

How can I get C# to extract this information from the OS?

---

EDIT: As zett42 pointed out, the FileSystemWatcher won't quite work as for example, file events created from processes themselves won't be intercepted. For instance, none of these transactions show up, even though I added the events: Changed, Created, Renamed, and Deleted to the FileSystemWatcher and set the EnableRaisingEvents flag to true.

---

EDIT 2: Using SimonMourier's suggestion of the Microsoft.Diagnostics.Tracing.TraceEvent nuget package, I managed to knock up the code below.

This section is put into a background worker:

Console.CancelKeyPress += (sender, e) => session.Stop();
session.EnableKernelProvider(KernelTraceEventParser.Keywords.All);
session.Source.Kernel.FileIOWrite += Kernel_FileIOWrite;            
session.Source.Process();

And then the FileIOWrite event created runs the following when called (automatically):

private void Kernel_FileIOWrite(Microsoft.Diagnostics.Tracing.Parsers.Kernel.FileIOReadWriteTraceData obj)
{
    string filename = obj.FileName;
    string processpath = "";
    if (obj.ProcessID == 0) processpath = "System Idle Process";
    else if (obj.ProcessID == 4) processpath = "System";
    else
    {
        try { processpath = "ID: " + obj.ProcessID + ": "+ Process.GetProcessById(obj.ProcessID).MainModule.FileName; }
        catch { processpath = "error ID: "+ obj.ProcessID; }
    }
    richTextBox1.AppendText(filename + " ............."+ processpath +"\n");
}

Unfortunately, FileIOReadWriteTraceData.FileName is not picking up things Procmon picks up such as (for example), very common Chrome stuff such as writing to C:\Users\Dan\AppData\Local\Google\Chrome\User Data\Default\Cookies-journal or C:\Users\Dan\AppData\Local\Google\Chrome\User Data\Default\Current Session

Brendabrendan

Have you tried System.IO.FileSystemWatcher? – Burden

@bhmahler FileSystemWatcher doesn't give you the information that Process Monitor gives, e. g. you won't get information about the process that caused the change. – Bullnecked

@Bullnecked @bhmahler zett42 is correct. For instance, none of these transactions show up, even though I added the events: Changed, Created, Renamed, and Deleted to the FileSystemWatcher and set the EnableRaisingEvents flag to true. – Brendabrendan

Process Monitor does exactly what you are asking for: It logs low-level I/O operations. It doesn't duplicate anything. It merely shows the events that happen. If you are going to write a tool that does what you are asking for, you will wind up with the same situation: You are going to get occasional near-identical (yet unique) entries. – View

@IInspectable: See the link I gave. Technically, you're right, they're unique entries, but for practical purposes, I don't want to see 100 events on the same file being opened, closed, and queried at slightly different offset positions within the space of 0.1 seconds. That's perverse when you just want a decent overview of the files being manipulated. – Brendabrendan

Regardless, I believe File Monitor installs a file system minifilter driver to monitor I/O. Probably can't do that in C#, though. – View

You can use ETW and the Microsoft.Diagnostics.Tracing.TraceEvent nuget package as described here: https://mcmap.net/q/1168701/-command-line-version-of-procmon-closed – Player

@SimonMourier: That looks interesting. I've had bad experiences with nuget in the past though. – Brendabrendan

nugets are just compiled pieces of code. In this case, it's provided by Microsoft and that's mostly the way Microsoft works now for distributing development libraries. You can use the source: github.com/Microsoft/perfview/tree/master/src/TraceEvent but the nuget is much easier – Player

@SimonMourier: Many thanks! Got it to work in the end. I still can't seem to pick up certain transactions though. Using the FileIOWrite and DiskIOWrite events for instance, I can't obtain the Path and ImagePath columns that ProcessMonitor shows (e.g: Chrome updating its cache or Cookies-journal file). – Brendabrendan

FileIOReadWriteTraceData.FileName will give you the equivalent of Path when available (you'll get much more information with TraceEvent than with ProcessMonitor) and full process information is easy to get with Process.GetProcessById(FileIOReadWriteTraceData.ProcessID). You shouldn't need DiskIOWrite if you want file information. – Player

@SimonMourier: Ah, although GetProcessById() appears to work (albeit somewhat relatively slowly), unfortunately FileIOReadWriteTradeData.FileName fails to pick up things Procmon picks up such as (for example), very common Chrome stuff such as writing to C:\Users\Dan\AppData\Local\Google\Chrome\User Data\Default\Cookies-journal or C:\Users\Dan\AppData\Local\Google\Chrome\User Data\Default\Current Session. At the least, they show as blank. I'll add code to my main question. – Brendabrendan

Answer 1

You can never capture all the things that Process Monitor captures in C#. One of the reasons that ProcMon is so good at capturing all of those things is because ProcMon contains a driver that is loaded in kernel mode and hooks itself and listens for those events. If you want to replicate the process monitor, you will have to write your own driver to capture all the things that you want to. A windows driver cannot be written in C# and you will have to write the driver in C/C++.

The other option is to get Process Monitor to log everything to file and filter the events yourself.

Unaccountable

Answer 2

Did you try to use WMI? The ManagementEventWatcher could provide all information you need, even though it's not that handy as the FileSytemWatcher. I'm not sure it will work and unfortunately cannot try it myself at the moment, but this is the point where I would start.

Take a look at the anwer of RRUZ in this thread, which does exactly what you want. You will just need to find out if WMI provides all required information.

Soften