How does a Windows antivirus hook into the file access process?

Question

The subject says it all. A normal antivirus has to intercept all file accesses, scan the files and then optionally deny access to the file (possibly even displaying a prompt to the user). How can this be done?

I'm aware of a method called API hooking, but that's a really dirty undocumented hack - and as such isn't really reliable. What's the "official" way of doing this?

Alternatively, I would be interested in intercepting the loading of executable modules (.DLL, .EXE, etc.), not just arbitrary file reads.

Phelips

It depends on the version of Windows in play. You will find they do lots of unsupported things, especially on older versions of Windows. I believe newer versions have an API... but I am unfamiliar with it so I'll let others answer. – Drin

There's a fine line between "virus" and "antivirus". They do a lot of the same dirty tricks to get their work done. – Gretel

Never worked on Anivirus software before, but I would assume they are implemented as a File System Filter Driver – Unhitch

Just to mention, techniques used by Antivirus software is near the same exploited by those fragments the software is written against. – Oneill

@MareInfinitus Thanks, hadn't spotted the antivirus tag – Stockdale

If you think something is impossible to do under Windows, put it in a kernel driver. – Spectra

@Amazed but only on 32-bit systems - otherwise you have to dodge PatchGuard – Stockdale

@Stockdale True, but that doesn't sound as poetic :) – Spectra

Greg - whut? There's a huge line in that AV software is not malicious. Yes they might use the same APIs but so what? That's what file system APIs are for. – Move

Answer 1

In the recent versions of windows (at least XP onwards) there is the concept 'filters' which can be viewed using MS Filter Manager, (fltmc.exe from a command prompt)

This provides a low level I/O hook that AV programs can access and automatically register to be passed all I/O requests to the file system. It is a kit you can get the drivers for an develop your own filters for.

http://www.microsoft.com/whdc/driver/filterdrv/default.mspx is a starting place to get in depth info.

Jerrilyn

Is there any other filter driver api implemented in windows (except network and filesyste)? – Maximilien

Answer 2

As you already noted, hooking is a key to what of-the-shelf AV software with "realtime" protection does.

You could have a look on the (widely discussed) winpooch, which already does API Hooking, but there are some major flaws in this software. Sourceforge of Winpooch

There is also an article on Codeproject on API hooking, providing some library to do hooking "in three layers". Dll Injection is somewhat hard, as you can image. CodeProject: EasyHook, reinvention of API Hooking

As you are probably interested in Antivirus strategies, i also suggest having a look at ClamAV, or WinClam, which is opensource (under GPL) ClamAV for windows

But i do not have a clue how to do API hooking with C#, i have to admit. In C / C++ this is (quite) easy...

ADD ON You may be interested in the sources of FileMon, a widely known FileSystem Monitor that was once by SysInternals and now by Microsoft: It uses Driver-Filter API by Microsoft, which is at least known as fragile.

Link may be found here in Sysinternals forum

Oneill

Good thinking re: FileMon. The irony is I use it but didn't even think of it. I'm reading your other links now – Stockdale

Answer 3

Through File System Filter Drivers. However, implementing such drivers is quite complicated and "fragile".

Mahlstick

Answer 4

File access is monitored using filesystem filter driver, which works in kernel mode. Filter drivers can be not just notified about filesystem operations, but alter the data passed via filters or deny filesystem requests.

You can create a minifilter yourself, yet maintenance and support of your kernel-mode code can be non-trivial, especially without kernel-mode development experience. One of problems is conflicts between various filters.

Our company offers CallbackFilter product, which provides a ready-to-use driver and lets you write business logic, related to filtering, in user mode.

Hengel

Answer 5

You can read about the detours library from microsoft and try it for free - it allows you to write user mode hooks in c#. No need for you to learn about drivers :]

However - for kernel mode hooks - you will need to know c and play around with the DDK - atleast afaik :[

And most modern anti-virus software intercept quite a few calls - registry apis, thread and process apis etc - not just the file system api. Again - afaik.

edit: There are also a few open source rootkits - google them and see how they perform their hooking, it will be educational I guess.

Ceyx

Thanks for the info on the detours library - I'm reading now and good suggestion re: rootkits – Stockdale

"DDK" - I'm assuming Driver Dev Kit? I can't see anything on google (But I did find the WDK - Windows Driver Kit) – Stockdale

Oops - my bad. It used to be the DDK a long time back. You are right - it is the windows driver kit. – Ceyx

In the end, the detours library looks like the most robust and reliable way to do this - especially since it's MS-backed. I did like the look of @MareInfinitus' EasyHook but it's fairly explicit about why it can't work on x64 – Stockdale

Detours is quite expensive when for commercial use. – Oliveolivegreen

For the record an alternative to Detours is Deviare – Rules

Answer 6

In general, these products intercept functions to get a HANDLE to a process like OpenProcess or NtOpenProcess. They also, hook CreateRemoteThread functions and memory allocation in a remote process: VirtualAlloc and VirtualProcect. Some AVs also hook SetWindowsHookEx function to detect global hooks to avoid key loggers.

Hooking these APIs they can control which modules (or dlls) can access remote processes and allow only those that the user know what they are doing.

You can use HookShark to see what user-mode functions are intercepted by each AV product.

To make your own user hooks you can use detours library but you have to develop an agent to run your in-process hooks and then communicate with an agent server. You can also use Deviare API Hook which is a framework that makes all the complex staff so you can code your hooks in your own process using any programming language.

Transect