Anti-virus in docker container - does fanotify works between host and container?

Question

I need to implement anti-virus on-access scanning solution for files inside docker containers using open-source software. Clamav On-Access works fine but have some requirements and limitations:

• require CAP_SYS_ADMIN capability for working inside a container
• needs to be run per-container, not per-host
• require 850Mb resident memory for signatures in each running container, even small one

Does this limitation - "fanotify not working for container events when watching from host", really exists or I just misconfigured ClamAV? I have no deep knowledge how fanotify works with namespaces, but it looks like kernel limitation to me.

UPDATE: Are there any workarounds for this limitation? Adding /var/lib/docker/overlay2/container_id/merged is one option, because of dynamic container nature clamd.conf needs to be updated on every container event. But even with added path ClamAV doesn't detect malicious files in the containers.

Running ClamAV per-container creates huge memory overhead, especially for small containers.

Links collection:

• fanotify events do not work between containers
• Using fanotify within Docker Containers
• Fanotify kernel interface does not support scanning inside containers
• Could not watch path /var/lib/docker/overlay2 error
• Patch for OnAccessIncludePath traversal across file systems

Apuleius

Could you run the AV on the host, and expose on-host volumes from the containers you want to scan? – Bula

@halfer, that's exactly what I'm trying to do now, run one clamd in inotify mode for /var/lib/docker/overlay2 and second clamd in fanotify mode for / at host – Apuleius

Answer 1

I have a solution with a patched ClamAV.

1. Must use ClamAV < 0.102.0 because of the splitting of scanning and detection: detected files can't be scanned because the path is observed from the container point of view
2. OnAccessMount doesn't work because you have to list each mount in ClamAV config then restart and docker creates mounts on the fly
3. Must use overlayfs not LVM so ClamAV can access the mount
4. OnAccessIncludePath doesn't work because the file and folder enumeration method doesn't traverse file systems (doesn't scan beyond mount for path specified)

I was able to get OnAccessIncludePath working with a patch I posted to clamav-devel mailing list: https://lists.gt.net/clamav/devel/77347#77347.

I ended up with one process using fanotify for static mounts and one using inotify to monitor /var/lib/docker ephemeral mounts. Having 2 instances is still much better than 1 per container. I did a fair bit of load testing and have had the patch in production since about the time I mailed the list.

Sophos didn't work for me but I gave up pretty quickly.

Voroshilov

thank you for the answer, works for me, do you have an idea, how your patch can be integrated with more recent clamav versions? – Apuleius

btw, why do we need two clamav daemons and cannot run just one with watching / recursively? – Apuleius

ClamAV 0.102.0 would have to pass info from clamscan to clamd using something other than path (inode entry ID?). – Voroshilov

The OnAccessIncludePath and OnAccessMountPath are mutually exclusive and OnAccessMountPath supersedes OnAccessIncludePath globally per process not by path. OnAccessPrevention only works with OnAccessMountPath (fanotify). Using fanotify is more efficient, doesn't use up inotify watches and is less likely to miss a file. – Voroshilov

looks like the opposite - lists.gt.net/clamav/users/74107: From the documentation here: github.com/Cisco-Talos/clamav-devel/blob/dev/0.101/docs/… Watch your entire filesystem only using the clamd.conf OnAccessMountPath option. While this will disable on-access prevention, it will avoid potential system lockups caused by fanotify’s blocking functionality. To use OnAccessPrevention, you'll need to use OnAccessIncludePath instead of OnAccessMountPath. – Apuleius

Right the entire filesystem but each docker container has a different filesystem. You have to tell ClamAV about that filesystem but you won't know the path ahead of time. – Voroshilov

Answer 2

Now fanotify can monitor events across the filesystem, regardless of the mount namespaces. You need to use the flag FAN_MARK_FILESYSTEM alongwith FAN_MARK_ADD (FAN_MARK_ADD | FAN_MARK_FILESYSTEM). From the fanotify_mark man page here is the snippet:

FAN_MARK_FILESYSTEM (since Linux 4.20)
       Mark the filesystem specified by pathname.  The filesystem
       containing pathname will be marked.  All the contained
       files and directories of the filesystem from any mount
       point will be monitored.

Selectman

Answer 3

Yes, fanotify only monitors events in the mount namespace that it is running in.

Union

thank you for the answer @Douglas Leeder, I've tested Sophos Antivirus as well, it is able to detect viruses in container when running on host. Unfortunately, unable to download it from official site due to some restrictions. – Apuleius