WatchService performance with many directories

Question

I want to use the Java WatchService to listen for changes on a big number of directories (many hundreds of thousands) but I don't know if it is appropriate for such numbers of watched directories.

Does anyone have experience with WatchService with such numbers of directories?

If it helps, the WatchService will be used on CentOS 6.5 with an EXT4 file system.

Thanks, Mickael

Peralta

Just curious : what is the application of that? Why so many dirs? – Hb

The application will be used to monitor the voicemail directories created by Asterisk. Asterisk has multiple backends to store voicemails: file system, ODBC and IMAP. I'm afraid about ODBC performance so I thought I should use the file system backend. – Peralta

Did you check for some asterisk api that maybe provides an event for this? – Hb

There should be something, I need to check. However, I'm still interested to know about the ability of WatchService for handling big number of watched directories. – Peralta

Answer 1

This situation is fairly common for IDEs. They often use directory watching for complex directory structures and many 10s of thousands of files.

There is two things to note:

• On Linux you often need to tune the OS to monitor so many files. https://confluence.jetbrains.com/display/IDEADEV/Inotify+Watches+Limit

To prevent this situation it is recommended to increase the watches limit (to, say, 512K). You can do it by adding following line to the /etc/sysctl.conf file:

    fs.inotify.max_user_watches = 524288

This example tunes the system to monitor 512k files.

• If you have an HDD, it won't make it spin any faster and it most likely does 80 - 120 IOPS (IO Per Second) and this is more likely to be a performance bottleneck than you might like.

Like many IO operations in Java, it is wrapper around a facility which is actually implemented by the OS.

Gateshead