Very high CPU usage by Windows Defender when using doParallel's foreach in R

Question

I have a Threadripper 1950X based workstation with 16 cores and 32 threads and plenty of memory. Running 64-bit R 3.6.0 (patched) on Windows 10, I frequently run parallel code in R using the doParallel library and the foreach command, frequently setting it to use 26-30 threads.

Recently, I checked out the task manager. As doParallel started spooling up all its processes, I wasn't surprised to find its CPU usage going up. But very strangely, Windows Defender (Microsoft's antivirus default) started spooling up too, and very aggressively, climbing to as high as 70% usage (it is listed as Antimalware Service Executable). Here's a screenshot of what I mean. When the R code completes, Defender goes back down to insignificant CPU usage.

I have read posts online on Defender's high CPU usage, but this seems extremely tied to parallel operations in R. I have tried to set the exclusions as per posts like this but it doesn't improve matters.

Should I be concerned that Windows Defender is constantly crowding out R when I run parallel code with lots of threads set?

Roberto

When I had a similar issue I got around it by using a virtual machine with Ubuntu on my laptop. Window's defender didn't need to know what was going on. – Fulsome

Answer 1

I think I've figured it out. My code creates thousands of temporary files to R's temp directory (in c:\Users\xxx\Local\Temp) if I have 20-30 threads. I think Windows Defender was going nuts scanning each one, even though it's on an SSD drive. I added the temp directory to Defender's exclusion list, as well as some output directories in my main folder hierarchy, and the CPU usage went way down, as can be seen here.

I hope this is helpful to others, because basic Google searches don't address this problem.

Roberto

This is very-very... bad in term of security, because virus tend to lurk in temp... for some reason I also tempted to exclude temp folder... but.. ugh.... – Picot