R only ever runs on a certain CPU in Linux

Question

I have an 8-core RHEL Linux machine running R 4.0.2.

If I ask R for the number of cores, I can confirm that 8 are available.

> print(future::availableWorkers())

[1] "localhost" "localhost" "localhost" "localhost" "localhost" "localhost"
[7] "localhost" "localhost"

> print(parallel::detectCores())

[1] 8

However, if I run this simple example

f <- function(out=0) {
    for (i in 1:1e10) out <- out + 1
}

output <- parallel::mclapply(1:8, f, mc.cores = 8)

my top indicates that only 1 core is being used (so that each worker is using 1/8th of that core, or 1/64th of the entire machine).

%Cpu0  :100.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu1  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu2  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu3  :  2.0 us,  0.0 sy,  0.0 ni, 98.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu4  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu5  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu6  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu7  :  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 32684632 total, 28211076 free,  2409992 used,  2063564 buff/cache
KiB Swap: 16449532 total, 11475052 free,  4974480 used. 29213180 avail Mem

  PID USER  PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
 3483 user  20   0  493716  57980    948 R  1.8  0.2   0:18.09 R
 3479 user  20   0  493716  57980    948 R  1.5  0.2   0:18.09 R
 3480 user  20   0  493716  57980    948 R  1.5  0.2   0:18.08 R
 3481 user  20   0  493716  57980    948 R  1.5  0.2   0:18.09 R
 3482 user  20   0  493716  57980    948 R  1.5  0.2   0:18.09 R
 3484 user  20   0  493716  57980    948 R  1.5  0.2   0:18.09 R
 3485 user  20   0  493716  57980    948 R  1.5  0.2   0:18.09 R
 3486 user  20   0  493716  57980    948 R  1.5  0.2   0:18.09 R

Does anyone know what might be going on here? Another StackOverflow question that documents similar behavior is here. It's clear that I messed up the install somehow. I followed these install instructions for RHEL 7. I'm guessing there is a dependency missing, but I have no idea where to look. If anyone has any ideas of diagnostics to run, etc., they would be most appreciated.

For further context, I have R 3.4.1 also installed on my machine, and when I run this code, everything works fine. (I installed that version through yum.)

I also installed R 4.0.3 yesterday using the same instructions linked above, and it suffers from the same problem.

Taishataisho

Do you get the same result with workers <- parallel::makeCluster(8, type = "PSOCK"); parallel::parLapply(1:8, f, cl = workers)? – Abjuration

@Abjuration Yes, I get identical behavior using the code you suggested. – Taishataisho

just tested it on a Linux / Debian virtual machine with two cores allocated : both cores are activated 100%. – Candiecandied

If you're having the same issue with a 'socket' approach, it may be an issue with openBLAS. Try the troubleshooting suggestion (system(sprintf("taskset -p 0xffffffff %d", Sys.getpid()))) and solution here: https://mcmap.net/q/1174995/-why-is-r-multicore-only-using-one-core-duplicate / #12925198 – Abjuration

Thanks @jared_mamrot. Here is the output I see from your suggestion: > system(sprintf("taskset -p 0xffffffff %d", Sys.getpid())) pid 10256's current affinity mask: ff pid 10256's new affinity mask: ff So it doesn't seem to be the problem. – Taishataisho

So running system(sprintf("taskset -p 0xffffffff %d", Sys.getpid())) then running your simple example still only uses one core... Can you please include the output from the command sessionInfo() in your question? Also it might be worth installing R from source – Abjuration

@Abjuration I ended up installing R 4.0.3 this afternoon, and the solution you referenced fixes my issue. If you wouldn't mind adding this as an answer, I'll be happy to award you the bounty. Thanks so much for your help! – Taishataisho

Fantastic! Glad it worked :) – Abjuration

Answer 1

First run

system(sprintf("taskset -p 0xffffffff %d", Sys.getpid()))

then your simple example

f <- function(out=0) { for (i in 1:1e10) out <- out + 1 }
output <- parallel::mclapply(1:8, f, mc.cores = 8)

works on all 8 cores.

Abjuration

And an even better solution is to add the system() call above to my .Rprofile file in my home directory, inside the function .First() as explained here: #33054664. This ensures that the command gets called at the beginning of every R session. – Taishataisho