Is there a way to limit the number of CPU cores Bazel uses?

Question

Is there a way to tell Bazel when building how many CPU cores it can use?

TL;DR

I build TensorFlow on a VMware Workstation and being a virtual machine I can adjust the number of processors and cores for the virtual machine.

In the process of building TensorFlow I found that using only one core works. When I give the Workstation four cores and build TensorFlow it eventually halts the system to the point I have to reboot.

If I wait a few hours (leave it alone overnight) it sometimes returns with the following error:

gcc: internal compiler error: Killed (program cc1plus)

While I can change the number of cores using the virtual machines configuration options I would prefer to do it without having to shut down and restart the virtual machine.

Marianmariana

Answer 1

From Bazel User Manual

--local_resources availableRAM,availableCPU,availableIO

This option, which takes three comma-separated floating point arguments, specifies the amount of local resources that Bazel can take into consideration when scheduling build and test activities. Option expects amount of available RAM (in MB), number of CPU cores (with 1.0 representing single full core) and workstation I/O capability (with 1.0 representing average workstation). By default Bazel will estimate amount of RAM and number of CPU cores directly from system configuration and will assume 1.0 I/O resource.

If this option is used, Bazel will ignore both --ram_utilization_factor.

Marianmariana

This does allow one to reduce the resources used, but not as I would expect. After struggling to make sense, I closed other applications and tested with different settings. It seems Bazel consumes approximately double the number of cores I set and sometimes consumes all cores for many seconds. Even with bazel test -c opt --local_resources 20000,1,0.25 I see significant load on my system. – Moniz

I was going to fix the broken link above to point to docs.bazel.build/versions/master/… but the text isn't the same as what used to be in the user manual. I'm not sure what the best approach here would be... – Pisci

@MattPassell Since I view this all as Creative Content and it is here to help others, and I haven't touched Tensorflow in some time, take your pick. 1) Edit this question for what needs to be changed. 2) Add your own answer. 3) Edit this question, but add a new section leaving in the original answer and noting what you would consider a correct answer. – Marianmariana

Fair enough. :) I'll try to come back to this. – Pisci

Answer 2

some examples for your .bazelrc

build --local_ram_resources=HOST_RAM*.5 --local_cpu_resources=HOST_CPUS-1 (leave one core free)

or

build --local_cpu_resources=1 (use a single core)

See https://docs.bazel.build/versions/master/command-line-reference.html#flag--local_cpu_resources

The currently accepted answer is deprecated.

Homemaking

yes, the current accepted answer is deprecated and this one worked – Iggy