Is there a maximum number of CPU's that a VirtualBox could bare?

Question

I am using VirtualBox 5.1 running in a host with 48 CPU's and 250GB of RAM the virtual machine that I am importing (the guest) initially had 2 CPU's and 4GB of RAM.

Inside this machine I am running a process with Java that starts a dynamic number of threads to perform some tasks.

I ran it with configurations below:

The whole process in my laptop (2 CPUs/4GB RAM) ~ 11 seconds

same program in the virtual machine in the server (15 CPUs and 32GB of RAM) ~ 45 seconds

same program in the virtual machine in the server (20 CPUs and 32GB of RAM) ~ 100+ seconds

same program in the virtual machine in the server (10 CPUs and 32GB of RAM) ~ 5+ seconds

First I thought that there was a problem in how I was managing the threads from Java but after many tests I figured out that there was a relation between the number of CPU's that the virtual machine has and its performance, the maximum was 10, after that the overall performance of the machine slows down(CPU starvation?)

The virtual machine runs Oracle Enterprise Linux 6.7 and the host runs Oracle Enterprise Linux 6.9

I couldn't found any hard limit in the Virtual Machine documentation regarding the number of CPU's.

Is there a setting that needs to be set to enable/take advantage of more than 10 CPU's in a VirtualBox instance?

Thorax

There can be no just "host with 48 CPUs". It have several physical CPU packages (sockets), so it is NUMA System. Depending on the tasks you run with java, there can be effects limiting scalability: pure computation task with low memory intensity will scale good, and working with memory actively will scale not linear, if the program was made without accounting for NUMA effects. So, probably the problem is not with your VM solution, but with your Java Dynamic Program and it will have limited scalability with any VM or with real NUMA. – Cheeks

Also: were there any users of the VM host (or in other VM) when you did timings? Please, describe your Java task, not the VM. – Cheeks

it was just me on the system, no other users/processes running – Lohse

how does it perform on the host? – Mixtec

Answer 1

Time has happened since I posted this question, just for the archive I will share my findings hoping they can help to save time for others.

It turns out that the performance issues were due to the way how VirtualBox works. Especially the relationship between the OS and the hypervisor.

The Virtual Machine (the guest OS) at the end is a single process for the host and when you modify the number of CPU's in the Virtual Machine settings what they will do is change the number of threads that the process will have to emulate the other CPU's. (at least in VirtualBox)

Having said that, when I assigned 10+ CPUs to the VM I ended up with:

• a single process with 10+ threads
• an emulated OS running hundreds of processes
• my Java code which was creating another bunch of threads

All of that together caused that the setup was saturating the host Virtual Machine process which I think it was due to the way how the host OS was handling the processes context switching

On my server, the hard limit was 7 virtual CPU's, if I added more than that it would slow down the performance of the Java software

Running the Java software outside of the VM didn't show any performance issue, it worked out of the box with 60+ isolated threads.

Thorax

I have similar experiences: forums.virtualbox.org/viewtopic.php?f=7&t=86041 A VM with 8 vCPUs under heavy load performed badly, more than 8 threads could have ran in theory, increasing to 20 vCPUs the VM behaved even worse. The host is mostly running this VM with 12 physical cores and HT enabled. With 8 vCPUs and 100% load in the VM the host had a load of 2, now with 20 vCPUs the load is higher, but mostly context switches. Up to 600'000 per second. Will give less vCPUs a try as well. – Naamann

some days I interchange a couple of emails with the VirtualBox development director and among other things he shared with me that there is a hard limit for VirtualBox CPU count which is 30, he did not recommend me to go further than plus that VirtualBox does not get along with HT and that the number of CPUs must be based on the real cores and not vCores also that VirtualBox is not optimized for server loads (management did not give them a time slice for that) and it is mostly for desktop virtualization, anyway, if you have chance, give it a look to OVM, it is better suited for server stuff :) – Lohse

Answer 2

We have almost the same setup as yours (Virtualbox running on a 48-core machine across 2 NUMA nodes).

I initially set the number of cores to the maximum supported in Virtualbox (e.g. 32), but quickly realized that one of the two NUMA nodes was always idling while the other stayed at medium loads, when the VM was under load.

Long story short, a process can only be assigned to a single NUMA node, and Virtualbox runs one user process with several threads... which means that we are limited to using 24 cores (and even less in practice considering that this is a 12-core cpu with hyperthreading).

Portable