How to benchmark virtual machines

Question

I am trying to perform a fair comparison of XenServer vs ESX and one comparison I would like to make is performance with multiple VMs. Does anyone know how to go about benchmarking VM performance in a fair way?

On each server I would like to run a fixed number of XP/Vista VMs (for example 8) and have some measure of how quickly each one runs when under load. Ideally I would like some benchmark of the overall system (CPU/Memory/Disk/Network) rather than just one aspect.

It seems to me this is actually a very tricky thing to do and obtain any meaningful results so would be grateful for any suggestions!

I would also be interested to see any existing reports or comparisons that have been published (preferably independent rather than vendor biased!)

Kassab

What's with the close votes? Certainly performance benchmarks are helpful when planning an application deployment – Haukom

@Kassab - are you looking for this in the context of a distributed web app, or similar? – Haukom

and, fwiw, I think this is a good follow-up to #495714 – Haukom

@Haukom - If this were bench overflow I'd agree with you...but its not. I fail to see the programming content... – Ideality

Answer 1

As a general answer, VMware (together with other virtualization vendors in the SPEC Virtualization sub-committee) has put together a hypervisor benchmarking suite called VMmark that is available for download. The VMmark website discusses why this benchmark may be useful for comparing hypervisors, including an FAQ and a whitepaper describing the benchmark.

That said, if you are looking for something very specific (e.g., how will it perform under your workload), you may have to roll your own variants of VMmark, especially if you are not trying to do the sorts of things that VMmark benchmarks (e.g., web servers, database servers, file servers, etc.) Nonetheless, the methodology behind its development should be of interest.

Disclaimer: I work for VMware, though not on VMmark.

Demotic

Answer 2

I don't see why you can't use common benchmarks inside the VMs: WinSAT, Passmark, Futuremark, SiSoftware, etc... Host the VMs over different hosts and see how it goes.

As an aside, benchmarks that don't closely match your intended usage may actually hinder your evaluation. Depending on the importance of getting this right, you may have to build-your-own to make it relevant.

Why do you want to bench?

Wyck

Try WinSAT in a Win7 VM sometime. Result I get is: "Error: Unable to run inside of a Virtual Machine. Please try again running directly on the native hardware." – Velazquez

Becase they tend to crash the VM. SiSandra kills my VirtualBox and my EC2 server on CPU or memory tests, for example. – Greatcoat

Answer 3

How about some anecdotal evidence?

I'm going to assume this is a test environment, because you're wanting to benchmark on XP/Vista. Please correct me if I'm wrong.

My current test environment is about 20 VMs with varying OS's (2000/XP/Vista/Vista64/Server 2008/Server 2003) in different configurations on a Dual Quad Core Xeon machine with 8Gb RAM (looking to upgrade to 16Gb soon) and the slowest machines of all are Vista primarily due to heavy disk access (this is even with Windows Defender disabled)

Recommendations - Hardware RAID. Too painful to run Vista VMs otherwise. - More RAM.

If you're benchmarking and looking to run Vista VMs, I would suggest putting your focus on benchmarking disk access. If there are going to be performance differences elsewhere I doubt they would be of anything significant.

Townshend

Consider disabling Superfetch for your Vista VMs. Forces memory up and does so by lots of memory caching of disk access. In some scenarios, this can lead to more disk access than you need. – Wyck

Also, with 20 VMs, you are generally pushing the limits on a quad-core. That's 400MB per VM, which for Vista is tremendously low. – Wyck

I'll try disabling superfetch, thanks for the tip. Vista VMs get at least 512Mb, offset by allocating only 256Mb to the Windows 2000 VMs. – Townshend

Answer 4

I recently came across VMware's ESX Performance documentation - http://www.vmware.com/pdf/VI3.5_Performance.pdf. There is quite a bit in there on improving performance and benchmarking.

Haukom