How to test 1000 clients windows 7 clients

Question

We have .net clients running on Windows 7 embedded with sql server express. These run disconnected and every so often will dial in to a central server and dump their information and pull down pertinent new info. We want to do a stress test of the entire distributed system and are trying to think of a cost effective way to do this. One of our devs had the idea to use amazon cloud and spin up a 1000 clients, but they only support windows server O/S and looks like it could run into some money if you have to re-run the test multiple times which is likely.

We thought about simulating the 1000 clients all on one machine, which would pretend to have a 1000 clients and make all the calls asynchronously, but if you are running on one machine some of your ceilings will be RAM and processor of one machine versus a larger system, so I'm not sure how "real" of a picture it will give us.

All the calls happen with WCF between the client and server.

Anyone have any experience in this area and if so how did you tackle it?

thanks,

Super Timmy

Rem

Answer 1

Have you looked at SOASTA? They provide cloud-based SOA loadtesting.

Mirandamire

One thing I haven't figured out on the "cloud" stuff yet - like in case of SOASTA, do they give you ability to essentially load a windows 7 machine and then clone it as many times as you want. Or is it more like a shared hosting environment where they give you access to a virtual directory and only let you load some files? – Rem

Given that it's a service interface, the test side doesn't really need to be Windows 7 clients, does it? You're testing the API, so given a workflow and payload, you should be able to accomplish appropriate loadtesting, yes? soasta.com/technology/cloudtest-builder/wsdl-recording – Mirandamire

Answer 2

Divide and conquer, see if you can get the target test server attached to the same switch as your developers. Use divide the clients across those machines and get everyone to run against the test server. (sticking to the same switch removes network effects from distorting the results on the hardware)

Developers usually have pretty grunty machines and there are normally a few of them. That way you will not destroy the test machine and can monitor what occurs.

Ideally you would be able to use Visual Studio 2010 Load Test agent on the dev machines. This tool will give you the ability to monitor stats from all the machine in the test and give a really good picture of how your server responds.

If the machines running test test host a webpage on localhost that calls the required web service may be the best bet for getting the test going.

This depends on the exact nature of your dev environment and Visual Studio licencing arrangments.

One of the key things to figure out is how many requests/second each client would make of the WCF web service. Once you have that number you can divide that number be the maximum requests/second of each test machine in order to figure out how many test machines are required.

For the purposes of load testing, you do not need to have "actual" clients of your web service. As the web service is an interface you can simulate the activity of a normal client and run that without the normal "think time" pauses giving you the ability test the load of 1000 clients without needing an instance of each client running.

E.g. if each client accesses the web service once every minute for a total of 1000/60 = 16.7 calls per second. IF each call completes in 3 seconds, 50 virtual users would be able to create the same load (50/3=16.7). From experience, two machines running Visual Studio load test would be able to generate the load and not overload thier CPU.

Metabolism

That sounds like a good worse case scenario, but I was hoping to run a lot more instances than the devs have computers. The other problem is that the server is managed at rackspace so I don't have physical access to it. – Rem

A single dev machine should be able to call the WCF webservice at the most once every 2-3 seconds/virtual users, giving you a throughput of at least 50 requests per second. – Metabolism

Answer 3

If you are using WCF with a text-based protocol over HTTP (which seems likely), then you have a number of tools available to you. Most Web Load Testing tools can record HTTP traffic by acting as a proxy to a browser, which typically allows for recording from rich clients as well. You may need to do some additional customization of your testcases since the tools are optimized to simulate browsers. We've done this for a few of our clients and it worked pretty well.

A number of the load testing tools support generating load from the cloud (including ours - Web Performance Load Tester - and others like BrowserMob, LoadStorm, etc) so start your evaluation with those - it will making simulating 1000 virtual users very easy and affordable as well.

Amos