Automated Browser Testing using selenium, Nunit, Selenium Grid, C#, webdriver/remote control

Question

I have been researching on how to automate browser testing for a number of weeks now using all kind of different methods. Seleniums website is very vague on which is the best route to take.

I have installed

1. Selenium Webdriver
2. Remote Control
3. Selenium Grid
4. Apache Ant
5. Nunit (and pretty much everything else you could need to do this)

I finally give up trying on my own and want to know the best way to do this.

I want to be able to

1. Load the same webpage on a number of different browsers
2. Load the same webpage on a number of virtual machines(which I have set up)
3. Be able to take snapshots comparing the different browser results.

I have knowledge of programming in C# and would prefer to run my tests through Nunit.

Can anyone give me directions or point me to a website that already has them? Thank you.

Sourwood

When you say 1. Load the same webpage on a number of different browsers do you mean at once? or sequentially? – Athens

At once. So multiple browser open loading the same webpage. – Sourwood

Is it possible for you to switch to MBunit? After some investigation I came to conclusion that Nunit is not the best tool for parallelization. – Guayule

Answer 1

I have built up a test framework using junit with Selenium WebDriver that satisfies every one of your points. While its not exactly what you're asking for, I feel it may be beneficial to you regardless.

Load the same webpage on a number of different browsers

Using Selenium's grid, this is very simple to set up. Set up some virtual machines with the environments you're looking to test in. In our environment, for example, we have a grid running with four nodes (as virtual machines) with a setup like the following

• Windows with IE7 and FireFox
• Windows with IE8 and FireFox
• Windows with IE9 and Firefox
• Linux with FireFox

Note that Selenium recommends that only one instance of IE be allowed to run on the Windows nodes. On each of the aforementioned nodes, there is one instance of the specified IE and five instances of the specified FF allowed to run at any given time. With the grid setup and the hub configured, firing off tests is a breeze. In WebDriver, use the DesiredCapabilities object to set up the desired environment and then just send the test off and wait for the result to return.

Platform desiredPlatform;
DesiredCapabilities desiredCapabilities;
desiredPlatform = Platform.LINUX;
desiredCapabilities = DesiredCapabilities.firefox();
desiredCapabilities.setPlatform(desiredPlatform);
desiredCapabilities.setVersion("11");

WebDriver driver = new RemoteWebDriver("http://hubURL", desiredCapabilities);

Load the same webpage on a number of virtual machines(which I have set up)

I solved this one by forcing the tests to run, albeit in an unconvential way, in a threaded manner. Each JUnit test uses a shared thread library I put together which creates all the necessary RemoteWebDrivers needed in separate threads. Each of these threads runs simultaneously on its node while the parent thread sits and waits for all to terminate. Then on to the next test which is run multithreaded as well.

There were a couple problems I encountered such as retrieving the Junit stack traces in all of the child threads. I solved this by redirecting Std.err to a bytestream on the parent thread. All errors get routed to that stream which I then convert to a string and print out to Std.out at the end of each test. The html pages generated at the end of the tests include Std.out which worked out perfectly.

Be able to take snapshots comparing the different browser results

While I have gotten this to work, there are some inherent problems with grabbing screenshots remotely. IE will return black screenshots if the process is running as a service. The workaround was to just run the jar from the command line and keep the user logged in, in which case the screenshots return correctly. This is a known issue in the browser and there really is no nice solution to the problem. Taking screenshots works roughly like this

WebDriver augmentedDriver = new Augmenter().augment(driver);
TakesScreenshot ss = (TakesScreenshot) augmentedDriver;
String base64Screenshot = ss.getScreenshotAs(OutputType.BASE64);
byte[] decodedScreenshot = Base64.decodeBase64(base64Screenshot.getBytes());
FileOutputStream fos = new FileOutputStream(new File(imageName));
fos.write(decodedScreenshot);
fos.close();

which saves the captured screenshot from the remote machine's running browser onto the local machine.

In reality, browser automation is still struggling to stabilize itself. There are a number of important features, such as the ones you're asking about, that just aren't implemented solidly that I know of in any framework. With time, though, I'm sure a lot of this will settle down and QA developers everywhere will rejoice.

Ketcham

I've noticed your work on a couple other questions and you seem very skilled at automated testing. Do you by chance, have a tutorial or a published code open to the public? – Sourwood

Answer 2

As for the 2nd point: instead of using Grid you can let your continuous integration server do the job. In my company we use Jenkins and so called Configuration Matrix - it let's you run the same job on multiple Jenkins nodes.

As for the 1st one, I think Jenkins could be helpful here too. You can run multiple jobs on the same node. Although I've never tried that so I am not perfectly sure. And this is just an idea, I wouldn't really recommend such solution. You may also want to read this blog post describing how to run test in parallel using Selenium Grid. For people using Java I would recommend reading about parallel tests with TestNG.

Your third point is a little bit vague. What do you mean by snapshot? And what kind of result you want to compare?

Punctilio

Answer 3

Selenium RC is outdated and webdriver is more reliable way of creating selenium tests. I see the responses above cater more on java side. Below mentioned is more information on how to achieve the questions asked here using C# and selenium webdriver

On how to setup the IDE (VS express), nUnit and selenium refer How to setup C#,nUnit and selenium client drivers on VSExpress for Automated tests

On Creating simple script that launches a browser does few steps refer Creating Basic Selenium web driver test case using Nunit and C#

On how to Load the same webpage on a number of different browsers suggest referring How to invoke locally different types of browser driver using selenium and c#

On Load the same webpage on a number of virtual machines(which I have set up) for this, you need to use Remote webdriver instead of normal webdriver. Also with remote webdriver, you can launch different types of browser. Refer this webpage How to invoke/run different type of web driver browser using remote webdriver in C#

To take snapshot on different browser you can refer the link Capturing screen shots using remote/local webdriver in C#/Selenium webdriver

Overlook

Answer 4

You might also consider the free Telerik Testing Framework. This is the underpinning for Telerik's commercial Test Studio product. The Testing Framework provides cross-browser support, does a great job with dynamic content situations (AJAX), and also lets you handle OS-level dialogs like file upload/download dialogs. You can also take snapshots of the browser at any point.

You can wrap the framework inside whatever runner you prefer. I've used NUnit and MbUnit without trouble.

There's also an option for a support package if you need help with your automation.

(Disclosure: I work for Telerik as their Test Studio evangelist)

Chuch