Suggest a test automation framework for my scenario

Question

I am in the process of evaluating few test automation frameworks and would appreciate some input. We have a large credit processing business app and it has three endpoints: web ui, soap and basic http/post. My requirement is as below.

Our QA is very much used to data-driven testing (all the input data comes from excel and so is expected result for assertions). Basically each row in the excel sheet contains both input data and corresponding expected outcome. We want our QA to just edit the excel sheets. Our development team will be responsible for providing necessary test libraries/fixtures. Given the above scenario, I would want to use 1 single automation framework that can read these excel sheets and test all the endpoints of my application i.e. web ui, soap and http/post. Please let me know which test automation framework(s) suit my needs. Appreciate some pointers and tutorials to get me started.

Cannibalism

Answer 1

I have gone through few of the suggested frameworks but the final choice I made was to write my own domain libraries (DSL if you will) on top of

1. Spock
2. Geb
3. Groovy and
4. HTTP Builder
5. Sonar and Hudson

We currently run 14000+ (and growing) regression tests every night as part of CI.

Cannibalism

Answer 2

Robotframework makes it possible to use test data that comes from external files. I haven't tried with excel sheets but I've done it with CSV formatted data as well as JSON. At worst you might have to export your excel data into a python-friendly format, but that can be done transparently to the people maintaining the spreadsheet.

In short, you can create a keyword in python (or any JVM-based language if you use jython) that reads in your spreadsheet and converts it to data usble by test cases.

Though, if absolutely all your tests are driven from this data file you don't need a framework. Just write a short script in any language that iterates over each row and calls a function that does the equivalent of curl or wget. Frameworks are useful if you need a way to represent your test scenarios but you already have that.

That being said, though, the advantage to using robot -- even if you only have one test that say "validate using each row of the spreadsheet" -- is that you get the nice reporti g that robot has, and can integrate it with jenkins/hudson pretty easily.

Glyph

Answer 3

It's recommend Fit/Fitness. It's a good way to allow business analysts to run tests on their own.

Shammer

Thanks for recommending Fit. Though I like it, the whole process of writing tests in wiki syntax is overwhelming to the QA and also we want to version control the tests – Cannibalism

Fitnesse is perfect fit for this, it can work with selenium for webui and has a restfixture that will do the soap bits. It will also work fine with any VCS, and seriously a partially trained monkey can use a wiki..... – Immediately

Answer 4

I'm not aware of a tool that would allow you to test all the endpoint with data that is stored in a spreadsheet. You would need to look at writing a custom test runner that is built on an existing test runner to allow the data to be read from spreadsheets.

However, I would focus more of picking a tool that suited the endpoint you are trying to test. For the Web UI I would recommend selenium or HtmlUnit. You would need to look at alternatives methods for get the data from the spreadsheet into these tests. But these will exercise you UI as close to a user as possible.

To test the soap endpoint I would suggest just writing unit tests using JUnit and mocking out the backend. You can test the backend separately with unit tests to ensure that this is working correctly. Depending on your application architecture this might not be an option for you. You may want to set these up as integration tests and execute them through JUnit and assert the message returned. Again you would need to write a library that got the data and expected values from a spreadsheet.

Hope this helpful for you.

Metagenesis

Answer 5

I can recommend soapUI (http://www.soapui.org/) or the JMeter Application (http://jmeter.apache.org/).

With SoapUI I've made functional tests for my (soap-) webservice. You can automate these tests.

With Apache JMeter I've done automatic performance tests on a Web UI. The test data also was based on an excel sheet.

Kind regards, patrick

Skilled

Answer 6

You may want to take a look at Twist from Thoughtworks. It's a functional testing IDE. QA/BAs can use natural language to define test cases. Twist uses Selenium and JUnit under the hood, so the generated test cases can be version controlled.

I agree with @stuartf that you're not going to find an out of the box solution that will load excel data and assert against it. You could use POI or some other excel reader to load the data in your testing framework.

Ardussi