Software Test Automation - Masters Thesis [closed]

Question

i am trying to write a thesis about Software Test Automation. I plan to compare the two Approaches of Recording and Programming of Test Scripts, and to discuss about several Automation Frameworks, for example Abbot, Selenium, Yemmy, FEST, etc ... Also in my Thesis will be a short overview about Softwaretesting Techniques and maybe a comparison of automated testing to software testing.

EDIT: I am planning to the aspects of testing an Application over it's GUI. So my Tests would be mostly on the Blackbox Side of the testing world. I have not planned to write about Unit Testing.

At the Moment i read pretty much about the different Automation Frameworks, but i may not have the time to review all of them. So i plan to read about them and make the Thesis more literature - based.

• Do you think this Topic could be successful?
• Do you have any other Ideas concerning this Topic?
• Can you reccommend Literature?
• What is your Opinion about this Topic?

Benavidez

You might also want to include things like DejaGNU's role in open-source automated software testing post compilation. – Potion

What scope of testing are you interested in? Developer (unit testing), functional, integration (these all have wikipedia articles on their terminology)? – Andeee

Answer 1

A survey of the literature should be a fine focus for a MS thesis. It sounds like you want to just talk about black-box GUI-driving customer-facing tools, which is a reasonably small niche.

You /might/ want to have a page or two on the whole world of test tools - unit testing, security, load, etc, as someone mentioned above. But I think you targeted your niche pretty well.

I would think with a 6-credit thesis you should have plenty of time to explore and try out some of the bigger commercial and open-source tools as well as survey the literature. I would encourage you to look into both commerical tools (quick test pro, test complete) and also keyword-driven automation - selenium RC, for example. Someone else mentioned testing "behind the GUI" eg FIT/Fitnesse, it might be worth discussing and evaluating.

I cover black-box, functional test automation in my monthly column in the December 2008 issue of software test and performance magazine:

http://www.stpmag.com/issues/stp-2008-12.pdf (page 7)

That's the one page scratch-the-surface introduction. The five-sentence introduction is that screen record/playback tools compare everything, so if your GUI changes at all, in any way (even if you just change the screen resolution) that can come back as a false error. Keyword-driven tools only check what you tell them to check - they miss if a button is suddenly disabled for no good reason or an icon is not transparent.

Only a human is good at checking that hidden assertion at the end of every test case "... and nothing else strange happened."

So computer-based test execution and evaluation can have some value, but it should be part of a balanced breakfast.

Other things to look into:

• James Bach's "Software Test Automation Snake Oil"
• Kaner, Bach and Pettichord's book "Lessons Learned in Software Testing"
• My blog post on test Frameworks - http://xndev.blogspot.com/2007/09/whats-test-framework.html (it's the number 4 google result for "what is a test framework", so I'm comfortable recommending it)
• The minefield analogy ( http://www.testingperspective.com/tpwiki/doku.php?id=minefield )
• The papers of Doug Hoffman on test automation: http://www.softwarequalitymethods.com/H-Papers.html
• The classic "shelfware" problem of test automation
• The anti-intellectualism pushed by some proponents of the blackbox test automation community
• Kaner's Black Box Software Testing Course
• James Bach's work on /cognitive/ testing
• Context Driven Software Testing
• Jon Kohl's work on "Man and Machine", or the cyborg approach (instead of computer-alone test execution and evaluation)

I hope that helps.

Umpteen

Matt, +1, great answer. But if by "record/playback tools compare everything, so if your GUI changes" you are referring to QTP and its brethren, I think you are off the mark. Current generation tools are out of the box resilient against simple screen res changes and object moves. You are right, but record/playback usually fails for other reasons - such as the app underwent more radical change. Developers typically do not just change a button, rather the workflow changes in some significant way. Hence the need for sound development practices to keep things maintainable. – Lingua

Answer 2

Software Test Automation is a big topic, and you may want to narrow your focus rather than attempt to cover a mix of frameworks, playback/record, overview of techniques, automated vs. not.

Entire books have been written about software test automation:

• As a general topic
• Focusing on functional/feature tests (FIT)
• Focusing on unit tests
• Focusing on unit tests using one particular language and framework

Frameworks are aimed at different types of testing:

• Unit testing
  • Test-Driven Development
  • Behavior-Driven Development
• Feature/Functional testing
• GUI testing (Windows, Java GUIs, X Windows, etc.)
• Web testing
• Performance Testing
• Security Testing

I would consider focusing on frameworks (or techniques, or whatever) in one of these areas rather than trying to cover them all. Or pick a couple of these areas and contrast them.

The issue of playback/record vs. handwritten tests seems old to me. In the 1980's vendors liked to push playback/record for Windows GUI automation. It made for great demos and high hopes. But it also made for brittle tests and shelfware. Playback/record is nice to get you started with a tool, but to be maintainable, you generally need scripts written at a higher level. That ushered in a new era of spreadsheet and keyword-based approaches, and eventually FIT/FitNesse.

Effort

In fairness, quite a few Masters programs are very accepting of this sort of survey thesis. – Passe

Thank you for your comment! I added some Details to my original Question, i was so focussed on my topic that i forgot that there are other methods of Test Automation instead of GUI Testing. – Benavidez

Answer 3

I don't know about literature, but I think the ACM publications in your school library would probably produce results. Particularly the SIG* newsletters. (Perhaps SIGSOFT?)

It does sound like a good Master's thesis to me. Of course your thesis advisor is the final word on that. You should go talk to them.

Maurits

Answer 4

As a literature-based review, this makes an excellent topic; there's plenty of material out there. Obviously I'm not going to start to get into all the details of that, since that's your job as the author. :-)

However, though I'm not familiar with the original research requirements for a master's thesis, this certainly wouldn't be enough for a PhD thesis. I would look for original work you could add to this. One idea would be a taxonomy of testing methods and systems. You might also examine the role of testing as compared to formal verification.

Derte

Yes, most terminal master's programs in North America do not require empirical research. Mine did not. A literature survey that takes a few stands is usually fine. This is a 3-6 credit project, not a 21+ credit project. :-) – Umpteen

Answer 5

I'd be interested in reading the thesis if it's available online. Worth considering programmatic access to GUI - both web and application. Then there's record and playback tools like Selenium or WatiR. And of course the pros and cons of automation - limitations of the tools (most can't see into java applets or flash on webpages, for example) and the most important thing some people forget when automating - NOT everything should be automated!

But if at all possible for you to comment on this to notify us when it's done, I'd genuinely like a read.

Norry

Answer 6

An excellent book on test automation has just been published this year: “Implementing Automated Testing”, Elfriede Dustin, Thom Garrett & Bernie Gauf, Addison Wesley.

Hypothec