What's your approach to testing iPhone / iPad apps?

Question

When developing for iPhone/iPad do you

1. Do unit/integration/etc testing?
2. What framework(s) do you use?
3. What other framework(s) have you tried (if you decided to not use them, why not?)

---

NOTE This is based on a question asked a few days ago (that has since been heavily edited). Question generated some interesting responses that may be useful to aggregate in one place.

Appanage

Answer 1

Here is my current approach to testing before shipping an app.

1) Build and analyze using the Xcode integrated Clang analyzer.

This is helpful in many ways, it catches many little stupid things and also some memory leaks (even though sometimes its results contain false positives).

2) Check thoroughly all of the warnings. Some of them may actually be really harmful. Remove the warnings where possible.

3) Use Instruments to check for memory leaks, memory footprint at runtime, etc.

4) Use Shark to identify performance problems when I feel it is needed.

5) Execute (manually) all of a predefined series of tests, to check that both the UI and the underlying code works as expected. When adding new functionalities to an app, always do regression tests again to ensure everything runs smoothly and properly. I have discarded unit tests a long time ago, due to the unbelievable amount of time needed to setup and run them.

6) Ship an ad hoc version of the app to alpha testers, correct bugs reported, then ship the app again to both alpha and beta testers and correct bugs reported.

7) Final test done personally on different devices, at the moment iPhone 3GS, iPhone 3G and iPod Touch.

Artemas

Answer 2

Personally I only do unit-testing of classes and/or methods that makes sense to unit test and are worth the effort. In my oppinion this boils down to the following types of code:

• Pure computational algorithms
• Data parser of various sorts (files, internet data, etc.)
• Methods that have a input/output relation
• And probably one or two more I forgot about.

I seldom use unit-testing for test of GUI-related code.

I have only used the OCUnit/SenTesting framework shipped with iPhone so far.

I've also found that making a short checklist of things to do as part of the release process makes up for a lot of testing. It is often not the directly test related stuff that makes problems in the product, but also the small (forgotten) steps in the build/release process.

Manstopper

Answer 3

1. No
2. N/A
3. N/A

In all honesty, I found the amount of work required to create a single unit test to be a complete pain. There's a ton of mocking that's required even for the most basic of tests. As well, it's been difficult to separate the models, views, and controllers in iPhone. And given that my app was quite small, it wasn't worth the time.

If/When I was writing something larger, I'd re-investigate unit testing again.

With that said, I did a ton of user/QA style testing. Watching others use the app (really helpful!), me using the app, etc. You can't skimp there if you skimp on unit tests!

Lashelllasher

Do happen to know of any tools that at least help automate UI testing or is it pretty much all manual at this point? – Bruton

Re: Automation .. there is a test automation framework in 4.0 SDK which uses javascript – Kelwin

@nolim1t: You should put up an answer with information about the automation framework =). – Bruton

Answer 4

The only testing do is with the application Instruments, and thats only sometimes.

Autophyte

Any chance you can explain why (and link to the applicaton?). – Bruton

The application comes standard with the developer tools and gives you a lot of information especially on memory usage. – Autophyte

Answer 5

I've done some unit testing on my apps. But since mine is a calculator its easier to unit test vs other more complex apps. I find it epically helpful when I find a case that doesn't work right (usually from manual testing), then I write a test so that (hopefully obscure) case will always be tested in the future. That way I don't repeat bugs further down the line.

Obscuration