Unit tests for memory management in Cocoa/Objective-C

Question

How would you write a unit test—using OCUnit, for instance—to ensure that objects are being released/retained properly in Cocoa/Objective-C?

A naïve way to do this would be to check the value of retainCount, but of course you should never use retainCount. Can you simply check whether an object's reference is assigned a value of nil to indicate that it has been released? Also, what guarantees do you have about the timing at which objects are actually deallocated?

I'm hoping for a concise solution of only a few lines of code, as I will probably use this extensively. There may actually be two answers: one that uses the autorelease pool, and another that does not.

To clarify, I'm not looking for a way to comprehensively test every object that I create. It's impossible to unit test any behavior comprehensively, let alone memory management. At the very least, though, it would be nice to check the behavior of released objects for regression testing (and ensure that the same memory-related bug doesn't happen twice).

About the Answers

I accepted BJ Homer's answer because I found it to be the easiest, most concise way of accomplishing what I had in mind, given the caveat that the weak pointers provided with Automatic Reference Counting aren't available in production versions of XCode (prior to 4.2?) as of July 23rd, 2011. I was also impressed to learn that

ARC can be enabled on a per-file basis; it does not require that your entire project use it. You could compile your unit tests with ARC and leave your main project on manual retain-release, and this test would still work.

That being said, for a far more detailed exploration of the potential issues involved with unit testing memory management in Objective-C, I highly recommend Peter Hosey's in-depth response.

Greg

I'm upvoting this not because I think it's a good thing to do, but because I think it will provoke a good answer, which I'm guessing will be basically "don't". – Influx

Answer 1

If you can use the newly-introduced Automatic Reference Counting (not yet available in production versions of Xcode, but documented here), then you could use weak pointers to test whether anything was over-retained.

- (void)testMemory {
    __weak id testingPointer = nil;
    id someObject = // some object with a 'foo' property

    @autoreleasepool {
        // Point the weak pointer to the thing we expect to be dealloc'd
        // when we're done.
        id theFoo = [someObject theFoo];
        testingPointer = theFoo;

        [someObject setTheFoo:somethingElse];

        // At this point, we still have a reference to 'theFoo',
        // so 'testingPointer' is still valid. We need to nil it out.
        STAssertNotNil(testingPointer, @"This will never happen, since we're still holding it.")

        theFoo = nil;
    }


    // Now the last strong reference to 'theFoo' should be gone, so 'testingPointer' will revert to nil
    STAssertNil(testingPointer, @"Something didn't release %@ when it should have", testingPointer);
}

Note that this works under ARC because of this change to the language semantics:

A retainable object pointer is either a null pointer or a pointer to a valid object.

Thus, the act of setting a pointer to nil is guaranteed to release the object it points to, and there's no way (under ARC) to release an object without removing a pointer to it.

One thing to note is that ARC can be enabled on a per-file basis; it does not require that your entire project use it. You could compile your unit tests with ARC and leave your main project on manual retain-release, and this test would still work.

The above does not detect over-releasing, but that's fairly easy to catch with NSZombieEnabled anyway.

If ARC is simply not an option, you may be able to do something similar with Mike Ash's MAZeroingWeakRef. I haven't used it much, but it seems to provide similar functionality to __weak pointers in a backwards-compatible way.

Answer

That really shouldn’t work with ARC, because you aren’t allowed to allocate autorelease pools. Use the @autorelease keyword instead. :-) – Humidity

Yeah, I considered that. But I wasn't sure if @autoreleasepool was publicly documented yet. Turns out it is. :) – Answer

why do we need theFoo = nil; shouldnt it be automatically released after the releasepool is drained? – Ruben

Yes, theFoo = nil is not necessary. I was either trying to explicitly call out what was happening when I wrote it (9 years ago), or I wasn't fully aware of the semantics at the time. – Answer

Answer 2

Can you simply check whether an object's reference is assigned a value of nil to indicate that it has been released?

No, because sending a release message to an object and assigning nil to a variable are two different and unrelated things.

The closest you can get is that assigning anything to a strong/retaining or copying property, which translates to an accessor message, causes the previous value of the property to be released (which is done by the setter). Even so, watching the value of the property—using KVO, say—does not mean you will know when the object is released; most especially, when the owning object is deallocated, you will not get a notification when it sends release directly to the owned object. You will also get a warning message in your console (because the owning object died while you were observing it), and you do not want noisy warning messages from a unit test. Plus, you would have to specifically observe every property of every object to pull this off—miss one, and you may be missing a bug.

A release message to an object has no effect on any variables that point to that object. Neither does deallocation.

This changes slightly under ARC: Weak-referencing variables will be automatically assigned nil when the referenced object goes away. That doesn't help you much, though, because strongly-referencing variables, by definition, will not: If there's a strong reference to the object, the object won't (well, shouldn't) go away, because the strong reference will (should) keep it alive. An object dying before it should is one of the problems you're looking for, not something you'll want to use as a tool.

You could theoretically create a weak reference to every object you create, but you would have to refer to every object specifically, creating a variable for it manually in your code. As you can imagine, a tremendous pain and certain to miss objects.

Also, what guarantees do you have about the timing at which objects are actually released?

An object is released by sending it a release message, so the object is released when it receives that message.

Perhaps you meant “deallocated”. Releasing merely brings it closer to that point; an object can be released many times and still have a long life ahead of it if each release merely balanced out a previous retain.

An object is deallocated when it is released for the last time. This happens immediately. The infamous retainCount doesn't even go down to 0, as many a clever person who tried to write while ([obj retainCount] > 0) [obj release]; has found out.

There may actually be two answers: one that uses the autorelease pool, and another that does not.

A solution that uses the autorelease pool only works for objects that are autoreleased; by definition, objects not autoreleased do not go into the pool. It is entirely valid, and occasionally desirable, to never autorelease certain objects (particularly those you create many thousands of). Moreover, you can't look into the pool to see what's in it and what's not, or attempt to poke each object to see if it's dead.

How would you write a unit test—using OCUnit, for instance—to ensure that objects are being released/retained properly in Cocoa/Objective-C?

The best you could do is to set NSZombieEnabled to YES in setUp and restore its previous value in tearDown. This will catch over-releases/under-retains, but not leaks of any kind.

Even if you could write a unit test that thoroughly tests memory management, it would still be imperfect because it can only test the testable code—model objects and maybe certain controllers. You could still have leaks and crashes in your application caused by view code, nib-borne references and certain options (“Release When Closed” comes to mind), and so on.

There's no out-of-application test you can write that will ensure that your application is memory-bug-free.

That said, a test like you're imagining, if it were self-contained and automatic, would be pretty cool, even if it couldn't test everything. So I hope that I'm wrong and there is a way.

Cadal

I envy your writing stamina. :) – Parasynapsis

It's a pain that it isn't trivial to profile tests for Leaks in Instruments. – Spasm

Answer 3

this is possibly not what you're looking for, but as a thought experiment I wondered if this might do something close to what you want: what if you created a mechanism to track the retain/release behavior for particular objects you wanted to test. Work it something like this:

1. create an override of NSObject dealloc
2. create a CFMutableSetRef and set up a custom retain/release functions to do nothing
3. make a unit test routine like registerForRRTracking: (id) object
4. make a unit test routine like clearRRTrackingReportingLeaks: (BOOL) report that will report any object in the set at that point in time.
5. call [tracker clearRRTrackignReportingLeaks: NO]; at the start of your unit test
6. call the register method in your unit test for every object you want to track and it'll be removed automatically on dealloc.
7. At the end of your test call the [tracker clearRRTrackingReportingLeaks: YES]; and it'll list all the objects that were not disposed of properly.

you could override NSObject alloc as well and just track everything but I imagine your set would get overly large (!!!).

Even better would be to put the CFMutableSetRef in a separate process and thus not have it impact your program runtime memory footprint overly much. Adds the complexity and runtime hit of inter-process communication though. Could use a private heap ( or zone - do those still exist?) to isolate it to a lesser degree.

Dora

Interesting! Coming from a C++ background, I didn't even think to (or even know that you could safely) override NSObject's dealloc. Could you clarify steps 1–4 with some sample code? – Greg

not today, swamped. Not that difficult though; bit of documentation readying should get you there. ObjectiveC runtime routines for swapping method implementations (#1), CoreFoundation reference for #2. #3 & #4 are just describing a couple of methods I'd put in a MemoryTracker class implementation (it would also have the CFMutableSetRef as an instance variable). If you dive into it and get stuck, post another comment here and I'll pitch in next week. – Dora

So basically, in test -setup swizzle (replace) NSObject's -init and dealloc methods to add and remove the the object respectively from a global array. By the end of -teardown the global array should be empty (you would need to wrap the test in a releasepool also). Your mileage may vary. I have actually used something like this to clean up particularly tricky retain cycles. – Spasm