How to unit test asynchronous APIs?

Question

I have installed Google Toolbox for Mac into Xcode and followed the instructions to set up unit testing found here.

It all works great, and I can test my synchronous methods on all my objects absolutely fine. However, most of the complex APIs I actually want to test return results asynchronously via calling a method on a delegate - for example a call to a file download and update system will return immediately and then run a -fileDownloadDidComplete: method when the file finishes downloading.

How would I test this as a unit test?

It seems like I'd want to the testDownload function, or at least the test framework to 'wait' for fileDownloadDidComplete: method to run.

EDIT: I've now switched to using the XCode built-in XCTest system and have found that TVRSMonitor on Github provides a dead easy way to use semaphores to wait for async operations to complete.

For example:

- (void)testLogin {
  TRVSMonitor *monitor = [TRVSMonitor monitor];
  __block NSString *theToken;

  [[Server instance] loginWithUsername:@"foo" password:@"bar"
                               success:^(NSString *token) {
                                   theToken = token;
                                   [monitor signal];
                               }

                               failure:^(NSError *error) {
                                   [monitor signal];
                               }];

  [monitor wait];

  XCTAssert(theToken, @"Getting token");
}

Beanery

I changed the question wording to be much more generic, as I would like to hear from other people general approaches to unit testing asynchronous operations. – Horseweed

PLEASE everybody, SEE answer from "Thomas Tempelmann" at the bottom – Gow

Answer 1

I ran into the same question and found a different solution that works for me.

I use the "old school" approach for turning async operations into a sync flow by using a semaphore as follows:

// create the object that will perform an async operation
MyConnection *conn = [MyConnection new];
STAssertNotNil (conn, @"MyConnection init failed");

// create the semaphore and lock it once before we start
// the async operation
NSConditionLock *tl = [NSConditionLock new];
self.theLock = tl;
[tl release];    

// start the async operation
self.testState = 0;
[conn doItAsyncWithDelegate:self];

// now lock the semaphore - which will block this thread until
// [self.theLock unlockWithCondition:1] gets invoked
[self.theLock lockWhenCondition:1];

// make sure the async callback did in fact happen by
// checking whether it modified a variable
STAssertTrue (self.testState != 0, @"delegate did not get called");

// we're done
[self.theLock release]; self.theLock = nil;
[conn release];

Make sure to invoke

[self.theLock unlockWithCondition:1];

In the delegate(s) then.

Georgiana

And what if it never unlocks... ? – Fugue

@Julian - Huh? You're the programmer. You make sure it unlocks. That's part of the algorithm. To clarify: Your delegate method is supposed to invoke "[self.theLock unlockWithCondition:1];". And the calling of the delegate method is ensured by whatever you call, right? If that delegate never gets called, well, then you've found a bug. – Georgiana

So the timeout should happen in the Code under Test. – Hew

this answer deserves far much credit, as its the best ansewer, and also it showed me, how to make any asynch funct, synch... thx!!! – Gow

and what if the delegate unlocks before the main guy locks? – De

@Dirty Henry: It just works :) That's because the lockWhenCondition call will then find that the lock is not engaged and therefore will fall thru. – Georgiana

This won't work on any async function that relies on calling the delegate callback via the main runloop since the runloop is blocked while waiting on the condition. – Realtor

@MattConnolly, that's a good point. Also, being a rather old school programmer, I hadn't realized when writing this response that the unit test system expects to get "failed" answers. My usual testing rather works like this: I run automatic tests and if any of them gets stuck (i.e. it would deadlock in my above example), I look for the problem right away --- Anyway, I agree that different runloops might not play well with my approach. Therefore, take my suggestion as a guide and everyone better test whether it works in their specific case. – Georgiana

@MattConnolly : this answer doesn't lock the main thread : https://mcmap.net/q/120017/-sentestingkit-in-xcode-4-asynchronous-testing – Biserrate

Thanks Ben, that's very close to how i do it. Note that you can simply use a flag instead of a semaphore since we're only running on the main thread. – Realtor

@MattConnolly : I sometimes have xcode reporting "Test succeeded" immediately, waiting for a bit, then displaying errors for my test in the event log. Do you have the same issue (i tried with both semaphore and flag) ? Note that i'm testing a lib, not an app. – Biserrate

@MattConnolly : it seems like using big nslog in the main thread for tests may have this weird side-effect.. Not sure it's related to that NSRun loop issue, so just forget about my comment. – Biserrate

@MattConnolly: Thanks for the warning about the main thread. A quick search through the code to be tested showed a dispatch back to the main thread. Lots of hair-pulling avoided! – Kowalczyk

There is a key point:[self.theLock unlockWithCondition:1]; must be called on the other thread, not the same with the thread call: [self.theLock lockWhenCondition:1]; – Scriabin

Answer 2

I appreciate that this question was asked and answered almost a year ago, but I can't help but disagree with the given answers. Testing asynchronous operations, particularly network operations, is a very common requirement, and is important to get right. In the given example, if you depend on actual network responses you lose some of the important value of your tests. Specifically, your tests become dependent on the availability and functional correctness of the server you're communicating with; this dependency makes your tests

• more fragile (what happens if the server goes down?)
• less comprehensive (how do you consistently test a failure response, or network error?)
• significantly slower imagine testing this:

Unit tests should run in fractions of a second. If you have to wait for a multi-second network response each time you run your tests then you're less likely to run them frequently.

Unit testing is largely about encapsulating dependencies; from the point of view of your code under test, two things happen:

1. Your method initiates a network request, probably by instantiating an NSURLConnection.
2. The delegate you specified receives a response via certain method calls.

Your delegate doesn't, or shouldn't, care where the response came from, whether from an actual response from a remote server or from your test code. You can take advantage of this to test asynchronous operations by simply generating the responses yourself. Your tests will run much faster, and you can reliably test success or failure responses.

This isn't to say you shouldn't run tests against the real web service you're working with, but those are integration tests and belong in their own test suite. Failures in that suite may mean the web service has changes, or is simply down. Since they're more fragile, automating them tends to have less value than automating your unit tests.

Regarding how exactly to go about testing asynchronous responses to a network request, you have a couple options. You could simply test the delegate in isolation by calling the methods directly (e.g. [someDelegate connection:connection didReceiveResponse:someResponse]). This will work somewhat, but is slightly wrong. The delegate your object provides may be just one of multiple objects in the delegate chain for a specific NSURLConnection object; if you call your delegate's methods directly you may be missing some key piece of functionality provided by another delegate further up the chain. As a better alternative, you can stub the NSURLConnection object you create and have it send the response messages to its entire delegate chain. There are libraries that will reopen NSURLConnection (amongst other classes) and do this for you. For example: https://github.com/pivotal/PivotalCoreKit/blob/master/SpecHelperLib/Extensions/NSURLConnection%2BSpec.m

Kirstenkirsteni

I looked for information about this issue, and I definitely agree with you. Nonetheless I used St3fan's method, since I need to test if an image is correctly displayed in a WebView. Thanks to your answer, I don't download it from the internet, but I still have to wait for the local file to be loaded (in another thread). Anyways, thank you very much to you two guys ! – Fugue

@Adam, Your explanation is great. Simply loved your vivid style. – Lonnie

Answer 3

St3fan, you are a genius. Thanks a lot!

This is how I did it using your suggestion.

'Downloader' defines a protocol with a method DownloadDidComplete that fires on completion. There's a BOOL member variable 'downloadComplete' that is used to terminate the run loop.

-(void) testDownloader {
 downloadComplete = NO;
 Downloader* downloader = [[Downloader alloc] init] delegate:self];

 // ... irrelevant downloader setup code removed ...

 NSRunLoop *theRL = [NSRunLoop currentRunLoop];

 // Begin a run loop terminated when the downloadComplete it set to true
 while (!downloadComplete && [theRL runMode:NSDefaultRunLoopMode beforeDate:[NSDate distantFuture]]);

}


-(void) DownloaderDidComplete:(Downloader*) downloader withErrors:(int) errors {
    downloadComplete = YES;

    STAssertNotEquals(errors, 0, @"There were errors downloading!");
}

The run-loop could potentially run forever of course.. I'll improve that later!

Beanery

This works very nicely! You can just change beforeDate: to something like 30s into the future to avoid running forever. – Floorman

I recommend you to use [NSDate dateWithTimeIntervalSinceNow:1.0f] as beforeDate argument. – Leastwise

@RomanTruba why? It is concise, and the point is, allow the run loop to process some events and return. Are you looking to return constantly? I'm guessing the if you're writing 'unit tests' with a run loop you're really not running unit tests. – Rudolph

@CameronLowellPalmer now I don't remember why. Maybe there was a problem with something – Leastwise

@RomanTruba I know that feeling. Maybe you wanted to guarantee you could bail as soon as possible rather than waiting on an event to allow the loop to exit. – Rudolph

Answer 4

I wrote a little helper that makes it easy to test asynchronous API. First the helper:

static inline void hxRunInMainLoop(void(^block)(BOOL *done)) {
    __block BOOL done = NO;
    block(&done);
    while (!done) {
        [[NSRunLoop mainRunLoop] runUntilDate:
            [NSDate dateWithTimeIntervalSinceNow:.1]];
    }
}

You can use it like this:

hxRunInMainLoop(^(BOOL *done) {
    [MyAsyncThingWithBlock block:^() {
        /* Your test conditions */
        *done = YES;
    }];
});

It will only continue if done becomes TRUE, so make sure to set it once completed. Of course you could add a timeout to the helper if you like,

Onus

This is the best and most useful answer so far! Thank you! – Bought

Blocking main thread using while loop is crazy in app product code, but it's OK for me to use this code in UT. Thanks! – Thigh

Answer 5

This is tricky. I think you will need to setup a runloop in your test and also the ability to specify that runloop to your async code. Otherwise the callbacks won't happen since they are executed on a runloop.

I guess you could just run the runloop for s short duration in a loop. And let the callback set some shared status variable. Or maybe even simply ask the callback to terminate the runloop. That way you you know the test is over. You should be able to check for timeouts by stoppng the loop after a certain time. If that happens then a timeout ocurred.

I've never done this but I will have to soon I think. Please do share your results :-)

Whippersnapper

Answer 6

If you're using a library such as AFNetworking or ASIHTTPRequest and have your requests managed via a NSOperation (or subclass with those libraries) then it's easy to test them against a test/dev server with an NSOperationQueue:

In test:

// create request operation

NSOperationQueue* queue = [[NSOperationQueue alloc] init];
[queue addOperation:request];
[queue waitUntilAllOperationsAreFinished];

// verify response

This essentially runs a runloop until the operation has completed, allowing all callbacks to occur on background threads as they normally would.

Realtor

So glad I found your response here - sure saved me a bunch of headaches - thnx :) – Someway

Note that waitUntilAllOperationsAreFinished only waits until the current thread is finished. If you're using an AFNetworking class method like AFJSONRequestOperation then there are additional blocks (success, failure) which will run on different threads. – Beatriz

AFNetworking handles its callbacks from the OS on whatever arbitrary queue the OS has chosen, and by default calls your callback on the main queue. (You can tell AFNetworking to use a specific background queue for your callbacks from AFNetworking if you like). – Realtor

Answer 7

To elaborate on @St3fan's solution, you can try this after initiating the request:

- (BOOL)waitForCompletion:(NSTimeInterval)timeoutSecs
{
    NSDate *timeoutDate = [NSDate dateWithTimeIntervalSinceNow:timeoutSecs];

    do
    {
        [[NSRunLoop currentRunLoop] runMode:NSDefaultRunLoopMode beforeDate:timeoutDate];
        if ([timeoutDate timeIntervalSinceNow] < 0.0)
        {
            break;
        }
    }
    while (!done);

    return done;
}

Another way:

//block the thread in 0.1 second increment, until one of callbacks is received.
    NSRunLoop *theRL = [NSRunLoop currentRunLoop];

    //setup timeout
    float waitIncrement = 0.1f;
    int timeoutCounter  = (int)(30 / waitIncrement); //30 sec timeout
    BOOL controlConditionReached = NO;


    // Begin a run loop terminated when the downloadComplete it set to true
    while (controlConditionReached == NO)
    {

        [theRL runMode:NSDefaultRunLoopMode beforeDate:[NSDate dateWithTimeIntervalSinceNow:waitIncrement]];
        //control condition is set in one of your async operation delegate methods or blocks
        controlConditionReached = self.downloadComplete || self.downloadFailed ;

        //if there's no response - timeout after some time
        if(--timeoutCounter <= 0)
        {
            break;
        }
    }

Tabriz

Answer 8

I find it very convenient to use https://github.com/premosystems/XCAsyncTestCase

It adds three very handy methods to XCTestCase

@interface XCTestCase (AsyncTesting)

- (void)waitForStatus:(XCTAsyncTestCaseStatus)status timeout:(NSTimeInterval)timeout;
- (void)waitForTimeout:(NSTimeInterval)timeout;
- (void)notify:(XCTAsyncTestCaseStatus)status;

@end

that allow very clean tests. An example from the project itself:

- (void)testAsyncWithDelegate
{
    NSURLRequest *request = [NSURLRequest requestWithURL:[NSURL URLWithString:@"http://www.google.com"]];
    [NSURLConnection connectionWithRequest:request delegate:self];
    [self waitForStatus:XCTAsyncTestCaseStatusSucceeded timeout:10.0];
}

- (void)connectionDidFinishLoading:(NSURLConnection *)connection
{
    NSLog(@"Request Finished!");
    [self notify:XCTAsyncTestCaseStatusSucceeded];
}

- (void)connection:(NSURLConnection *)connection didFailWithError:(NSError *)error
{
    NSLog(@"Request failed with error: %@", error);
    [self notify:XCTAsyncTestCaseStatusFailed];
}

Sashasashay

Answer 9

I implemented the solution proposed by Thomas Tempelmann and overall it works fine for me.

However, there is a gotcha. Suppose the unit to be tested contains the following code:

dispatch_async(dispatch_get_main_queue(), ^{
    [self performSelector:selector withObject:nil afterDelay:1.0];
});

The selector may never be called as we told the main thread to lock until the test completes:

[testBase.lock lockWhenCondition:1];

Overall, we could get rid of the NSConditionLock altogether and simply use the GHAsyncTestCase class instead.

This is how I use it in my code:

@interface NumericTestTests : GHAsyncTestCase { }

@end

@implementation NumericTestTests {
    BOOL passed;
}

- (void)setUp
{
    passed = NO;
}

- (void)testMe {

    [self prepare];

    MyTest *test = [MyTest new];
    [test run: ^(NSError *error, double value) {
        passed = YES;
        [self notify:kGHUnitWaitStatusSuccess];
    }];
    [test runTest:fakeTest];

    [self waitForStatus:kGHUnitWaitStatusSuccess timeout:5.0];

    GHAssertTrue(passed, @"Completion handler not called");
}

Much cleaner and doesn't block the main thread.

Archducal

Answer 10

I just wrote a blog entry about this (in fact I started a blog because I thought this was an interesting topic). I ended up using method swizzling so I can call the completion handler using any arguments I want without waiting, which seemed good for unit testing. Something like this:

- (void)swizzledGeocodeAddressString:(NSString *)addressString completionHandler:(CLGeocodeCompletionHandler)completionHandler
{
    completionHandler(nil, nil); //You can test various arguments for the handler here.
}

- (void)testGeocodeFlagsComplete
{
    //Swizzle the geocodeAddressString with our own method.
    Method originalMethod = class_getInstanceMethod([CLGeocoder class], @selector(geocodeAddressString:completionHandler:));
    Method swizzleMethod = class_getInstanceMethod([self class], @selector(swizzledGeocodeAddressString:completionHandler:));
    method_exchangeImplementations(originalMethod, swizzleMethod);

    MyGeocoder * myGeocoder = [[MyGeocoder alloc] init];
    [myGeocoder geocodeAddress]; //the completion handler is called synchronously in here.

    //Deswizzle the methods!
    method_exchangeImplementations(swizzleMethod, originalMethod);

    STAssertTrue(myGeocoder.geocoded, @"Should flag as geocoded when complete.");//You can test the completion handler code here. 
}

blog entry for anyone that cares.

Withdraw

Answer 11

Looks like Xcode 6 will solve the issue. https://developer.apple.com/library/prerelease/ios/documentation/DeveloperTools/Conceptual/testing_with_xcode/testing_3_writing_test_classes/testing_3_writing_test_classes.html

Granoff

Exactly what I was looking for, and preventing me from "rolling my own". – Calmas

This link is invalid. – Kantos

Answer 12

My answer is that unit testing, conceptually, is not suitable for testing asynch operations. An asynch operation, such as a request to the server and the handling of the response, happens not in one unit but in two units.

To relate the response to the request you must either somehow block execution between the two units, or maintain global data. If you block execution then your program is not executing normally, and if you maintain global data you have added extraneous functionality that may itself contain errors. Either solution violates the whole idea of unit testing and requires you to insert special testing code into your application; and then after your unit testing, you will still have to turn off your testing code and do old-fashioned "manual" testing. The time and effort spent on unit testing is then at least partly wasted.

Kaolin