Xcode and Asynchronous Unit Testing

Unit Test support is greatly improved in Xcode 5, with new features such as the Test Navigator and individual test runs. XCTest is the new framework, replacing OCUnit. I’d hoped there would be better support for testing asynchronous calls with all of this new stuff, but sadly this is not the case. So, what’s the issue here?

Some methods in the iOS API work asynchronously, on a background queue, and are completed on a completion block, at some point after invoking them. Here’s an example:

In this Unit Test, we’re testing the creation and saving of a blank UIManagedDocument. The completion handler block is invoked when the operation completes and we test for a True value of the “success” block parameter. If we run the test from Xcode it will pass. It will always pass, and never fail. That’s because the test exits before the completion handler is invoked. We can check this is so by setting a breakpoint on the following line:

Run the test. The breakpoint is never reached and so we can never test the value.

So how can we ensure that the test method waits until the asynchronous method invokes the completion block? There have been a few solutions proposed on places like Stack Exchange, and after playing with a few, I use the following one.

In this solution, we need to declare a Boolean flag that indicates that we’re waiting for the operation to complete, then call the asynchronous method, and then block the run loop until the test completes when the flag is set to NO.

Here’s how we do it. We need to declare our Boolean flag before calling the asynchronous method:

Then we must set it to NO inside the method’s completion block:

And loop while the condition is true:

Here’s the full code:

Run the test again, and this time the breakpoint is reached.

This seems somewhat cumbersome, especially if we need to do this in many Unit Tests. So we can convert this into a set of Macros and declare them in a supporting header file:

Now we can simplify our test:

Using the Macros makes things much clearer. We’re simply initialising a flag to YES with the StartBlock() pseudo-function, setting it to NO with EndBlock() and waiting for the flag to be set to NO in the WaitUntilBlockCompletes() call.

It’s not the most elegant solution, but it suffices for my needs.

I’ve uploaded the Macros to GitHub as a Gist:

https://gist.github.com/Phillipus/6537635

A caveat – sometimes this solution does not work using Xcode 5 with iOS 7, especially when dealing with a UIManagedDocument. In some cases there seems to be a race condition or threading issue going on. (See the workaround in the comments for UIManagedDocument.) Also, with the iOS 7 simulator, I’m seeing more and more cases where this wait loop is not working, and some unit tests are not being run. It’s as if some tests are not waiting for the loop to break and failing to run. I’ve tried increasing the value of dateWithTimeIntervalSinceNow in this line of the macro:

But it seems as if the main UI thread seems to get stuck in some tests, and can only be unstuck by clicking on the simulator, or waiting a few seconds. Something has changed in Xcode 5 and/or iOS 7, but I don’t know what it is. If you have problems with this, experiment with the date value in this line. It may just be an issue with the simulator. Try testing on the device as well.

I still have my reservations about this technique, and I’m still looking for the perfect solution for asynchronous unit testing in Xcode. You would think that Apple might have provided a solution in XCTest, perhaps similar to the implementation in GHUnit.

Update 14 July 2014 – Good news! Apple has introduced a great new framework to support asynchronous unit testing in Xcode 6 (still in beta at the moment). I’ve written a new post on how to implement this here.

Share this...Tweet about this on TwitterShare on LinkedInShare on Facebook

11 thoughts on “Xcode and Asynchronous Unit Testing

  1. I gave it a try but then the notifications such as kReachabilityChangedNotification is causing a crash. Anyone else having the same issue?

  2. It is great.
    I use your way to test my code of Core Data and it works perfect.

    However, when I adding more than 3 tests in the test suit and testing them all, the application cannot create the document in setup() in 3rd test. But the first and second tests are pass.

    Here is my sample code: http://pastie.org/8381057
    Can you give me any advice?
    Thanks.

    • Tom, I get this error too. But only using XCTest. If I use the old SensTesting framework it’s OK. There must be a threading issue. One way round this is to use a different named UIManagedDocument for each test. Something like:

      static int count;

      - (void)createTestDocument {
      NSURL *url = [[[NSFileManager defaultManager] URLsForDirectory:NSDocumentDirectory inDomains:NSUserDomainMask] lastObject];
      url = [url URLByAppendingPathComponent:[NSString stringWithFormat:@"TestDocument%d", count++]];
      ...

      • It’s Awesome.
        It solved the issue perfectly.

        Thanks again for the test asynchronous test method.^^

  3. Thanks. Exactly what I was looking for in doing async unit testing. Much easier than importing a library / etc. I have it set up as an x-code snippet and it gets used frequently. So far .. working well.

  4. An alternative way of doing it that I think is more semantic:

    @interface Tests : XCTestCase
    @property (nonatomic) BOOL waitingForBlock;
    @end

    – (void)setUp
    {
    [super setUp];
    self.waitingForBlock = YES;
    }
    – (void)waitForBlock
    {
    while(self.waitingForBlock) {
    [NSRunLoop.currentRunLoop runMode:NSDefaultRunLoopMode
    beforeDate:[NSDate dateWithTimeIntervalSinceNow:0.1]];
    }
    }
    – (void)testBlock
    {
    [self.testWithBlock: ^ {
    XCTAssertTrue(YES, @””);
    self.waitingForBlock = NO;
    }];
    [self waitForBlock];
    }
    – (void)testOther
    {
    XCTAssertTrue(YES, @””);
    }

Comments are closed.