multiple asynchronous tests and expectation

Question

I have multiple tests and each test is testing the same asynchronous method for different results with given parameters.

I found out for asynchronous tests we have to declare an expectation, wait for expectation, and fulfil the expectation. This is fine. Each test works out correctly when done separately, but when I try to run the whole test class some tests pass and others crash or fail when they run and pass normally.

I've looked all over online for "swift 3 multiple tests with expectation" and everyone who explains expectation only ever has an example in one test method. Is it not possible to have expectations in multiple methods in the same class?

An example of a test is as follows:

func testLoginWrongUsernameOrPasswordFailure() {
  let viewModel = LoginViewModel()
  let loginAPI = APIManager()
  let expect = expectation(description: "testing for incorrect credentials")
        
  viewModel.loginWith(username: "qwerty", password: "qwerty", completion: { loginCompletion in
            
      do {
        try loginCompletion()
          XCTFail("Wrong Login didn't error")
          expect.fulfill()
        } catch let error {
          XCTAssertEqual(error as? LoginError, LoginError.wrongCredentials)
          expect.fulfill()
        }
      })
        
      waitForExpectations(timeout: 10) { error in
        XCTAssertNil(error)
      }
}

As far as I'm aware, this is the correct use of expectation and each test follows the same pattern

As requested by Rob I will provide an MCVE here https://bitbucket.org/chirone/mcve_test The test classes use a mock API Manager but when I was testing with the real one the errors still occurred.

As an explanation for the code, the view-model communicates with a given API manager who invokes a server and gives back the response to the view-model for him to interpret the errors or success.

The first test tests for empty fields, something that the view-model validates rather than the APIManager. The second test tests for incorrect username and password The third test tests for valid username and password

The three tests run separately will run fine, however when the whole file is run I will get a SIGABRT error with the following reasons:

XCTAssertEqual failed: ("Optional(MCVE.LoginError.wrongCredentials)") is not equal to ("Optional(MCVE.LoginError.emptyFields)") -

*** Assertion failure in -[XCTestExpectation fulfill], /Library/Caches/com.apple.xbs/Sources/XCTest_Sim/XCTest-12124/Sources/XCTestFramework/Async/XCTestExpectation.m:101

*** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'API violation - multiple calls made to -[XCTestExpectation fulfill] for testing for empty fields.'

The SIGABRT happens usually on the second test method and if you hit play then it fails on one of the XCTest methods claiming the error it got is not the error it was expecting.

I hope the MCVE helps explain my problem.

Franchot

When you say "setup", are you referring to the setUp method or are you using the term more generally (i.e. you call testLoginWrongUsername at the beginning of the test, itself, not from setUp)? – Torques

Can you share the details of the crash? Even better, can you share MCVE? FYI, I can't think of anything that would be inherently problematic with multiple asynchronous tests. Those of us who do asynchronous tests invariably have many of them, without incident. If you have problems stemming from multiple tests, I'd generally suspect singletons or other static variables (which is why we try to avoid them), not the fact that they happen to be asynchronous. – Torques

Hi Rob, I've added an MCVE as requested. I realise now the ambiguity of my use of the word "setup" and have removed it from the post for clarification. I too thought maybe it was an issue caused by a singleton or static variables, but making the API manager an instance didn't help either. – Franchot

Hey Chirone, Is your query resolved? I'm also facing the same issue. Could you please post your answer, if already resolved? – Tetraploid

I Observed, when my first test starts running in test class it access singleton instance perform certain functionality, and it waits for the expectation. In the mean time it starts running my other test async in same class which further resets one of my singleton objects functionality. – Tetraploid

Hi Mohnish, I didn't find a solution to it. I ended up not writing the tests. – Franchot

Answer 1

If you have multiple tests (methods) in one XCTestCase don't use

let expectation = expectation(description: "")

Instead, use

let expectation = XCTestExpectation(description: "")

The self.expectaion() is shared between XCTestCase tests. In some cases, it brings to weird behavior. For example, you can get an API violation error even if you fulfill the expectation zero times.

Serg

Do you have a source for this that you could add? The closest thing I see is in the docstring for the expectation(description:) method, it says "Creates and returns an expectation associated with the test case." – Incinerator

@Incinerator I don't have any sources. I also doubt there is a bug in the XCTest package. So it's just a dirty quick fix. – Serg

Answer 2

Refactored the code as given below.

func testLoginWrongUsernameOrPasswordFailure() {
  let viewModel = LoginViewModel()
  let loginAPI = APIManager()
  let expect = expectation(description: "testing for incorrect credentials")

  viewModel.loginWith(username: "qwerty", password: "qwerty", completion: { loginCompletion in

      do {
        try loginCompletion()
        XCTFail("Wrong Login didn't error")

      } catch let error {
        XCTAssertEqual(error as? LoginError, LoginError.wrongCredentials)
      }
      expect.fulfill()
   })

  waitForExpectations(timeout: 10) { error in
    XCTAssertNil(error)
  }
}

If you are still getting the following crash, that means your completion handler for the asynchronous code is calling multiple time. And there by invoking expect.fulfill() multiple times. Which is not allowed.

*** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'API violation - multiple calls made to -[XCTestExpectation fulfill] for testing for empty fields.'

For an expectation fulfill() should call only once. If there are some rare scenarios and you need to invoke expect.fulfill() more than once then set the following property.

expectedFulfillmentCount

Please refer the following link https://developer.apple.com/documentation/xctest/xctestexpectation/2806572-expectedfulfillmentcount?language=objc

Argol

Thanks @Argol – Geller

Answer 3

Is it possible to wait for multiple expectations; yes. Here is a signature for an XCTestCase method that shows this.

func wait(for: [XCTestExpectation], timeout: TimeInterval)

There is a version that also makes sure that the expectations are fulfilled in the same order as they appear in the for: array.

See the documentation provided by Apple in XCode->Window->Documentation and API Reference, then search for XCTestCase.

Skiver

Thanks for your advice. I do know of that method, but from what I can tell you use it when you're instantiating multiple expectations in the one test method. I have multiple test methods. I'm sorry for the confusion and have adjusted my original post and included a small project to illustrate my issue as requested by Rob – Franchot

Double check your loginWith completion handler. It seems very strange to pass in the loginCompletion function in the arguments of the completion handler instead of just capturing it from the context. There may be some state in loginCompletion that interferes with subsequent runs. – Skiver

Unfortunately I had to do it this way as I wanted the loginWith method to throw an error (to which the caller can do what it wants with the error). However, because I'm using Alamofire I can't throw an error from the class that's calling the Alamofire request method because of incompatible return types (_) throws -> () is no the same as (DataResponse<Any>) -> Void (hopefully that makes sense). – Franchot