Wait until swift for loop with asynchronous network requests finishes executing
Asked Answered
N

10

205

I would like a for in loop to send off a bunch of network requests to firebase, then pass the data to a new view controller once the the method finishes executing. Here is my code:

var datesArray = [String: AnyObject]()

for key in locationsArray {       
    let ref = Firebase(url: "http://myfirebase.com/" + "\(key.0)")
    ref.observeSingleEventOfType(.Value, withBlock: { snapshot in

        datesArray["\(key.0)"] = snapshot.value
    })
}
// Segue to new view controller here and pass datesArray once it is complete 

I have a couple concerns. First, how do I wait until the for loop is finished and all the network requests are complete? I can't modify the observeSingleEventOfType function, it is part of the firebase SDK. Also, will I create some sort of race condition by trying to access the datesArray from different iterations of the for loop (hope that makes sense)? I've been reading about GCD and NSOperation but I'm a bit lost as this is the first app I've built.

Note: Locations array is an array containing the keys I need to access in firebase. Also, it's important that the network requests are fired off asynchronously. I just want to wait until ALL the asynchronous requests complete before I pass the datesArray to the next view controller.

Nassi answered 10/3, 2016 at 2:37 Comment(0)
S
439

You can use dispatch groups to fire an asynchronous callback when all your requests finish.

Here's an example using dispatch groups to execute a callback asynchronously when multiple networking requests have all finished.

override func viewDidLoad() {
    super.viewDidLoad()

    let myGroup = DispatchGroup()

    for i in 0 ..< 5 {
        myGroup.enter()

        Alamofire.request("https://httpbin.org/get", parameters: ["foo": "bar"]).responseJSON { response in
            print("Finished request \(i)")
            myGroup.leave()
        }
    }

    myGroup.notify(queue: .main) {
        print("Finished all requests.")
    }
}

Output

Finished request 1
Finished request 0
Finished request 2
Finished request 3
Finished request 4
Finished all requests.
Subrogate answered 10/3, 2016 at 2:53 Comment(14)
This worked great! Thanks! Do you have any idea if I will run into any race conditions when I am trying to update the datesArray?Nassi
I don't think there is a race condition here because all requests add values to datesArray using a different key.Subrogate
@Nassi Regarding race condition: a race condition occurs, if the same memory location will be accessed from different threads, where at least one access is a write - without using synchronisation. All accesses within the same serial dispatch queue are synchronised, though. Synchronisation also occurs with memory operations occurring on dispatch queue A, which submits to another dispatch queue B. All operations in queue A are then synchronised in queue B. So, if you look at the solution, it's not automatically guaranteed that accesses are synchronised. ;)Fault
@josh, be aware that "racetrack programming" is, in a word, stupendously difficult. It is never possible to just instantly say "you do/don't have a problem there." For hobbyist programmers: "simply" always work in a way that means racetrack problems are, simply, impossible. (For example, things like "only do one thing at once" etc.) Even doing that is a huge programming challenge.Omphale
Super cool. But I have a question. Suppose request 3 and request 4 failed (eg server error, authorisation error, anything), then how to call for loop again for only remaining requests (request 3 & request 4)?Tetzel
@JD. You can check out the Request Retrier that Alamofire 4 introduced.Subrogate
How to do with Apple's URLSession?Tetzel
if i have 2 network requests, one nested with the other, inside a for loop, then how to make sure that for each iteration of for loop, both requests have been completed. ?Guilford
Could not find the solution after 1/2 day until sees this answer. thanksBialystok
I tried this as seen here: #57068230 and I still can't get it to run in the correct order!....Turoff
This is amazing, how the heck have I never used DispatchGroup before??Isometric
Thaaaaanks ! just what I needed :DCullis
async /await versionMadagascar
I've more request after first request is completed. How can I handle that?Lindbergh
G
67

Swift 3 or 4

If you don't care about orders, use @paulvs's answer, it works perfectly.

else just in case if anyone wants to get the result in order instead of fire them concurrently, here is the code.

let dispatchGroup = DispatchGroup()
let dispatchQueue = DispatchQueue(label: "any-label-name")
let dispatchSemaphore = DispatchSemaphore(value: 0)

dispatchQueue.async {

    // use array categories as an example.
    for c in self.categories {

        if let id = c.categoryId {

            dispatchGroup.enter()

            self.downloadProductsByCategory(categoryId: id) { success, data in

                if success, let products = data {

                    self.products.append(products)
                }

                dispatchSemaphore.signal()
                dispatchGroup.leave()
            }

            dispatchSemaphore.wait()
        }
    }
}

dispatchGroup.notify(queue: dispatchQueue) {

    DispatchQueue.main.async {

        self.refreshOrderTable { _ in

            self.productCollectionView.reloadData()
        }
    }
}
Goshen answered 20/10, 2017 at 15:17 Comment(2)
My app has to send multiple files to an FTP server, which also includes logging in first. This approach guarantees that the app only logs in once (before uploading the first file), instead of trying to do it multiple time, all at basically the same time (like with the "unordered" approach), which would trigger errors. Thanks!Parcenary
I've got one question though: Does it matter if you do dispatchSemaphore.signal() before or after leaving the dispatchGroup? You'd think that it's best to unblock the semaphore as late as possible but I'm not sure if and how leaving the group interferes with that. I tested both orders and it didn't seem to make a difference.Parcenary
R
49

Xcode 8.3.1 - Swift 3

This is the accepted answer of paulvs, converted to Swift 3:

let myGroup = DispatchGroup()

override func viewDidLoad() {
    super.viewDidLoad()

    for i in 0 ..< 5 {
        myGroup.enter()
        Alamofire.request(.GET, "https://httpbin.org/get", parameters: ["foo": "bar"]).responseJSON { response in
            print("Finished request \(i)")
            myGroup.leave()
        }
    }

    myGroup.notify(queue: DispatchQueue.main, execute: {
        print("Finished all requests.")
    })
}
Retrad answered 18/1, 2017 at 17:58 Comment(5)
Hi, does this work for let´s say 100 requests? or 1000? Because I´m trying to do this with about 100 requests and is crashing on the completion of the request.Abad
I second @lopes710-- This appears to allow all requests to operate in parallel, right?Ferren
if i have 2 network requests, one nested with the other, inside a for loop, then how to make sure that for each iteration of for loop, both requests have been completed. ?Guilford
@Retrad , please is there a way I can get this ordered?Veteran
@IsraelMeshileya - Yes, store the results in a dictionary and then collate the results into your ordered array at the end. But to force them to run sequentially just because you want the results ordered (as advised elsewhere) is a mistake.Nibbs
P
17

Details

  • Xcode 10.2.1 (10E1001), Swift 5

Solution

import Foundation

class SimultaneousOperationsQueue {
    typealias CompleteClosure = ()->()

    private let dispatchQueue: DispatchQueue
    private lazy var tasksCompletionQueue = DispatchQueue.main
    private let semaphore: DispatchSemaphore
    var whenCompleteAll: (()->())?
    private lazy var numberOfPendingActionsSemaphore = DispatchSemaphore(value: 1)
    private lazy var _numberOfPendingActions = 0

    var numberOfPendingTasks: Int {
        get {
            numberOfPendingActionsSemaphore.wait()
            defer { numberOfPendingActionsSemaphore.signal() }
            return _numberOfPendingActions
        }
        set(value) {
            numberOfPendingActionsSemaphore.wait()
            defer { numberOfPendingActionsSemaphore.signal() }
            _numberOfPendingActions = value
        }
    }

    init(numberOfSimultaneousActions: Int, dispatchQueueLabel: String) {
        dispatchQueue = DispatchQueue(label: dispatchQueueLabel)
        semaphore = DispatchSemaphore(value: numberOfSimultaneousActions)
    }

    func run(closure: ((@escaping CompleteClosure) -> Void)?) {
        numberOfPendingTasks += 1
        dispatchQueue.async { [weak self] in
            guard   let self = self,
                    let closure = closure else { return }
            self.semaphore.wait()
            closure {
                defer { self.semaphore.signal() }
                self.numberOfPendingTasks -= 1
                if self.numberOfPendingTasks == 0, let closure = self.whenCompleteAll {
                    self.tasksCompletionQueue.async { closure() }
                }
            }
        }
    }

    func run(closure: (() -> Void)?) {
        numberOfPendingTasks += 1
        dispatchQueue.async { [weak self] in
            guard   let self = self,
                    let closure = closure else { return }
            self.semaphore.wait(); defer { self.semaphore.signal() }
            closure()
            self.numberOfPendingTasks -= 1
            if self.numberOfPendingTasks == 0, let closure = self.whenCompleteAll {
                self.tasksCompletionQueue.async { closure() }
            }
        }
    }
}

Usage

let queue = SimultaneousOperationsQueue(numberOfSimultaneousActions: 1, dispatchQueueLabel: "AnyString")
queue.whenCompleteAll = { print("All Done") }

 // add task with sync/async code
queue.run { completeClosure in
    // your code here...

    // Make signal that this closure finished
    completeClosure()
}

 // add task only with sync code
queue.run {
    // your code here...
}

Full sample

import UIKit

class ViewController: UIViewController {

    private lazy var queue = { SimultaneousOperationsQueue(numberOfSimultaneousActions: 1,
                                                           dispatchQueueLabel: "AnyString") }()
    private weak var button: UIButton!
    private weak var label: UILabel!

    override func viewDidLoad() {
        super.viewDidLoad()
        let button = UIButton(frame: CGRect(x: 50, y: 80, width: 100, height: 100))
        button.setTitleColor(.blue, for: .normal)
        button.titleLabel?.numberOfLines = 0
        view.addSubview(button)
        self.button = button

        let label = UILabel(frame: CGRect(x: 180, y: 50, width: 100, height: 100))
        label.text = ""
        label.numberOfLines = 0
        label.textAlignment = .natural
        view.addSubview(label)
        self.label = label

        queue.whenCompleteAll = { [weak self] in self?.label.text = "All tasks completed" }

        //sample1()
        sample2()
    }

    func sample1() {
        button.setTitle("Run 2 task", for: .normal)
        button.addTarget(self, action: #selector(sample1Action), for: .touchUpInside)
    }

    func sample2() {
        button.setTitle("Run 10 tasks", for: .normal)
        button.addTarget(self, action: #selector(sample2Action), for: .touchUpInside)
    }

    private func add2Tasks() {
        queue.run { completeTask in
            DispatchQueue.global(qos: .background).asyncAfter(deadline: .now() + .seconds(1)) {
                DispatchQueue.main.async { [weak self] in
                    guard let self = self else { return }
                    self.label.text = "pending tasks \(self.queue.numberOfPendingTasks)"
                }
                completeTask()
            }
        }
        queue.run {
            sleep(1)
            DispatchQueue.main.async { [weak self] in
                guard let self = self else { return }
                self.label.text = "pending tasks \(self.queue.numberOfPendingTasks)"
            }
        }
    }

    @objc func sample1Action() {
        label.text = "pending tasks \(queue.numberOfPendingTasks)"
        add2Tasks()
    }

    @objc func sample2Action() {
        label.text = "pending tasks \(queue.numberOfPendingTasks)"
        for _ in 0..<5 { add2Tasks() }
    }
}
Pekingese answered 16/11, 2017 at 17:9 Comment(0)
M
16

Update for iOS 15+ (Swift 5.5)

I've added a more modern solution for Swift 5.5 and iOS 15+ because this toolchain includes major URLSession API improvements, that are not specific to Firebase or Alamofire. The code uses async / await i.e. Structured Concurrency. It's what Apple recommends for concurrent requests on the latest iOS versions (iOS 13.0+).

We now achieve the same result as DispatchGroups with fewer lines of code and more customisation. This answer will help users who used to queue URLSession requests and wait for these to complete.

Task group example code

The right tool is a TaskGroup if we have a dynamic number of requests (variable-sized array).

func fetchThumbnails(for ids: [String]) async throws -> [String: UIImage] {
  var thumbnails: [String: UIImage] = [:]
  try await withThrowingTaskGroup(of: (String, UIImage).self) { group in
    for id in ids {
      group.addTask {
        return (id, try await fetchOneThumbnail(withID: id))
      }
    }
    for try await (id, thumbnail) in group {
      thumbnails[id] = thumbnail
    }
  }
  return thumbnails
}


func fetchOneThumbnail(withID id: String) async throws -> UIImage {
    // Just for demo purpose. In PROD, we may use dynamic URLs for each ID.
    guard let url = URL(string: "http://placekitten.com/200/300") else {
        throw ThumbnailError.invalidURL
    }
    // I have used `data(from: URL, delegate: URLSessionTaskDelegate? = nil)`
    // but we can also use `data(for: URLRequest, delegate: URLSessionTaskDelegate? = nil)`)`.
    // If we want to observe the delegate changes like when the
    // request fails, completes, or redirects, use the delegate param.
    // e.g. try await URLSession.shared.data(from: url, delegate: delegate)
    let result: (data: Data, response: URLResponse) = try await URLSession.shared.data(from: url)
    guard let image = UIImage(data: result.data) else {
        throw ThumbnailError.missingImageData
    }
    return image
}

enum ThumbnailError: Error {
    case invalidURL
    case missingImageData
}

Task {
    let images = try await fetchThumbnails(for: ["1", "2", "3"])
    // Show thumbnails in UI.
}

This also uses the for await loop (AsyncSequence) to wait for tasks to complete. for try await is an example of a throwing AsyncSequence. The throwing syntax is because the new asynchronous URLSession.data(for:) family of methods are throwing functions.

async let example code

async let syntax works for a fixed number of requests.

let reqOne = urlRequest(for: keyOne) // Function that returns a unique URLRequest object for this key. i.e. different URLs or format.
async let (dataOne, _) = URLSession.shared.data(for: reqOne)

let reqTwo = urlRequest(for: keyTwo)
async let (dataTwo, _) = URLSession.shared.data(for: reqTwo)

guard let parsedData = parseInformation(from: try? await dataOne) else {
    // Call function to parse image, text or content from data.
      continue 
}
// Act on parsed data if needed.

guard let parsedDataTwo = parseInformation(from: try? await dataTwo) else {
    // Call function to parse image, text or content from data.
      continue 
}
// Act on the second requests parsed data if needed.

// Here, we know that the queued requests have all completed.

The syntax where I don't await for the request to finish immediately is called async let.

This code example could be adapted with variable-sized arrays but isn't recommended by Apple. This is because async let doesn't always allow the requests to be processed as soon as they arrive.

The benefits of this approach are cleaner code that's easier to write, safer, and avoiding deadlocks/threading issues.

Note

The exact syntax of TaskGroup and async let may change in the future. Currently, Structured Concurrency has improved a lot during its early releases and is now stable for production.

Apple has clarified that the underlying mechanics of grouped and asynchronous tasks are mostly finalised (approved in Swift Evolution). An example of some syntax changes already includes the replacement of async { with Task {.

Madagascar answered 10/6, 2021 at 15:58 Comment(0)
A
5

You will need to use semaphores for this purpose.

 //Create the semaphore with count equal to the number of requests that will be made.
let semaphore = dispatch_semaphore_create(locationsArray.count)

        for key in locationsArray {       
            let ref = Firebase(url: "http://myfirebase.com/" + "\(key.0)")
            ref.observeSingleEventOfType(.Value, withBlock: { snapshot in

                datesArray["\(key.0)"] = snapshot.value

               //For each request completed, signal the semaphore
               dispatch_semaphore_signal(semaphore)


            })
        }

       //Wait on the semaphore until all requests are completed
      let timeoutLengthInNanoSeconds: Int64 = 10000000000  //Adjust the timeout to suit your case
      let timeout = dispatch_time(DISPATCH_TIME_NOW, timeoutLengthInNanoSeconds)

      dispatch_semaphore_wait(semaphore, timeout)

     //When you reach here all request would have been completed or timeout would have occurred.
Appraisal answered 10/3, 2016 at 4:5 Comment(0)
U
5

We can do this with recursion. Get idea from below code :

var count = 0

func uploadImages(){

    if count < viewModel.uploadImageModelArray.count {
        let item = viewModel.uploadImageModelArray[count]
        self.viewModel.uploadImageExpense(filePath: item.imagePath, docType: "image/png", fileName: item.fileName ?? "", title: item.imageName ?? "", notes: item.notes ?? "", location: item.location ?? "") { (status) in

            if status ?? false {
                // successfully uploaded
            }else{
                // failed
            }
            self.count += 1
            self.uploadImages()
        }
    }
}
Underact answered 1/8, 2019 at 14:33 Comment(0)
M
3

Swift 3: You could also use semaphores on this way. It results very helpful, besides you can keep exact track on when and what processes are completed. This has been extracted from my code:

    //You have to create your own queue or if you need the Default queue
    let persons = persistentContainer.viewContext.persons
    print("How many persons on database: \(persons.count())")
    let numberOfPersons = persons.count()

    for eachPerson in persons{
        queuePersonDetail.async {
            self.getPersonDetailAndSave(personId: eachPerson.personId){person2, error in
                print("Person detail: \(person2?.fullName)")
                //When we get the completionHandler we send the signal
                semaphorePersonDetailAndSave.signal()
            }
        }
    }

    //Here we will wait
    for i in 0..<numberOfPersons{
        semaphorePersonDetailAndSave.wait()
        NSLog("\(i + 1)/\(persons.count()) completed")
    }
    //And here the flow continues...
Maemaeander answered 1/12, 2016 at 22:34 Comment(0)
N
0

In the original question, it was contemplated how to know when all of the queries were done, returning the results in a dictionary. paulvs has answered (+1) that question: If using old completion handler closure pattern, use a dispatch group to know when they’re done. And, nowadays, if using Swift concurrency, use the pattern proposed by Pranav Kasetti.

But, if you need the results in an ordered array, though, one should not make the requests, themselves, run sequentially. You pay a serious performance penalty (often more than 3× slower) if you do that. And if you achieve this through the use semaphores, you introduce all sorts of other inefficiencies and deadlock risks.

Instead, if you really need the results in an ordered array, you should employ paulvs’s answer to populate a dictionary from the concurrent requests, but then build an array of the results at the end. E.g.,

let array = ids.compactMap { resultsDictionary[$0] }

For example, Swift concurrency handles this all very gracefully:


func fetch(for ids: [Id]) async throws -> [Foo] {
    try await withThrowingTaskGroup(of: (Id, Foo).self) { [self] group in
        for id in ids {
            group.addTask { (id, try await fetch(for: id)) }
        }
        
        let dictionary = try await group.reduce(into: [:]) { $0[$1.0] = $1.1 }
        return ids.compactMap { dictionary[$0] }
    }
}

But even if you are using the older completion handler closure pattern the idea is the same: Store your results in a dictionary, enjoy concurrency, and build the sorted array at the end if you really need it.

Nibbs answered 24/2, 2022 at 17:19 Comment(0)
H
-1

Dispatch group is good but the order of sent requests is random.

Finished request 1
Finished request 0
Finished request 2

In my project case, each requests needed to be launch is the right order. If this could help someone :

public class RequestItem: NSObject {
    public var urlToCall: String = ""
    public var method: HTTPMethod = .get
    public var params: [String: String] = [:]
    public var headers: [String: String] = [:]
}


public func trySendRequestsNotSent (trySendRequestsNotSentCompletionHandler: @escaping ([Error]) -> () = { _ in }) {

    // If there is requests
    if !requestItemsToSend.isEmpty {
        let requestItemsToSendCopy = requestItemsToSend

        NSLog("Send list started")
        launchRequestsInOrder(requestItemsToSendCopy, 0, [], launchRequestsInOrderCompletionBlock: { index, errors in
            trySendRequestsNotSentCompletionHandler(errors)
        })
    }
    else {
        trySendRequestsNotSentCompletionHandler([])
    }
}

private func launchRequestsInOrder (_ requestItemsToSend: [RequestItem], _ index: Int, _ errors: [Error], launchRequestsInOrderCompletionBlock: @escaping (_ index: Int, _ errors: [Error] ) -> Void) {

    executeRequest(requestItemsToSend, index, errors, executeRequestCompletionBlock: { currentIndex, errors in
        if currentIndex < requestItemsToSend.count {
            // We didn't reach last request, launch next request
            self.launchRequestsInOrder(requestItemsToSend, currentIndex, errors, launchRequestsInOrderCompletionBlock: { index, errors in

                launchRequestsInOrderCompletionBlock(currentIndex, errors)
            })
        }
        else {
            // We parse and send all requests
            NSLog("Send list finished")
            launchRequestsInOrderCompletionBlock(currentIndex, errors)
        }
    })
}

private func executeRequest (_ requestItemsToSend: [RequestItem], _ index: Int, _ errors: [Error], executeRequestCompletionBlock: @escaping (_ index: Int, _ errors: [Error]) -> Void) {
    NSLog("Send request %d", index)
    Alamofire.request(requestItemsToSend[index].urlToCall, method: requestItemsToSend[index].method, parameters: requestItemsToSend[index].params, headers: requestItemsToSend[index].headers).responseJSON { response in

        var errors: [Error] = errors
        switch response.result {
        case .success:
            // Request sended successfully, we can remove it from not sended request array
            self.requestItemsToSend.remove(at: index)
            break
        case .failure:
            // Still not send we append arror
            errors.append(response.result.error!)
            break
        }
        NSLog("Receive request %d", index)
        executeRequestCompletionBlock(index+1, errors)
    }
}

Call :

trySendRequestsNotSent()

Result :

Send list started
Send request 0
Receive request 0
Send request 1
Receive request 1
Send request 2
Receive request 2
...
Send list finished

See for more infos : Gist

Hereon answered 1/9, 2017 at 15:43 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.