Waiting for multiple asynchronous download tasks

Question

I want to download some files, for example 100 files, at the same time. So I decided to add my download threads to a dispatch queue, and GCD will adjust how many threads will run at the same time.

The problem here is: the block in dispatch_async will be completed immediately, because task will run on another thread. So, if urls's length is 100, it will create 100 threads immediately.

var queueDownloadTask = dispatch_queue_create("downloadQueue", nil)

for url in urls {
    dispatch_async(queueDownloadTask) {

        let config = NSURLSessionConfiguration.defaultSessionConfiguration()
        let fileTransferSession = NSURLSession(configuration: config)

        let task = fileTransferSession.downloadTaskWithURL(url, completionHandler: { (responseUrl, response, error) -> Void in
            println("completed")
        })
        task.resume()
    }
}

How can I configure the block in dispatch_async to wait for the download task to complete? I don't want use dispatch_semaphore, because it only allow run one download task at the same time.

Chemosh

Possible duplicate of Waiting for multiple blocks to finish – Eudoca

Answer 1

To expand on Abhinav's answer, you should:

1. Use dispatch_group_create() to create a group.
2. Call dispatch_group_enter(group) before starting each download task.
3. Call dispatch_group_leave(group) inside the task's completion handler.
4. Then call dispatch_group_notify(group, queue, ^{ ... }) to enqueue a block that will be executed when all the tasks are completed.

You can see an example in this post.

(By the way, it's not true that doing 100 dispatch_asyncs in a row will create 100 threads immediately. The system still retains control over how many threads to use to satisfy the queue. However, your code does not wait for any of the tasks to complete before it returns, nor does it attempt to synchronize between multiple tasks completing.)

Bastien

I think this is a valid answer. Additionally, using a separate queue to schedule tasks (like the queueDownloadTask) is unnecessary as task.resume() returns immediately. Just using the main queue to schedule them is good enough. – Furr

True. I would recommend that as well. – Bastien

Answer 2

In Swift 4,

func executeMultiTask() {
     //1. Create group
     let taskGroup = DispatchGroup()

     //2. Enter group
     taskGroup.enter()
     myTask1.execute(completeHandler: {
         // ...
         //3. Leave group
         taskGroup.leave() //< balance with taskGroup.enter()
     })

     /* Add more tasks ...
     //2. Enter group
     taskGroup.enter()
     myTask2.execute(completeHandler: {
         //3. Leave group
         defer {
            // Use `defer` to make sure, `leave()` calls are balanced with `enter()`.
            taskGroup.leave() 
         }
         // ... more
     })
     */

     //4. Notify when all task completed at main thread queue.
     taskGroup.notify(queue: .main) { 
         // All tasks are done.
         // ...   
     }

}

Sumer

Just curious, how can defer help balance leave() and enter() calls? Wouldn't it also be balanced without defer? – Hodges

@Hlung, defer is Swift syntax. It will be executed finally. I google a document. It's like try/finally. – Sumer

defer is nice if you have multiple exit conditions in a scope. But if you don't, I think defer is not needed. We can just put taskGroup.leave() at the end of the scope. – Hodges

I think it is a habit problem. When I write taskGroup.enter(), I put a taskGroup.leave() at next 2 ~ 3 lines. If the sub-routine is 20 ~ 30 lines long, it's hard to trace that the enter(), and leave() are balanced. – Sumer

This doesn't work 100% of times in iOS 11 for some reason. Just a heads up. – Florida

Answer 3

A working objective-c example for downloading multi files

- (void) downloadFiles: (NSMutableArray *) fileArray : (NSString *)destParentDir{
    dispatch_group_t serviceGroup = dispatch_group_create();

    for (id fileInfo in fileArray) {
        dispatch_group_enter(serviceGroup);

        NSFileManager *fileManager = [NSFileManager defaultManager];
        NSString *fileName = [fileInfo valueForKey:@"name"];

        //Create SubDirs if needed, note that you need to exclude file name for the dirsString :)
        //[fileManager createDirectoryAtPath:dirsString withIntermediateDirectories:true attributes:nil error:NULL];

        //Download file
        NSURL  *url = [NSURL URLWithString:@"YOUR_FILE_URL"];
        NSData *urlData = [NSData dataWithContentsOfURL:url];
        if(urlData)
        {
            NSString  *localPath = [NSString stringWithFormat:@"%@/%@", destParentDir, fileName];
            [urlData writeToFile:localPath atomically:YES];
        }
        dispatch_group_leave(serviceGroup);
    }

    dispatch_group_notify(serviceGroup, dispatch_get_main_queue(),^{
        NSLog(@"Complete files download");
    });
}

Won

Answer 4

You should use dispatch_group_t. Please refer to Apple documentation to solutionize your case.

Lytton

I don't know how to apply dispatch_group to this situation :( – Chemosh

"solutionize"! Great! – Goldin

Answer 5

Create a DispatchGroup. Wrap each download in enter and leave. Provide a callback to notify, which will be called when all pairings have completed.

var dispatchGroup = DispatchGroup()

for url in urls {
    // ... Set up the download
    dispatchGroup.enter()
    let task = fileTransferSession.downloadTaskWithURL(url) { (responseUrl, response, error) -> Void in
        println("Individual download complete.")
        dispatchGroup.leave()
     })
    task.resume()

    dispatchGroup.notify(queue: .main) {
        println("All downloads complete.")
    }
}

Devonne