Unable to sustain a Constant Speed while Uploading files in the background with NSURLSession

Question

I am trying to upload some 100 images to S3 in the background with AFURLSessionManager in small batches of 10 like what is being done here- Manage the number of active tasks in a background NSURLSession

I am using a shared NSURLSession and adding tasks according more tasks to this when some tasks are completed. Average size of each file is about 1.6 MB and the number of tasks that are guaranteed to run per a task queue is 5

Here is my method for adding the tasks: (also available as an easier-to-read gist)

    - (void) addTasksToSessionWithTaskObject:(Task*)taskObject withSessionInitialisationNeeded:(BOOL) needed{

        NSString *filePath = [[NSBundle mainBundle] pathForResource:pathForResourceFile ofType:resourceFileType];
        S3PutObjectRequest *putObjectRequest = [[S3PutObjectRequest alloc] initWithKey:targetFileKey
                                                                              inBucket:_bucketname];
        putObjectRequest.cannedACL = [S3CannedACL publicReadWrite];
        putObjectRequest.filename = filePath;
        putObjectRequest.contentType = [resourceFileType isEqualToString:@"MOV"] ? @"movie/mov" : @"image/jpg";
        putObjectRequest.endpoint = @"http://s3.amazonaws.com";
        putObjectRequest.contentLength=[[[NSFileManager defaultManager]
                                         attributesOfItemAtPath:filePath error:nil] fileSize];
        putObjectRequest.delegate = self;
        [putObjectRequest configureURLRequest];
        NSMutableURLRequest *request = [s3Client signS3Request:putObjectRequest];
        NSMutableURLRequest *request2 = [[NSMutableURLRequest alloc]initWithURL:[NSURL URLWithString:[NSString stringWithFormat:@"http://s3.amazonaws.com/UploadTest/%@",taskObject.fileKey]]];

        [request2 setHTTPMethod:request.HTTPMethod];
        [request2 setAllHTTPHeaderFields:[request allHTTPHeaderFields]];

        if(needed) {

                sharedSession = [self backgroundSession];   
        }
   NSURLSessionUploadTask *task = [sharedSession uploadTaskWithRequest:request2           fromFile:forFileUrl];

    task.taskDescription = pathForResourceFile;
    [currentlyActiveTaskIdArray addObject:@([task taskIdentifier])];

    [task resume];
}

And this what I've done with the delegate

- (void)URLSession:(NSURLSession *)sessionI task:(NSURLSessionTask *)task
didCompleteWithError:(NSError *)error{

    dispatch_async(dispatch_get_main_queue(), ^{

        __block UIBackgroundTaskIdentifier bgTaskI = [[UIApplication sharedApplication] beginBackgroundTaskWithExpirationHandler:^{
            [[UIApplication sharedApplication] endBackgroundTask:bgTaskI];
        }];

        if([currentlyActiveTaskIdArray containsObject:@([task taskIdentifier])]){
            [currentlyActiveTaskIdArray removeObject:@([task taskIdentifier])];
        }
        if(currentlyActiveTaskIdArray.count < LOWER_SLAB_FOR_TASKS + 1){   
            [self initiateS3UploadForSetOfTasksIsItBeginningOfUpload:NO];
        }
        [[UIApplication sharedApplication] endBackgroundTask:bgTaskI];
    }); 
}

Here is the Code to add more tasks

 - (void) initiateS3UploadForSetOfTasksIsItBeginningOfUpload:(BOOL)beginning{
        int i=0;
        for(Task *eachTaskObject in tasksArray){
            if(i < numberOfTasksTobeAdded){  
                [self addTasksToSessionWithTaskObject:eachTaskObject WithSessionInitialisationNeeded:NO];
                i++;
            }
        }
    }

I've been running tests with 100 files in Foreground mode and Background mode. In Foreground mode, it uploads the files at a consistant, steady and constant speed, it completes 90 files in the first 3 minutes, and the remaining 10 files in 20 seconds.

When I run the app in Background mode, I would expect it to upload the first 90 files just as fast as it did in the 3 minute Foreground window, and then slow down after that... but that's not the case. In Background mode, it uploads 15 files in the first minute, then it starts slowing down... a lot. It starts uploading in 8 file batches in slower and slower intervals: 1 minute, 3 minutes, 5 minutes, 10 minutes, and now 17 minutes. We're at 65 files 46 minutes in.

Is there a way to either keep it fast for at least the fist 3 minutes, or keep consistent speed in the background?

UPDATE: Following the comments from Clay here Ive switched back to NSURLSession from AFURLSessionManager because as he points out using block based callbacks is an extremely risky business with NSURLSession. Further I've played with HTTPMaximumConnectionsPerHost and set this around 10-this has given better results but nowhere near what I would want to be.

Talbott

Don't have a ton of time to look through your code, but a couple things to check: 1) Did you set HTTPMaximumConnectionsPerHost in your NSURLSessionConfiguration? Use this to set the 'width' of your task queue. 2) Are you adding all of these requests/tasks to the same NSURLSession using uploadTaskWithRequest? – Margret

1)I have set the HTTPMaximumConnectionsPerHost as 1. The task queue I have, runs 5 to 9 tasks at a time. So do would want me to set it as something around these values? 2)I am using a single shared AFURLSessionmanager instance for adding the tasks.. – Talbott

I stuck your code into a gist, just so it would be easier to read, here: gist.github.com/claybridges/8ca22217830e0ce1a0f1. – Mcchesney

Answer 1

From what I can tell, setTaskDidCompleteBlock: is not an Apple API, NSURLSession-associated method. It is an AFURLSessionManager method (docs). If you are using AFNetworking on this, then you need to be announcing that bold, top, front and center. That is not the same, at all, as using NSURLSession. I would guess AFNetworking's background NSURLSession-based implementation comes with its own foibles and idiosyncrasies.

For my part, whatever success I've had with sustained background NSURLSession uploads are using only the stock API.

---

Addressing questions, etc.

• Regarding AFNetworking: we use it for general web api I/O. At the time NSURLSession came out, AFNetworking really didn't robustly support app-in-background ops, so I didn't use it. Perhaps because I went through the background NSURLSession pain & hazing, I look askance at AFNetworking backgrounding under the rubric of "Now you have two problems". But maybe they have cracked the nut by now.
  

  • I strive for one NSURLSession. I started out being cavalier about creation & destruction of sessions, but found this made for some truly gnarly problems. Experiences seem to vary on this.

  • I use the default HTTPMaximumConnectionsPerHost, no problems there. The Apple docs are silent on the default value, but here's what lldb tells me in the random particular device/OS I chose:
    (lldb) p [config HTTPMaximumConnectionsPerHost] (NSInteger) $0 = 4
    If you are having troubles with backgrounding slowing down, I doubt tweaking this is on the right track.

  • FWIW, background NSURLSessions do not support the block interfaces, delegate only.

Mcchesney

Nice spot Clay. I had shifted to AFURLSessionManager because it could account for security policies that would allow invalid SSL certificates-This was done as a remedy for SSL errors that I was getting- As I switch back to the NSURLSession and try to achieve what you've done- I would love to know a couple of things a) were you maintaining a single session and adding more tasks into this session 2) As Dinkman points out in the above comment about the HTTPMaximumConnectionsPerHost , what should I set this value given The task queue I have, runs 5 to 9 tasks. – Talbott

I've seen your post on how you were able to upload 1000 files in True Background Mode. I'm just curious to know. How much time does this app of your's takes to upload 1000 images (of say 1 MB each). Right now the best I'm able to achieve is 300~350 MB within 5 minutes - Do you think I can go further? – Talbott

That rate seems pretty solid. I set up instrumenting to measure my BG upload rate, and to keep a windowed running average, so I could see whether I was settling on a quiescent rate that was sufficient for my purposes. Moreover, I imagine BG uploading vs. large data occurring during (people's) sleep, etc., where an 8 hour time frame is acceptable. – Mcchesney

To add to this- Apparently I have found that the OS prefers uploading 100 files of 1 MB each to a single huge file of the same size- The throttling rate is much higher in latter case. – Talbott

@Xcoder: you should post those results, more thoroughly documented, in a self-created Q/A (you ask question, then you answer question). It would be useful information for the tribe. – Mcchesney

@Talbott I am having a similar issue wherein video uploads will not complete (for example 18/23 video uploads will complete, leaving 5 uncompleted.) Do you have updated results? Here is my code if you're willing to help out: gist.github.com/axpence/ced1a19900a87a0ecb1f – Schultz

I find that the upload rate is fantastic while the debugger is connected to my iPhone. Once the debugger is no longer connected everything slows down and takes long breaks. Did you run your test with the iPhone in your pocket, or with the iPhone connected to the debugger? – Strive

@AndrewPaulSimmons Don't know if you're addressing me, but what I describe seems to work fine in ad hoc builds. Obviously, the lldb stuff was run in the debugger. I haven't experienced what you're describing. – Mcchesney