How to solve timeout issues caused by bad HTTP persistent connection?

Question

I've been struggling with an HTTP timeout issue recently. After more than one month of investigation I'm quite sure that it is caused by bad HTTP persistent connections. Details are as follows:

1. It is an iOS app.
2. Most users are running iOS 8.
3. I'm using NSURLConnection.
4. iOS 8 has one known keep alive bug but mine is a different issue. More specifically, that bug causes NSURLErrorNetworkConnectionLost but my error is NSURLErrorTimedOut. However, I'm not sure whether my issue is caused by another bug of iOS 8.
5. The behavior of my issue: After some time of using — after some HTTP requests are successfully sent and corresponding responses received — one request would cause NSURLErrorTimedOut, and all following(not too far away from the last one in time to reuse the persistent connection) requests would causes NSURLErrorTimedOut.
6. Some working workaround:
   1. Kill and restart the app.
   2. Turn WiFi connection off on iPhone to force using 3G/4G.
   3. Turn air mode on and then turn it off.
7. My analysis: From the behavior the issue seems to be caused by a gone-bad persistent connection. All the subsequent requests keep using this persistent connection so all fail with NSURLErrorTimedOut. From the workaround we can see all of them work because they cause the bad persistent connection to be dropped and a new persistent connection to be created.

My questions:

1. Have anyone else encountered this issue?
2. Is it a known bug of iOS 8?
3. Is it caused by some unconventional configuration of the servers? I don't control the servers but I know they use nginx 1.6.1 and their engineers are working with me in investigating this issue. What information should I ask them for?
4. Is there any way to force NSURLConnection to not reuse the current persistent connection but to create a new one so I can work around this issue after I detect it in my code?

---

Update:

I successfully mitigated this issue on iOS 8 by using CFNetwork and controlling Connection header directly. However it seems the issue becomes worse on iOS 9.

Since my hope that Apple would fix it on iOS 9 is broken I finally fired a radar: http://www.openradar.me/22770738.

If you also encounter this issue please duplicate my radar, or even better, fire your own radar if you have a more reliably reproducible sample.

Topographer

Can you share code on setting up the request and the callback delivering you a response and data? – Homopolar

@MarkusSchumann sure, the most mundane code: github.com/an0/MokeTest/blob/master/MokeTest/ViewController.m. – Topographer

@Topographer - you could ask server engineers to turn off keep-alive for iOS clients and try to reproduce the issue.Just to confirm that we're going in right direction. – Waldrop

KeepAliveTimeout in server ? – Waldrop

when above error is encountered try sending your requests with a Connection: close header.This will make server drop the current connection and try again it might work – Waldrop

Try setting the KeepAliveTimeout to higher value and make sure KeepAliveTimeout and NSURLConnectionTimeout is equal..Because I remember reading somewhere that nginx default timeout is 75s and NSURLConnection is 30s and setting the NSURLConnection to 75s solved the persistent connection problem – Waldrop

@Duraiamuthan.H my first though was also setting connection header to close, but unfortunately it is impossible with NSMutableURLRequest . – Topographer

NSURLErrorTimedOut, meaning the connection failed because it took too long. This can happen for a number of reasons. For example, it can happen if the device has a weak Internet connection, or if some software (e.g., a proxy or firewall) on your network connection is interfering. If this is recurring, I recommend checking on your network connection for issues like that – Waldrop

@Duraiamuthan.H It is not a personal issue. Quite some users encounter it. I think it is neither a bad internet issue because simply switching off and on Air Mode resolves it immediately every time. – Topographer

@Topographer - I have asked a new question to be helpful for this question https://mcmap.net/q/1012987/-how-to-disable-keepalive-in-nsurlconnection/730807 – Waldrop

Answer 1

After 2 weeks of research, I can give answers to question 3 and 4:

3. nginx's persistent connection timeout is set to 5s on server, which should not be the cause. Server engineers found those timed-out requests are actually normally received and responded. So it is more likely a client side issue. Since I have a minimal reproducible code to rule out my code as the cause, the cause should be in iOS.
4. The only way I found is to use CFNetwork. Higher level API such as NSURLConnection or NSURLSession's Connection header will be overwritten by system.

Topographer

iOS doesn't respect KeepAliveTimeout value AFAIK and In client it seems no way to solve it as of now.So why not try something like ignoring keepalive for iOS 8 browsers or try increasing the value of timeout to higher value just for testing. – Waldrop

I am running into this exact same issue on iOS 8. Disabling keep alive on the server did not seem to help. The only thing solved the issue was to enable keep alive on the server AND set the value to 30 seconds or higher. This looks like a bug with iOS 8 and related to the known keep alive bug. – Zoogloea

@mkwon if you have a persistently reproducible sample project you should definitely report to Apple. I haven't because my sample can only reproduce the issue occasionally and I don't control the server. – Topographer

Any new findings on this @an0? – Brassard

Answer 2

Same issue here, iOS just try to reuse the connection after server drops it.

Why CFNetwork is NOT Enough

About two years ago, I switched to CFNetwork to workaround this issue, but recently I found it's not possible to implement SSL pinning with CFNetwork API. So now I'm considering go back to NSURLSession.

The Workaround

After some dig around, I found system will NOT reuse connections across NSURLSessions, so creating new sessions within period of time, should solve the issue.

But I also found (at least on macOS): each connection made by NSURLSession can persistence last by 180 seconds, and that connection didn't close by release or reset the session, so you may need to implement some caching mechanism to avoid creating a lots of connections at the same time.

Here is the simple mechanism I'm currently use:

@interface HTTPSession : NSObject

@property (nonatomic, strong) NSURLSession * urlSession;
@property (nonatomic, assign) CFTimeInterval flushTime;

@end

+ (NSURLSession *)reuseOrCreateSession
{
    static NSMutableArray<HTTPSession *> * sessions = nil;
    static dispatch_once_t onceToken;
    dispatch_once(&onceToken, ^{
        sessions = [NSMutableArray<HTTPSession *> array];
    });

    const CFTimeInterval serverTimeoutSeconds = 10;
    const CFTimeInterval systemTimeoutSeconds = 40;
    CFTimeInterval now = CFAbsoluteTimeGetCurrent();

    HTTPSession * resultSession = nil;
    for (HTTPSession * session in sessions) {
        CFTimeInterval lifeTime = now - session.flushTime;
        if (lifeTime < serverTimeoutSeconds) {
            resultSession = session;
            break;
        }
        if (lifeTime > systemTimeoutSeconds) {
            resultSession = session;
            resultSession.flushTime = now;
            break;
        }
    }

    if (!resultSession) {
        resultSession = [HTTPSession new];

        NSURLSession * session = [NSURLSession sessionWithConfiguration:[NSURLSessionConfiguration defaultSessionConfiguration]];

        // setup session

        resultSession.urlSession = session;
        resultSession.flushTime = now;

        [sessions addObject:resultSession];
    }

    return resultSession.urlSession;
}

Home

This bug is still there in iOS 10? Even iOS 11? – Topographer

@Topographer Not sure about iOS 11, but iOS 10 and macOS 10.13/10.12 still have this issue – Home

Answer 3

What if you add timestamp to all your request URL? I think this will make each request unique and maybe iOS will establish new connection every time you send request ( I'm not sure. Need trying )

Battalion