Application sticks on OSSpinLockLockSlow

Question

Update 2: I found a workaround which is to synchronize MOC deallocating and saving. Please see the updated project. https://github.com/shuningzhou/MOCDeadLock.git

Note: I made it fail more aggressively. Don't run it on a real device!

Update: A sample project to demonstrate this issue. https://github.com/shuningzhou/MOCDeadLock.git

XCode 6.2: Unable to reproduce.

XCode 6.3: Reproducible.

XCode 6.4 beta: Reproducible.

========================== The Issue ===============================

Our app randomly stuck on OSSpinLockLockSlow after upgrading to XCode 6.3. In our project, we used NSOperation and NSOperationQueue to fetch data from our server and used Core Data for data persistence.

This issue never happened before! You can see from the stack trace that no calls are made by our code. I am not sure where to start debugging this. Could someone provide some guidance?

Thank you in advance!

Please see the stack trace

Edit:

We are using AFNetworking and our NSOperations are subclasses of AFHTTPRequestOperation. We added some custom properties and overrode the method -(void)start:

- (void)start;
{
    //unrelated code...

    NSString *completionQueueID = [NSString uuid];
    const char *cString = [completionQueueID cStringUsingEncoding:NSASCIIStringEncoding];
    self.completionQueue = dispatch_queue_create(cString, DISPATCH_QUEUE_SERIAL);

    //unrelated code....

    [super start];
}

For Core Data, We are following the thread-confinement pattern. We have separate managed object context for each thread, and the contexts share a static persistent store coordinator.

Edit 2:

More info: I found that this issue happens when the system exits multiple threads at the same time. We store the Managed Object Context in the thread dictionary, and they get released when the threads exit.

[[[NSThread currentThread] threadDictionary] setObject:dataManager forKey:@"IHDataManager"];

CPU usage is around 20%.

Lashelllasher

Please post the relevant code relating to NSOperation. – Showiness

Do you have a signal handler in your system, particularly a crash handler? Look through your threads for any reference to _sigtramp. (Note that you can copy stack traces with Cmd-C; you don't have to take screenshots.) I assume that your CPU usage becomes very high (maybe turns the fans on)? – Balikpapan

I was about to post the same question, have been seeing these hangs since XCode6.3 ( I am not sure whether it is XCode or something I wrote, but I haven't changed networking or core data code recently ). Usually Core Data and AFNetworking threads are involved. In the hang I am looking at right now, the main queue, three com.apple.networking.connection queues and NSManagedObjectContext queue are locked up. – Groping

@Groping Have you found a solution to this issue? Please keep me posted. – Lashelllasher

i am putting an answer up now, although it may not be useful to you as it is a fairly specific workaround for my situation. – Groping

I've checked the sample project you posted. Trying on an iPad Mini (built with xCode v6.3.2), I was unable to reproduce it. Also no luck on the simulators either. – Nicaragua

@Nicaragua I can rarely reproduce it on a device. It is easier to reproduce on simulators. So far I am only able to reproduce on iPad 2/iOS 8.3, iPad 4s/iOS 8.3, and iPad retina/iOS 8.3 simulators. – Lashelllasher

@PeterZhou That's really interesting. Switched it to an iPad simulator, and it happened fairly quickly. Really curious as to what's causing it now... – Nicaragua

@Nicaragua Are you using CoreData in a multi-threaded environment? – Lashelllasher

@PeterZhou MagicalRecord specifically, but yes, CoreData in a multi-threaded environment (getting data from server -> device in BG, displaying data in FG). – Nicaragua

@PeterZhou However instead of extending NSThread, I've been using GCD – Nicaragua

@Nicaragua We used GCD in our project as well. However, I find it is easier to reproduce the issue with more direct control on the NSThread objects. – Lashelllasher

Answer 1

I have been experiencing precisely this issue. As per your stack trace, I have a bunch of threads stalled with _OSSpinLockLockSlow.

It appears to be a livelock situation with the spinlocks chained up together. Including some networking threads and core data. But as Rob pointed out, symptoms of livelock should include high CPU usages (spinlocks are all endlessly spinning). In my case (and in yours) this is not the case, CPU usage is low - simulator 'percent used' 20%, simulator overall in activity monitor 0.6% - so maybe it's a deadlock ;-)

Like you, I am using a thread-confinement pattern, separate managed object context per thread, single persistent store.

Following your observation that the hang always seems to follow deallocing of a bunch of threads, I checked that behaviour and can confirm that is the case.

This got me wondering why I had so many threads active. It turned out I was using gcd with a concurrent background queue:

 dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_BACKGROUND,0),^{
        modelClass = [WNManagedObject classForID:mongoID];
         dispatch_async(dispatch_get_main_queue(),^{
         ...
         });
        });

This snippet is part of some networking/JSON parsing code. 'classForID' was causing slight UI jitters on the main thread, so I backgrounded it.

In effect the concurrent background queue was spitting out a whole bunch of short-lived threads. This was completely unnecessary. Refactoring as a single serial queue fixed the thread excesses, which got rid of the spinlock issue. Finally I realised I didn't need to get the class at all, so this code has since been exorcised.

Problem fixed, but no explanation as to why this should suddenly become an issue with 8.3

I suspect that the same issue is touched on in this question (although Cocoalumberjack gets the blame there):
syscall_thread_switch iOS 8.3 race - CocoaLumberjack bug? how to debug this?

..and in this Cocoalumberjack bug report
https://github.com/CocoaLumberjack/CocoaLumberjack/issues/494

I am also using CocoaLumberjack but it does not feature in any of the problem threads, so I think that is a red herring. The underlying cause seems to be excess thread creation.

I have seen the issue in the simulator and on devices when tethered to XCode, but I have not experienced it when running independently of XCode. It is new to me in iOS 8.3 / XCode 6.3.1

Not really an answer, more of a diary of my own workaround for this weird issue, but maybe you'll find it useful.

Groping

I am using CocoaLumberJack as well and have read the two related questions before posting mine. I think CocoalumberJack is unrelated. – Lashelllasher

I often see this problem when using AsyncImageView to display many photos. – Papism

@Groping please look at my answer if you still interested in this – Coburg

Answer 2

If question is still actual - this is a bug in iOS: OpenRadar crash report
Also you may find this blog post useful: blog post

I think you should replace OSSpinLocks with something else to fix this in your app.

We encountered this bug in our Unity3d game. We didnt fixed this yet in our app because we do not have access to most of the native iOS code (we write our game on C# and we use a lot of 3-rd party native plugins). So I cannot recommend you something concrete about replacing OSSpinLock. Sorry for my English.

Update
Many Apple frameworks and libraries uses OSSpinLock internally, so you dont need to use it explicity to run into this issue.

Coburg