Crash in libdispatch _dispatch_semaphore_wait_slow

Question

I sometimes get a crash deep inside libdispatch with the following backtrace stemming from the SecItemCopyMatching function.

* thread #1: tid = 0x169ee8, 0x0374c830 libdispatch.dylib`_dispatch_semaphore_wait_slow + 278, queue = 'com.apple.main-thread, stop reason = EXC_BAD_INSTRUCTION (code=EXC_I386_INVOP, subcode=0x0)
    frame #0: 0x0374c830 libdispatch.dylib`_dispatch_semaphore_wait_slow + 278
    frame #1: 0x0374c711 libdispatch.dylib`dispatch_semaphore_wait + 37
    frame #2: 0x03921b54 libxpc.dylib`xpc_connection_send_message_with_reply_sync + 231
    frame #3: 0x01448e99 Security`securityd_message_with_reply_sync + 78
    frame #4: 0x01449098 Security`securityd_send_sync_and_do + 81
    frame #5: 0x01459926 Security`__SecItemCopyMatching_block_invoke + 218
    frame #6: 0x014589f9 Security`SecOSStatusWith + 37
    frame #7: 0x014597da Security`SecItemCopyMatching + 208
    frame #8: 0x0022b482 MyApp`+[BITKeychainUtils getPasswordForUsername:andServiceName:error:](self=0x00399e3c, _cmd=0x002d21e0, username=0x003a336c, serviceName=0x0c17c420, error=0xbfffeb24) + 738 at BITKeychainUtils.m:63
    frame #9: 0x00221eff MyApp`-[BITHockeyBaseManager stringValueFromKeychainForKey:](self=0x0c1b6400, _cmd=0x002d2322, key=0x003a336c) + 175 at BITHockeyBaseManager.m:297
    frame #10: 0x001ef2c2 MyApp`-[BITAuthenticator publicInstallationIdentifier](self=0x0c1b6400, _cmd=0x002d2678) + 194 at BITAuthenticator.m:749
    frame #11: 0x00228be1 MyApp`-[BITHockeyManager startManager](self=0x0c6a1e90, _cmd=0x002d2358) + 865 at BITHockeyManager.m:196
    frame #12: 0x00090ed4 MyApp`-[ApplicationDelegate init](self=0x0c1850e0, _cmd=0x0375f96b) + 212 at ApplicationDelegate.m:86
    frame #13: 0x01583cde UIKit`UIApplicationMain + 1132
    frame #14: 0x00002e32 MyApp`main(argc=1, argv=0xbfffee74) + 178 at main.m:15

This crash seems totally random, it never happens to my coworkers. It mainly happens in the 32-bit iOS simulator but it also occurred once on a device.

I have looked at libdispatch source code and I can see that a crash may happen like this: _dispatch_semaphore_wait_slow() → DISPATCH_SEMAPHORE_VERIFY_KR → DISPATCH_CRASH → _dispatch_hardware_crash() → __builtin_trap() but I don’t really understand why that’s happening.

Does anyone know what’s going on?

Edit: the same crash also happened when running unit tests with the following backtrace:

* thread #1: tid = 0x544131, 0x0375a830 libdispatch.dylib`_dispatch_semaphore_wait_slow + 278, queue = 'com.apple.main-thread, stop reason = EXC_BAD_INSTRUCTION (code=EXC_I386_INVOP, subcode=0x0)
    frame #0: 0x0375a830 libdispatch.dylib`_dispatch_semaphore_wait_slow + 278
    frame #1: 0x0375a711 libdispatch.dylib`dispatch_semaphore_wait + 37
    frame #2: 0x0392fb54 libxpc.dylib`xpc_connection_send_message_with_reply_sync + 231
    frame #3: 0x0153c20a SystemConfiguration`__SCNetworkReachabilityServer_targetStatus + 234
    frame #4: 0x015143ce SystemConfiguration`__SCNetworkReachabilityGetFlags + 415
    frame #5: 0x015122a0 SystemConfiguration`reachPerform + 305
    frame #6: 0x0344583f CoreFoundation`__CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 15
    frame #7: 0x03445295 CoreFoundation`__CFRunLoopDoSources0 + 437
    frame #8: 0x0346229e CoreFoundation`__CFRunLoopRun + 910
    frame #9: 0x03461ac3 CoreFoundation`CFRunLoopRunSpecific + 467
    frame #10: 0x034618db CoreFoundation`CFRunLoopRunInMode + 123
    frame #11: 0x050b79e2 GraphicsServices`GSEventRunModal + 192
    frame #12: 0x050b7809 GraphicsServices`GSEventRun + 104
    frame #13: 0x01591d3b UIKit`UIApplicationMain + 1225
    frame #14: 0x000030ea Test Host`main(argc=10, argv=0xbfffeae0) + 138 at main.m:15

Aargau

Hey Cédric, can you supply the full crash log? – Durant

It only happens when I debug in Xcode, so I’m just stuck on a ud2 instruction and no crash log is generated. Or is there a way to let the program continue and generate a full crash log? – Aargau

fwiw (das already having addressed the issue), 'register read' and 'image list' in LLDB will provide most of what matters in the report, in addition to the above. – Durant

It happened again, so here are the registers and image list dump. – Aargau

that value of eax is definitely unexpected, c.f. comment below. – Clarke

It happened again, with a different backtrace and on the device this time. – Aargau

Answer 1

When hitting this type of assertion on x86_64, the rax register contains the error code returned from the kernel syscall.

You should be able to inspect that value from the debugger console via reg read. Detaching the debugger from the process after hitting the exception should also generate a crashreport (which you may want to file).

Unfortunately I don't think the rax trick works on i386 so you may need to retrieve the syscall return value via a breakpoint before hitting the assertion.

In this case the syscall is semaphore_wait and the error is most likely going to be 0xf, i.e. KERN_INVALID_NAME (c.f. /usr/include/mach/kern_return.h)

Hitting that error is an indication of one of the following (in order of likelihood):

• over-release/user-after-free of a dispatch semaphore object
• heap memory corruption
• mach port mismanagement in the process

Clarke

eax is 0xffffffdc in your dump, that is actually the syscall number of the semaphore_wait trap (c.f. disassembly of _semaphore_wait_trap symbol in libsystem_kernel.dylib), i.e. what eax is set to before the sysenter instruction. Having that input register returned unchanged from the sysenter is a sign the syscall wasn't performed at all, e.g. unexpectedly interrupted by an exception. You are definitely in bug-filing territory here, please file a radar with instructions on how to reproduce! – Clarke