OS X 3.2.1 CI "pending integration"

Asked 30/9, 2014 at 19:15 Answered 18/4, 2016 at 18:13

Solved ios continuous-integration osx-server xcode-server

Over the past few days, I've been working on getting CI working with an external mac mini running OS X Server. However, I have been having many problems with OS X Server 3.2.1 and XCode 6.1b3.

It looks like Apple fixed an issue in Xcode 6.1b3 which didn't put the correct provisioning profiles into Portal.keychain. However, my integrations aren't even running now.

After running a clean OS X build, XCode server won't integrate. I succesfully connected to the server and created a bot. If I visit "SERVER.local" on my development machine, I see the bot that I created.

enter image description here

Everything is set up properly (including the integrate immediately checkbox), however my integrations sit in the "pending" state. I checked the system.log, and nothing seems to be happening.

sidebar main

This could be completely unrelated, but every time I click on a pending integration, I receive this error in system.log:

NSFileCoordinator only handles URLs that use the file: scheme. This one does not:
x-code-xcsbot://XXX

I'm not sure if this is a new problem introduced in OS X server 3.2.1, or if it's just a set up issue. Apparently no one else has had this issue, couldn't find anything on Google/ SO.

Dunc answered 30/9, 2014 at 19:15 Comment(1)

I don't know if this is related: I hosted a number of repositories on a Mac Mini, and everything worked fine. After I upgraded to OS X Server 3.2.1 with Xcode 6.0.1, I can no longer access my repositories <#25995839>. I believe it is an authorization issue, but I don't know how to approach it. – Farewell 30/9, 2014 at 20:8

This appears to happen when running XCode beta builds in OS X Server.

Note that this command will ❗️delete all your bots❗️
Run sudo xcrun xcscontrol --reset to reset.

https://devforums.apple.com/message/1051403#1051403

Dunc answered 6/10, 2014 at 21:45 Comment(1)

This just only happened to me with Xcode 6.2 (no beta) as well. Your solution did the trick for me, thank you – Knacker 18/3, 2015 at 13:50

This is still happening, but if you just want to nudge the server to wake up and run and aren't prepared to delete your x-code server configuration (provisioningProfiles, credentials and bots are deleted (as I recall)), simply run this terminal command

sudo -u _xcsbuildd /Applications/Xcode.app/Contents/Developer/usr/bin/xcsbuildd

Note that if you have multiple revs of Xcode in your Applications folder you may have named them differently, so the command may be slightly different. In my case, I've got a file named 'Xcode 6.1.1'. So the command would be

sudo -u _xcsbuildd /Applications/Xcode\ 6.1.1.app/Contents/Developer/usr/bin/xcsbuildd

Joselow answered 12/5, 2015 at 15:56 Comment(4)

this answer should get much more attention! – Swisher 2/8, 2015 at 15:31

This worked for me and is a much better option than deleting everything. Thanks! – Tobacco 29/10, 2015 at 17:23

I tried so many different things, but this is the one that worked. It is much better than resetting the whole server. Thanks so much for finding this out. – Darrel 30/10, 2015 at 17:55

This works for me but I have to do this every time i need to kick off a build. is that expected behaviour ? – Flanigan 10/3, 2016 at 16:54

This appears to happen when running XCode beta builds in OS X Server.

Note that this command will ❗️delete all your bots❗️
Run sudo xcrun xcscontrol --reset to reset.

https://devforums.apple.com/message/1051403#1051403

Dunc answered 6/10, 2014 at 21:45 Comment(1)

This just only happened to me with Xcode 6.2 (no beta) as well. Your solution did the trick for me, thank you – Knacker 18/3, 2015 at 13:50

Here's a solution that may fix the problem without requiring to reset Xcode Server.

What's the problem?

First, check whether this answer applies by inspecting the xcsnginx.log log file:

sudo tail /Library/Developer/XcodeServer/Logs/xcsnginx.log

Search for the following line at the end of the log:

nginx: [emerg] SSL_CTX_use_PrivateKey_file("/Library/Developer/XcodeServer/Certificates/xcsnginx.key") failed (SSL: error:0906A068:PEM routines:PEM_do_header:bad password read error:140B0009:SSL routines:SSL_CTX_use_PrivateKey_file:PEM lib)

I you don't see that entry in the log, I'm afraid this answer won't help you. If you do see that entry, you may continue.

Why does it happen?

Xcode Server internally runs an Nginx web server (on port 20543) named xcsnginx that acts as a proxy between some services. This server uses a TLS/SSL certificate to ensure communications remain secure. The involved files are the following:

xcsnginx.crt: contains the PEM certificate.
xcsnginx.key: contains the private key for the certificate.
xcsnginx.pass: contains the passphrase for the private key.

As far as I understand, the private key is stored unencrypted, which means the xcsnginx.pass should be empty (and seems to be reset every time Xcode Server starts).

However for some reason, as some point, the private key in xcsnginx.key was exported as an encrypted key. I've no idea how and why this could happen but it did happen on my server so I'll assume it may happen on your server too. The consequence is that xcsnginx cannot load the certificate and fails to launch.

You can verify that xcsnginx is not running by executing:

pgrep xcsnginx || echo "Not running"

How to fix it?

Rather than resetting Xcode Server from scratch, we can:

export the identity again from the xcsnginx.keychain keychain or
restore the previous certificate and key or
create a new certificate and key for xcsnginx.

So let's have a look at each option.

Option 1

Copies of the certificate and private key are stored in the xcsnginx.keychain keychain located in /Library/Developer/XcodeServer/Keychains. This keychain is protected by a passphrase stored in a file named XCSNginxKeychainSharedSecret in the /Library/Developer/XcodeServer/SharedSecrets folder.

If you're familiar with OS X keychains, you may retrieve the certificate and the key from the keychain.

However manipulating keychains using the command-line is a real nightmare so I'll let this as an exercise for the reader (or an editor).

Option 2

The /Library/Developer/XcodeServer/Certificates folder may contains a backup of your certificate and key. Let's find out:

sudo find /Library/Developer/XcodeServer/Certificates -name "*.original"

If you're lucky, you should get the following result:

/Library/Developer/XcodeServer/Certificates/xcsnginx.crt.original
/Library/Developer/XcodeServer/Certificates/xcsnginx.key.original
/Library/Developer/XcodeServer/Certificates/xcsnginx.pass.original

Which mean you can restore the original files:

sudo cp /Library/Developer/XcodeServer/Certificates/xcsnginx.crt.original /Library/Developer/XcodeServer/Certificates/xcsnginx.crt
sudo cp /Library/Developer/XcodeServer/Certificates/xcsnginx.key.original /Library/Developer/XcodeServer/Certificates/xcsnginx.key
sudo cp /Library/Developer/XcodeServer/Certificates/xcsnginx.pass.original /Library/Developer/XcodeServer/Certificates/xcsnginx.pass

Option 3

If you cannot restore the previous certificate and keychains, you may decide to just generate new ones like this:

sudo openssl req -new -x509 -newkey rsa:2048 -nodes -out /Library/Developer/XcodeServer/Certificates/xcsnginx.crt -keyout /Library/Developer/XcodeServer/Certificates/xcsnginx.key -subj /CN=your-server.example.com -days 1000 -batch

where your-server.example.com is replaced with the DNS address of your server. Ideally the certificate should be issued by the Xcode Server Root Certificate Authority but using a single-signed certificate doesn't seem to be a problem (as far as I now / for the moment / your mileage may vary).

Finally

Now we just have to wait until the system starts xcsnginx again. That should happen automatically after a minute or less. You can verify that xcsnginx did start with:

pgrep xcsnginx || echo "Not running"

Eclectic answered 18/4, 2016 at 18:13 Comment(2)

Thanks so much for this detailed answer! I just ran into this exact issue today and went with the /Applications/Xcode.app/Contents/Developer/usr/bin/xcsbuildd command before seeing your answer. Will definitely try this out next time this happens. – Stirring 22/4, 2016 at 20:42

Thanks for the great answer. I tried option 3, but unfortunately it did not help me - xcsnginx did not run again. – Farewell 5/2, 2018 at 18:46

You can use restart instead of reset from the command line to restart the server. This will terminate the bot you are running that has frozen up.

sudo xcrun xcscontrol --restart

Shote answered 6/11, 2015 at 20:17 Comment(1)

Worked for me. Spotted the same answer on Apple's Developer Forums forums.developer.apple.com/thread/3910 – Baguio 15/6, 2016 at 9:8

I've run into this several times. I've done the xcrun xcscontrol --reset and it does work, but I'm getting tired of that. As my bots get more complicated with pre/post integration triggers I don't want to set them up again.

What worked for me this time was just going through all mostly recent integrations and deleting any canceled ones. Cancelling a bot run can leave it in an odd state.

After I deleted a bunch of integrations I restarted the machine the server is running on and my integrations start working again!

Hope that helps someone.

Scratchboard answered 9/9, 2015 at 16:25 Comment(1)

This was exactly my case and weirdly enough, it worked for me as well. Thanks! – Linehan 26/11, 2015 at 11:18