TL;DR You can fix this by sending yourself the ota.mobileconfig
file - see at bottom of this answer
What is happening
When you click that Install button, the IPA file is not downloaded from https://yourxcode.local but instead is downloaded from https://yourxcode.local:20343
The server that listens on https://yourxcode.local:20343 is actually not the same Apache server that runs on your OS X Server. It is a separate Node.js application that is part of the Xcode Server setup.
For reasons only known to the people at Apple who built this, this Node.js app uses a self-signed certificate and NOT the SSL certificate that you maybe already had installed on your OS X Server. (I really don't understand why they do this, it makes no sense)
So, to allow that self-signed certificate to work on your device, your Xcode Server is offering you an over-the-air mobile configuration profile that contains the root certificate for your this self-signed Node.js app.
That is what you see when you first hit the Install button the first time: Safari is asking you if you want to accept and install that new certificate.
Now if the installation of this certificate failed, it seems iOS still marks it as installed, even though it did not actually install correctly. There is no UI to undo this, or to remove the certificate, so there is no way to repeat the above.
You are now stuck with a device that you cannot use for downloading builds from Xcode Server unless you wipe your device. (No joke)
( I think it used to be possible to reset this info with the iPhone COnfiguration Utility, but that stopped working alltogether with iOS8)
Solution
No worries, I have found a workaround.
On your OS X Server, there is a file called
/Library/Developer/XcodeServer/ConfigurationProfiles/ota.mobileconfig
Email this file to yourself. On the iOS device where you get the Cannot connect to yourxcode.local
error, open this file from Mail.app as an attachment. Your device will ask you again if you want to install this certificate. Click through it. Answer Yes.
Go back to build summary screen and hit Install again. For some reason it will again ask you to accept the certificate once more. Click Yes and Confirm.
The application will now install on your device.