Login/Register
Missing certificates and keys in the keychain while using Jenkins/Hudson as Continuous Integration for iOS and Mac development
Asked Answered
Solved ioshudsonjenkinskeychainlaunchd
A

10

48

I'm trying to improve Hudson CI for iOS and start Hudson as soon as system starts up. To do this I'm using the following launchd script:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>Hudson CI</string>
    <key>ProgramArguments</key>
    <array>
    <string>/usr/bin/java</string>
    <string>-jar</string>
    <string>/Users/user/Hudson/hudson.war</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
    <key>UserName</key>
    <string>user</string>
</dict>
</plist>

This works OK but when xcodebuild, which is started by Hudson, tries to sign an app it fails because it cant find the proper key/certificate in the keychain. However key/certificate pair is there since it's working correct if I start Hudson from command line.

Do you have any ideas why it happens?

Amphiprostyle answered 26/7, 2011 at 9:20 Comment(0)
O
23

After spending hours and days with this issue I found a fairly easy solution to this. It doesn't matter if you have a distinct username in your launchd configuration as stated above:

<key>UserName</key>
<string>user</string>

The missing certificates and keys have to be on the system keychain (/Library/Keychains/System.keychain). I found this after I setup a jenkins job which executes several security shell calls. The one which's interesting is security list-keychains:

+ security list-keychains
    "/Library/Keychains/System.keychain"
    "/Library/Keychains/applepushserviced.keychain"
    "/Library/Keychains/System.keychain"

That are the keychains jenkins will search the certificates and keys for so they should be there. After I moved my certs there it works. Make sure you also copy the »Apple Worldwide Developer Relations Certification Authority« certificate to the system keychain, otherwise you will see a CSSMERR_TP_NOT_TRUSTED error from codesign.

It is also possible to register more keychains with security list-keychains -s [path to additional keychains]. I haven't tried it but something like security list-keychains -s $HOME/Library/Keychains/login.keychain as a pre-build shell execution in jenkins might work.

EDIT: I've tried to add a user keychain to the search path with -s but I wasn't able to get it to work. So for now, we have to copy our certs and keys into the system keychain.

EDIT^2: Read and use joensson' solution instead of mine, he managed it to access the users keychain instead of just the system keychain.

Occipital answered 18/10, 2011 at 9:54 Comment(4)
But how do you unlock the system keychain when no one is logged in? A sudo command? – Omor
@Omor I think you don't need a password for the System.keychain. But in case I'm messed up here, you can call security unlock -p password /path/to/System.keychain with a password. – Occipital
I created a simple job that executed "security list-keychains" found that Jenkins used $JENKINS_HOME/Library/Keychains/login.keychain regardless of which user the daemon ran as or which options I added to the command. So, I gave in and copied my desired keychain to $JENKINS_HOME/Library/Keychains/login.keychain and it worked. – Ula
Fiddling with the System.keychain has side effects for all installed applications. joensson has a better solution below. – Hereditary
F
80

I have found a solution giving me access to the regular keychains for my Jenkins user.

Find this plist: /Library/LaunchDaemons/org.jenkins-ci.plist then:

  • Add the UserName element with a value of jenkins.
  • Add a SessionCreate element with a value true to the plist file. This gives access to the normal keychains for the user you specified in UserName

Example:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>EnvironmentVariables</key>
    <dict>
        <key>JENKINS_HOME</key>
        <string>/Users/Shared/Jenkins/Home</string>
    </dict>
    <key>GroupName</key>
    <string>wheel</string>
    <key>KeepAlive</key>
    <true/>
    <key>Label</key>
    <string>org.jenkins-ci</string>
    <key>ProgramArguments</key>
    <array>
        <string>/bin/bash</string>
        <string>/Library/Application Support/Jenkins/jenkins-runner.sh</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
    <key>UserName</key>
    <string>jenkins</string>
    <key>SessionCreate</key>
    <true/>
</dict>
</plist>

Then restart the daemon and try running a job in Jenkins that calls security list-keychains. You should no longer see System.keychain as the only entry but the regular login and any custom key chains you might have added to the list of keychains for the "jenkins" user.

With the above setup I am able to use codesigning certificates from a custom keychain on my Jenkins build server. I don't have to install any certificates or keys in my System keychain.

Footprint answered 28/2, 2012 at 13:22 Comment(12)
Great solution! Use this one, instead of mine! – Occipital
THANK YOU! This is exactly what I needed. Too bad it isn't documented. I created a bug report for it rdar://11708751 – Mercurous
I already have my login.keychain in the list when I execute the command security list-keychains. The problem is that Jenkins still outputs: security: SecKeychainUnlock /Users/Shared/Jenkins/Library/Keychains/login.keychain: The specified keychain could not be found. – Saliva
FYI, a change log of Jenkins 1.477, "Changed defaults for the Mac installer to make iOS codesigning easier.", seems to resolve this issue by default. – Vibratile
In addition I had to add the custom keychain to the list of keychains with security default-keychain -s ~/Library/Keychains/codesign.keychain as otherwise security find-identity ~/Library/Keychains/codesign.keychain would show the identities as CSSMERR_TP_NOT_TRUSTED. – Laddy
In addition to this, I had to add in a jenkins job 'security unlock-keychain -p "password" path/to/keychain. Build job and remove the line in the jenkins job. – Terri
Would it be possible to use this approach for org.apache.httpd.plist? I have a php script that is running security commands and am running into this exact issue. – Boson
Sorry for the late reply, but yes I think you can. I have not tried it though. The plist format/commands above are not specific to jenkins - as you can see it is simply executing a bash script - the rest is just setting the environment for the script. The script can contain anything you like. – Footprint
I wish I could say this worked for me, because I need this very thing. But after adding it, list-keychains only shows System.keychain and default-keychain the same. It's as if it had no effect at all. [using Yosemite] – Mcnamara
That was the last missing piece on a journey full of pitfalls. Thank you very much! – Parthenogenesis
The piece that I was missing to have the needed Keychain be available was the <key>SessionCreate</key><true/> BUT ALSO had to do a reboot because a launchctl unload and then launchctl load was not sufficient. A full reboot was needed. – Exclave
Wow, that's obscure. This also worked for me to let CircleCI Self-Hosted Runners sign code on a Mac, just added this to the plist. – Cantlon
O
23

After spending hours and days with this issue I found a fairly easy solution to this. It doesn't matter if you have a distinct username in your launchd configuration as stated above:

<key>UserName</key>
<string>user</string>

The missing certificates and keys have to be on the system keychain (/Library/Keychains/System.keychain). I found this after I setup a jenkins job which executes several security shell calls. The one which's interesting is security list-keychains:

+ security list-keychains
    "/Library/Keychains/System.keychain"
    "/Library/Keychains/applepushserviced.keychain"
    "/Library/Keychains/System.keychain"

That are the keychains jenkins will search the certificates and keys for so they should be there. After I moved my certs there it works. Make sure you also copy the »Apple Worldwide Developer Relations Certification Authority« certificate to the system keychain, otherwise you will see a CSSMERR_TP_NOT_TRUSTED error from codesign.

It is also possible to register more keychains with security list-keychains -s [path to additional keychains]. I haven't tried it but something like security list-keychains -s $HOME/Library/Keychains/login.keychain as a pre-build shell execution in jenkins might work.

EDIT: I've tried to add a user keychain to the search path with -s but I wasn't able to get it to work. So for now, we have to copy our certs and keys into the system keychain.

EDIT^2: Read and use joensson' solution instead of mine, he managed it to access the users keychain instead of just the system keychain.

Occipital answered 18/10, 2011 at 9:54 Comment(4)
But how do you unlock the system keychain when no one is logged in? A sudo command? – Omor
@Omor I think you don't need a password for the System.keychain. But in case I'm messed up here, you can call security unlock -p password /path/to/System.keychain with a password. – Occipital
I created a simple job that executed "security list-keychains" found that Jenkins used $JENKINS_HOME/Library/Keychains/login.keychain regardless of which user the daemon ran as or which options I added to the command. So, I gave in and copied my desired keychain to $JENKINS_HOME/Library/Keychains/login.keychain and it worked. – Ula
Fiddling with the System.keychain has side effects for all installed applications. joensson has a better solution below. – Hereditary
C
6

We had the same problem with a hudson slave started as a launchdaemon on Mac OSX Lion. It worked, when we started the slave with webstart. The only difference we spotted was a different environment variable.

com.apple.java.jvmTask=WebStart

works, if we started the slave without webstart the variable was

com.apple.java.jvmTask=CommandLine.java

We found no way to influence the value upfront. I suggest you create a new node in Hudson, running on the same machine and started by webstart. For starting the slave we use the following launchdaemon configuration:

<?xml version"1.0" encoding="UTF-8"?>
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>jenkins</string>
    <key>UserName</key>
    <string>apple</string>
    <key>Program</key>
    <string>/usr/bin/javaws</string>
    <key>ProgramArguments</key>
    <array>
        <string>-verbose</string>
        <string>-wait</string>
        <string>http://<hudson-hostname>:8080/computer/<node-name>/slave-agent.jnlp</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
    <key>KeepAlive</key>
    <true/>
    <key>WorkingDirectory</key>
    <string>/Users/apple</string>
</dict>
</plist>
Chromatid answered 15/9, 2011 at 15:10 Comment(3)
Thanks. I'll try it as soon as I have time. – Amphiprostyle
I'm facing the exact same problem here, did you find a solution for that? Would love to here it. – Occipital
@DASKAjA The LaunchDaemon configuration above is our solution to the problem. By starting the slave via webstart it is able to access the keychain. As another approach it may be possible to move the keys and certificate from the login keychain to the system keychain. But I only heard this would work but never tried it myself. – Chromatid
I
3

I faced the same problem, and tried changing the user name in /Library/LaunchDaemons/org.jenkins-ci.plist as described in one of the other posts. However, it still did not work, and some obscure NullPointerException did not help me identify the problem. Therefore, I would just share my solution: I had to also change the owner of the JENKINS_HOME directory (defined in org.jenkins-ci.plist as well):

chown -R myBuildUser /Users/Shared/Jenkins

myBuildUser is the user that has the certificates installed, and this is the user that I specified in the plist file.

This solution was quite obvious when I finally realized it - but it took me a couple of hours to find out about this, so hopefully this post can save the time for somebody else :-)

Intrastate answered 14/11, 2012 at 7:31 Comment(0)
C
3

Adding this since I had the same problem, but none of these solutions worked for me.

My problem was that my signing certificate had expired. After the update, xcode and running xcodebuild manually worked fine, BUT Jenkins could not sign the app.

Here is how I fixed it:

  1. Look into Keychain and search for the key. For some reason that I don't understand I had multiple results.

  2. Make sure that the private key is in the System level (if it isn't then drag and drop it to the System icon on the left.

enter image description here

Cottonseed answered 29/8, 2018 at 14:16 Comment(0)
U
2

We faced exactly the same issue on Lion as well as on SnowLeopard. We had to start a Tomcat/Hudson with xcodebuild jobs as a service. While starting from command line, the xcodebuild could access the login.keychain to use the certificate contained. But after reboot of the box, the login.keychain wasnt visible to xcodebuild and therefore the signing failed.

Since we needed to provide our company certificate by a keychain, the system keychain wasnt an option. Instead, we solved the issue by a simple workaround. We removed the user name, so that the launch daemon launches the process under root. 

<plist version="1.0">
 <dict>
   <key>Label</key>
   <string>${LAUNCH_LABEL}</string>
   <key>Disabled</key>
   <false/>
   <key>RunAtLoad</key>
   <true/>
   <key>ProgramArguments</key>
   <array>
     <string>${INSTALL_DIR}/start.sh</string>
   </array>
   <key>StandardOutPath</key>
   <string>${INSTALL_DIR}/tomcat-stdout.log</string>
   <key>StandardErrorPath</key>
   <string>${INSTALL_DIR}/tomcat-stderr.log</string>
 </dict>
</plist>

The launch daemon called a simple script (start.sh), simulation a full login and running the program wanted

su -l username -c program

Now, even after booting, the xcodebuild can access the login.keychain. This works on Snow Leopard too, but, if you close the user specific login.keychain in a parallel session (like vnc login/logout) the keychain gets lost. Lion behaves different. Seems that Lion decouples the keychain from the user and assigns it to a login-session.

Underbelly answered 8/11, 2011 at 13:11 Comment(0)
R
2

You could try my Jenkins.app, https://github.com/stisti/jenkins-app, an alternative way to run Jenkins. It runs Jenkins in the user session, so Keychain access is not a problem.

Rentschler answered 1/4, 2012 at 22:8 Comment(0)
E
0

To keep a compartmentalized keychain for Jenkins/Hudson, I moved the launchctl item from

/Library/LaunchDaemons/org.jenkins-ci.plist

to

/Users/Shared/Jenkins/Home/Library/LaunchAgents/org.jenkins-ci.plist

And that allows me to access the private keychain created for Jenkins.

Eskimoaleut answered 22/11, 2011 at 1:54 Comment(4)
I believe your answer only makes it start as soon as a Jenkins user logs in? – Omor
That's correct. I couldn't find a way without logging in the user. – Eskimoaleut
Because then our solutions aren't much different: 'my' Jenkins starts on boot, but I still have to log in the user :P – Omor
Having spent some time dealing with this, LaunchAgents and LaunchDaemons have different capabilities, beyond simply when and how they are started and these affect the behavior of the build process (especially when building mac apps). – Placatory
P
0

Adding SessionCreate and setting lots of certificates to 'always trust' in keychain manager worked for me with buildbot started from plist... but at some point, codesign started failing with CSSMERR_TP_NOT_TRUSTED. I recovered by setting the iPhone Distribution cert to 'use system defaults' in keychain manager. Even after a reboot, without logging in, the buildbot slave was then able to sign code, whew.

Pseudohemophilia answered 29/4, 2013 at 18:11 Comment(0)
M
0

For Manual Signing Move your certificate from login to System in keychain. Login not accessible during archive and generating iPA.

Moderate answered 6/9, 2018 at 4:10 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.