SSH Key authentication failing when connecting Mac Hudson slave to Linux master

Question

Ok, so I have Hudson (v1.393) running in an Ubuntu VM and everything's working fine. However I'm trying to add a Mac slave to the Ubuntu master and I've run in to a few problems.

I have set up SSH keys so that from the command line, the Ubuntu VM can ssh using the key into a user called hudson on the Mac.

In the Hudson slave configuration, I have "Launch slave agents on Unix machines via SSH" selected and have entered the host IP, username of the user on the slave and the location of my private key file on the master (which has been added to the authorised keys file on the slave).

However, the master fails to connect to the slave. Looking at the log (below), it's trying to authenticate using a password.

Is this a fall back for a failed key based SSH attempt?
Is Hudson only trying to authenticate using a password, and I need to change something else to get it to use the key file which is defined in the configuration?
Is it just not possible to launch slave agents via ssh on a mac? (I know the name of this type of slave launch method explicity states Unix, but I was thinking (read: hoping) that it would work with OS X too)

Log

[01/14/11 10:38:07] [SSH] Opening SSH connection to 10.0.1.188:22.
[01/14/11 10:38:07] [SSH] Authenticating as hudson/******.
java.io.IOException: Password authentication failed.
at com.trilead.ssh2.auth.AuthenticationManager.authenticatePassword(AuthenticationManager.java:319)
at com.trilead.ssh2.Connection.authenticateWithPassword(Connection.java:314)
at hudson.plugins.sshslaves.SSHLauncher.openConnection(SSHLauncher.java:565)
at hudson.plugins.sshslaves.SSHLauncher.launch(SSHLauncher.java:179)
at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:184)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.io.IOException: Authentication method password not supported by the server at this stage.
at com.trilead.ssh2.auth.AuthenticationManager.authenticatePassword(AuthenticationManager.java:289)
... 9 more
[01/14/11 10:38:07] [SSH] Connection closed.

If anyone has managed to conquer this type of set up before, or has any tips or ideas, I'd be very grateful! Thanks

Ryeland

Have you tried without the private key and with a password? See if it connects using that method.. – Aphyllous

Have you tried connecting to the OS X box from the VM with jsut ssh to see if key auth is enabled? Is the OS X box Server or Client version? Did you leave the password blank when you generated the key? – Motherwort

@Aphyllous : Just tried using a password rather than a key. No luck, same messages in the log. – Ryeland

@Motherwort : ssh from the command line on the Ubuntu VM to the Mac works fine. I originally had a password on the ssh key, but then I created a new key without a password as I thought that could be the issue. The OS X machine is not the server version, just standard OS X 10.6.6. – Ryeland

Answer 1

I've recently run into the same problem, trying to launch an agent on a Mac OS X 10.6 machine using SSH.

To get password authentication to work you'll need to edit /etc/sshd_config on the client node, setting PasswordAuthentication yes

In the Hudson dashboard take the node offline, make sure the configuration has a valid username and password, and launch the agent. Also make sure that the Remote FS root directory is owned by the build user you're connecting as.

For password-less ssh authentication, first check which user the Hudson master is running as. Lets assume that this is tomcat55. Generate a public/private SSH key pair (with an empty passphrase), then verify that the Hudson user can connect.

$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/tomcat55/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/tomcat55/.ssh/id_rsa.
Your public key has been saved in /home/tomcat55/.ssh/id_rsa.pub.

$ # authorize the hudson master on the hudson node
$ scp /home/tomcat55/.ssh/id_rsa.pub hudson@macnode:~/.ssh/authorized_keys
$ # test the connection
$ ssh -i /home/tomcat55/.ssh/id_rsa hudson@macnode

On the Hudson mac node, the /etc/sshd_config needs to allow for password-less access.

Protocol 2
PubkeyAuthentication yes

In the node configuration clear the password field, and set the private key field (in this example it is /home/tomcat55/.ssh/id_rsa). You should now be able to launch the agent:

[01/19/11 22:38:44] [SSH] Opening SSH connection to macnode:22.
[01/19/11 22:38:44] [SSH] Authenticating as hudson with /home/tomcat55/.ssh/id_rsa.
[01/19/11 22:38:45] [SSH] Authentication successful.

Ambiguity

Thanks Ronen! The password authentication works fine. However, if I remove the password and rely on the ssh key, it still fails, and Hudson still seems to be relying on the password - "[01/02/11 13:11:45] [SSH] Authenticating as mattbilson/******." Is it something else in my sshd_config? I have "RSAAuthentication yes", "PubkeyAuthentication yes", "AuthorizedKeysFile .ssh/authorized_keys" – Ryeland

The SSH plug-in falls back to password if it can't use the private key to log into the node. I'd leave the password field empty while you try to figure things out, to make sure it's not the PasswordAuthentication that's letting it in. Also, what's happening is ssh -i KEYFILE USER@HOST . If you can do so manually as the hudson user then Hudson should be able to do so as well. Are you able to su to hudson user, then try to ssh from command line? If there's any problem you'll most likely find it this way. – Ambiguity

That's it! Hudson runs on the hudson user, but I'm on the ubuntu machine as a different user. And hudson didn't have permissions for my key file. So now I've created a key for the hudson user and added it to the Mac. Turns out it was a pretty simple mistake. Thanks for the info on what hudson was actually doing Ronen! – Ryeland

@RonenBotzer: Having a same problem please have a look at it. #44195494 question posted. – Caleb

@mattbilson: Having a same problem please gave a look, #44195494 – Caleb

Answer 2

Check the /var/log/auth.log file on the Ubuntu machine. I'm betting you need to chmod 700 the .ssh directory of the hudson user.

Maltha

Just tried 700-ing that file, permissions are now "-rwx------". Still no luck. – Ryeland

Answer 3

I think the first answer (the selected one) is an awesome answer, but I did find a case where it is not the only solution.

In my case I have a Mac OS slave that was working and then I took that Mac down and brought up a new one. I thought I could just tweak the settings for the existing node's configuration to point it at the new Mac. It didn't work and I had all the same errors and problems described throughout this message thread.

Then I went in and deleted the node and recreated it with exactly the same settings and it worked. I suspect that SSH key fingerprint changed and by deleting the node and recreating it I was able to get it working. Whatever it is, the key component that caused it to fail is not a configuration option.

Mulligan