Why Jenkins says "Server rejected the 1 private key(s)" while launching the agent?

Question

I am successfully able to connect to remote machine using SSH but when I am launching the agent from Jenkins it throws the following error:

ERROR: Server rejected the 1 private key(s) for user1 (credentialId:xxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/method:publickey)
[01/19/17 05:35:15] [SSH] Authentication failed.
hudson.AbortException: Authentication failed.
    at hudson.plugins.sshslaves.SSHLauncher.openConnection(SSHLauncher.java:1219)
    at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:714)
    at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:709)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
[01/19/17 05:35:15] Launch failed - cleaning up connection
[01/19/17 05:35:15] [SSH] Connection closed.

I can establish SSH connection from master machine to the node machine using user1, however when I am trying to launch the agent using user1 from jenkins it is rejecting the private key. Is there any solution to overcome this issue?

Nerves

Answer 1

I solve this issue following below steps:

From the target slave node's console

1. Switch to the root user:

sudo su

2. Add a jenkins user with the home /var/lib/jenkins (Note: I am keeping my home directory in /var/lib/jenkins):

useradd -d /var/lib/jenkins jenkins

From the Jenkins Master

Copy the /var/lib/jenkins/.ssh/id_rsa.pub key from the Jenkins user on the master

From the target slave node's console

1. Create an authorized_keys file for the Jenkins user

mkdir /var/lib/jenkins/.ssh
touch /var/lib/jenkins/.ssh/authorized_keys

2. Paste the key from the Jenkins master into the file vim. Save with :wq!

3. Make sure the files have correct owner and permission.

chown -R jenkins /var/lib/jenkins/.ssh
chmod 600 /var/lib/jenkins/.ssh/authorized_keys
chmod 700 /var/lib/jenkins/.ssh

Smooth

On the first run Jenkins needs to copy remoting.jar to /var/lib/jenkins, so jenkins user must be owner of /var/lib/jenkins directory as well. Everything that is mentioned in this answer is correct, but in my case I had to execute following command additionally: chown -R jenkins /var/lib/jenkins – Poesy

If you're using Fedora or a similar distro (e.g. RHEL), do not disable PAM authentication on your SSH server. Check file /etc/ssh/sshd_config and make sure you have UsePAM yes. I found this by running sudo systemctl status sshd and seeing warnings about it. The file itself tried helping me: # WARNING: 'UsePAM no' is not supported in Fedora and may cause several problems.. – Kehoe

@Kehoe thank you so much. I've tried every single configuration on how to... I was banging my head to wall out of desperation and your solution helped me a lot. P.s I'm using oraclelinux. – Manifestation

Worked for me as well – Frequently

The key thing I was missing was changing authorized_keys file permissions to 600. Thanks! – Mikamikado

Another thing I had to do was chmod 700 . in the target user home directory (the parent of .ssh directory) – Oteliaotero

Answer 2

Changing type of ssh key from 'rsa' to 'ed25519' worked for me

ssh-keygen -t ed25519

Angstrom

OMG? I wasted 4 hours and changing this to ed25519 worked... – Hoch

Worked for me too! – Barger

Answer 3

I solved this issue by following the below steps:

1) Make sure you are on correct path in both slave and master machines. You also need to sign in to the machines with the right user. Say I need to create a new global jenkins user "jenkins" and I want my keys to be in the path "/home/jenkins/.ssh/", add "jenkins" user to the machines first.

2) Now create .ssh folder and generate ssh keys using the steps given in https://support.cloudbees.com/hc/en-us/articles/222978868-How-to-Connect-to-Remote-SSH-Slaves-

3) Make sure you do the above steps - 1 & 2 in your master machines as well

4) You need to have ssh keys in both master and slave machines in the same path and with same "jenkins" user permissions.

5) Finally, ssh both machine IPs to and fro to check the bidirectional connectivity from your terminal.

6) Configure jenkins credentials and nodes. Make sure you give the same remote root directory - "/home/jenkins" in your node configuration and select "manually trusted key verification strategy" - as suggested in https://linuxacademy.com/community/posts/show/topic/16008-jenkins-adding-a-slave

Nikolos

Answer 4

My Solution was:

$ user add -d /var/lib/jenkins jenkins
$ sudo su
$ passwd jenkins
$ chown -R jenkins /var/lib/jenkins/.ssh/*
$ chmod 700 .ssh

It worked after tampering around for 2 hours...

Teishateixeira

Not sure why you've been downvoted -- the chown change for the jenkins user is exactly what got me, and it would have been ages before I figured that out on my own. Thanks! – Retinite

Answer 5

The master needed to be added the list of known hosts for me. What you need to do is SSH to the master from your local. Then use the masters private key to SSH to the slave. If you can do this manually, then Jenkins will be able to do it as well.

I used the masters private key as the credential in Jenkins, followed @Aamir's answer then finally some success.

Yiddish

Answer 6

For CIS hardened machines, the problem could also be the user not being listed in AllowUsers option in sshd config file.

In my case, I changed the last line in /etc/ssh/sshd_config

from AllowUsers ec2-user

to AllowUsers ec2-user jenkins

and then restarted the sshd service: systemctl restart sshd

Whippletree