SSHD Gives error could not open Authorized Keys, although permissions seem correct

Question

I'm unable to login to SSH because of the following error in /var/log/secure (according to the debug logs):

Dec 19 18:01:05 hostname sshd[25119]: debug1: trying public key file /root/.ssh/authorized_keys
Dec 19 18:01:05 hostname sshd[25119]: debug1: Could not open authorized keys '/root/.ssh/authorized_keys': Permission denied

I have the following permissions set on root

chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
chmod go-wrx ~

ls -lah gives the following output for those directories:

drwx------.   6 root root 4.0K Dec 19 17:46 root
drwx------.  2 root root 4.0K Dec 19 17:41 .ssh
-rw-------. 1 root root  416 Dec 19 17:12 authorized_keys

I know the key I'm using is correct, as I just setup another server with it without any problems.

I'm running: CentOS release 6.4 (Final)

I've added my sshd config in case there's something misconfigured in there that might be causing the issue:

#       $OpenBSD: sshd_config,v 1.80 2008/07/02 02:24:18 djm Exp $

# This is the sshd server system-wide configuration file.  See
# sshd_config(5) for more information.

# This sshd was compiled with PATH=/usr/local/bin:/bin:/usr/bin

# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented.  Uncommented options change a
# default value.

#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::

# Disable legacy (protocol version 1) support in the server for new
# installations. In future the default will change to require explicit
# activation of protocol 1
Protocol 2

# HostKey for protocol version 1
#HostKey /etc/ssh/ssh_host_key
# HostKeys for protocol version 2
#HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_dsa_key

# Lifetime and size of ephemeral version 1 server key
#KeyRegenerationInterval 1h
#ServerKeyBits 1024

# Logging
# obsoletes QuietMode and FascistLogging
#SyslogFacility AUTH
SyslogFacility AUTHPRIV
LogLevel DEBUG

# Authentication:

#LoginGraceTime 2m
PermitRootLogin yes
StrictModes no
#MaxAuthTries 6
#MaxSessions 10

RSAAuthentication yes
PubkeyAuthentication yes
AuthorizedKeysFile      .ssh/authorized_keys
#AuthorizedKeysCommand none
#AuthorizedKeysCommandRunAs nobody

# For this to work you will also need host keys in /etc/ssh/ssh_known_hosts
#RhostsRSAAuthentication no
# similar for protocol version 2
#HostbasedAuthentication no
# Change to yes if you don't trust ~/.ssh/known_hosts for
# RhostsRSAAuthentication and HostbasedAuthentication
#IgnoreUserKnownHosts no
# Don't read the user's ~/.rhosts and ~/.shosts files
IgnoreRhosts yes

# To disable tunneled clear text passwords, change to no here!
#PasswordAuthentication yes
#PermitEmptyPasswords no
PasswordAuthentication yes

# Change to no to disable s/key passwords
#ChallengeResponseAuthentication yes
ChallengeResponseAuthentication no

# Kerberos options
#KerberosAuthentication no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
#KerberosGetAFSToken no
#KerberosUseKuserok yes

# GSSAPI options
#GSSAPIAuthentication no
GSSAPIAuthentication yes
#GSSAPICleanupCredentials yes
GSSAPICleanupCredentials yes
#GSSAPIStrictAcceptorCheck yes
#GSSAPIKeyExchange no

# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication.  Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to 'no'.
#UsePAM no
UsePAM yes

# Accept locale-related environment variables
AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE
AcceptEnv XMODIFIERS

#AllowAgentForwarding yes
#AllowTcpForwarding yes
#GatewayPorts no
#X11Forwarding no
X11Forwarding yes
#X11DisplayOffset 10
#X11UseLocalhost yes
#PrintMotd yes
#PrintLastLog yes
#TCPKeepAlive yes
#UseLogin no
#UsePrivilegeSeparation yes
#PermitUserEnvironment no
#Compression delayed
#ClientAliveInterval 0
#ClientAliveCountMax 3
#ShowPatchLevel no
UseDNS no
#PidFile /var/run/sshd.pid
#MaxStartups 10:30:100
#PermitTunnel no
#ChrootDirectory none

# no default banner path
#Banner none

# override default of no subsystems
Subsystem       sftp    /usr/libexec/openssh/sftp-server

# Example of overriding settings on a per-user basis
#Match User anoncvs
#       X11Forwarding no
#       AllowTcpForwarding no
#       ForceCommand cvs server

Any ideas would be much appreciated.

Shipping

I am assuming you set StrictModes to no for testing? You should normally leave that as the default yes. What else have you changed in the sshd config? – Benumb

Answer 1

If the permissions are correct, SELinux might still be preventing sshd from opening the file.

Try fixing the labels inside the .ssh directory (and maybe $HOME):

restorecon -FRvv ~/.ssh

If the user account uses non-standard home path, default labels for the path need to be added to the local configuration first:

semanage fcontext -a -t ssh_home_t "/srv/custom/\.ssh(/.*)?"

(I'm intentionally not suggesting disabling SELinux or setting it to the permissive mode.)

Wop

+1, especially for "I'm intentionally not suggesting to disable SELinux or set it to permissive mode :)" Sounds like you believe this is the best way (we are on the same page :)) but you already know there will be a lot of uptight "moralists" who would give you a lot of downvotes :-) – Talesman

I ran into the same issue. I guess I need to read up more on SELinux. Looks like a powerful but confusing tool? – Prance

I had the same problem with the same cause, but restorecon didn't work. I had to change the SELinux type explicitly. chcon -Rv -t ssh_home_t ~/.ssh – Iolite

Jonathan's comment above is what I needed to do to solve my problem. I created the .ssh folder myself, so I'm thinking no security context was added to it (which is why "restorecon" didn't work...). I was using CentOS 7. – Poche

When SELinux is causing problems despite my never having touched it (and breaking SSH is a serious problem), and when I'm in a sandboxed, single-purpose VM with no internet access, disabling SELinux is absolutely the right answer. – Galipot

This works when SELinux labels are unconfined_u:object_r:unlabeled_t:s0 (see ls -Z). This answer resolved the problem, setting the SElinux labels to unconfined_u:object_r:ssh_home_t:s0 for .ssh/ and unconfined_u:object_r:user_home_dir_t:s0 for $HOME. – Unicameral

On Ubuntu I had to install it first: sudo apt-get install policycoreutils – Rahr

3 years later. This tripped me up. And this was the solution – Hone

this did the opposite of what i was hoping. i got ssh working with @gavenkoa's solution then tried this and ssh reverted to not not working. selinux trips over its own crank and the errors/documentation are inscrutable. </rant> – Augustaaugustan

Avoid using chcon for non-standard home directory paths and rather adjusting the local configuration using semanage fcontext. That way the correct labels will be set in case the whole filesystem need to be relabeled. – Marasmus

Answer 2

In case if SELinux enabled:

$ getenforce
Enforcing

to temporary enable pub-key ssl login to non-standard user home directory location run:

$ sudo chcon -t ssh_home_t /srv/jenkins/.ssh/authorized_keys /srv/jenkins/.ssh

$ ls -ldZ /srv/jenkins/.ssh/authorized_keys /srv/jenkins/.ssh/
drwxr-xr-x. jenkins jenkins system_u:object_r:ssh_home_t:s0  /srv/jenkins/.ssh/
-rw-r--r--. jenkins jenkins system_u:object_r:ssh_home_t:s0  /srv/jenkins/.ssh/authorized_keys

See https://linux.die.net/man/8/ssh_selinux for the details.

To make SELinux settings permanent run:

$ sudo semanage fcontext -a -t ssh_home_t /srv/jenkins/.ssh/authorized_keys
$ sudo semanage fcontext -a -t ssh_home_t /srv/jenkins/.ssh
$ sudo restorecon -R -v /srv/jenkins/.ssh/

You hit this if you are on modern RHEL, Oracle Linux, CentOS.

Whitehouse

spent an entire day on this problem and this is the only thing the worked for me on CentOS7 SELinux enabled. I dont understand the difference when you say temporary vs permanent. – Physics

Temporary - lasts to next reboot. Permanent - survives reboot. – Whitehouse

I feel this is going to be a long journey :) – Physics

how on earth should somebody figure this out? thanks man, you saved my day! – Centurial

Answer 3

I was struggling to use key authentication as well.

Could not open authorized keys '/home/myUserName/.ssh/authorized_keys2': Permission denied

Had checked all the above things when I ended up here (first link on google). I realize that this is an old post but I will add it here in case somebody else has the same problem as me and end up here.

I had owner of the authorized_keys file to "root", so changing it with:

chown myUserName authorized_keys2

Solved it for me.

Tonina

This helped me . In my case authorized_keys file was created by root/sudo user , hence it was showing 'root' as userowner & 'root' as groupowner . I changed it to my user & group using chown & chgrp command respectively , afterwards I was able to do ssh to server using my user. – Tokyo

Answer 4

I've spent a good couple hours with this crap. Reading loads of illuminated solutions to the issue, here is a boil down of what none of them say clearly.

Sumarizing.

The following applies to these files:

• ~/.ssh/
• ~/.ssh/authorized_keys

Ownership

The owner and group must be the logging user.

Permissions

Group and public permissions must be set to 0 (no permissions). Yes! sshd thinks you're stupid by default.

And of course, the owner must be allowed to read.

Note. Permissions such as 777 give the same error as 000, even though it is an application policy to not open the file. This was probably the biggest source of confusion during this whole endeavour.

Users

All logging users must exist. By logging users we mean:

• The user sent with the url ssh root@localhost
• Or, the user in the public key present on the client machine (~/.ssh/id_rsa.pub in the client, not in the server)

Ringside

Answer 5

Check the /home directory permissions. It should be

• drwxr-xr-x. 9 root root 113 Jun 28 22:57 home

and then your home directory detail:

• drwxr----- 5 user group 124 May 18 17:00 User drwx------ 2 user group 29 May 18 12:05 .ssh -rw------- 1 user group 2235 Jun 28 23:09 authorized_keys

My error messages in logs

/var/log/secure > sshd[22565]: error: Received disconnect from X.X.X.X: 14: No supported authentication methods available [preauth]

On client side

ssh [email protected] Permission denied (publickey). ssh -vvv [email protected] ... debug2: we did not send a packet, disable method debug1: No more authentication methods to try. Permission denied (publickey).
On server side

• service sshd stop

• run sshd debug mode:

• /usr/sbin/sshd -ddd

  ... debug1: trying public key file /home/USER/.ssh/authorized_keys debug1: Could not open authorized keys '/home/USER/.ssh/authorized_keys': Permission denied ...

Gammadion

Answer 6

In my case, home directories are on NFS, which means that ~/.ssh is also on NFS.

Found this answer, and indeed running the following (as root) solved the problem:

setsebool -P use_nfs_home_dirs 1

Perpend

Answer 7

A couple ideas to check:

• Can you cat authorized_keys? What does the file look like?
• Is your sshd configured to allow root login? This is generally frowned upon,
• Are you doing it as root or as a sudoer?

Profess

1. Yes, it looks correct, it has my key in there and that's it. I compared it to another server that I can ssh into without a password and it's exactly the same. 2. Added SSHD config to question above, that is enabled. 3. I'm not sure what you mean. When I login to the server remotely from my client pc I use "ssh root@hostname" and on the server right now I'm logged in as root making these changes. – Shipping

Answer 8

1. Don't do chmod on ~/.ssh/.... Try to write the exact path: /root/.ssh/..., since sometimes (when using su etc), the ~ can be setup incorrectly. Check and post the permissions again for the full path without using ~ in the command.

2. Once you are absolutely sure the permissions are OK, check if your sshd is actually running under user root: ps -A u | grep sshd.

Talesman

Thanks for the help. I'm logged in as root and reran those commands from inside the directory itself. No luck. And sshd is running as root. – Shipping

Answer 9

A couple of things to double-check:

1. Are you sure you copied the PUBLIC key to the authorized_keys, not the private key? :-)
2. Do cat -tv authorized_keys. Any ^M characters at the end of each line? Do a dos2unix on authorized_keys
3. Did you restart the ssh daemon after making configuration changes?

Seethrough

Thanks for your help. Yes, no bad characters, and yes. – Shipping

Answer 10

I encountered this same issue and got it solved by changing both .ssh and authorized_keys's owner at the same time: chown MyUsername:Myusername .ssh chown MyUsername:Myusername .ssh/authorized_keys

Thanks to @niclaslindgren.

And BTW, it's no matter with whether there is ^M in authorized_keys or not, I had tested and proved it, it works with both the ways

Perish

Answer 11

I'm going to throw my answer in here as well since I just wasted the last hour trying to figure out a workaround for another hour I wasted previously on Azure.

This does not work:

AuthorizedKeyFiles ~/.ssh/authorized_keys

Generating server-side logs using an alternate port (i.e. /usr/sbin/sshd -ddd -p 12345) reveals the following when using the above configuration:

debug1: trying public key file /root/.ssh/authorized_keys
debug1: Could not open authorized keys '/root/.ssh/authorized_keys': Permission denied

Either of these are correct, however:

AuthorizedKeyFiles /home/%u/.ssh/authorized_keys

AuthorizedKeyFiles .ssh/authorized_keys

See also: https://www.ssh.com/ssh/authorized_keys/openssh

Maulmain

Answer 12

For me it was also changing ownership, but not just of authorized_keys but also of the .sdd directory:

chown -R user:user ./home/user/.ssd

Bushmaster

Answer 13

There is a set of generic steps if one has to find the reason why sshd is refusing to accept a connection or keys. The details below are for a systemd based system but alternative systems users would be able to find their way easily.

How to debug sshd systematically?

Start watching the journal

journalctl -u sshd -f

Set sshd logging to debug mode in /etc/ssh/sshd_config

LogLevel DEBUG

Restart the daemon to let the change take effect

systemctl restart sshd

We are set on the server side.

Try the client connection now

ssh -o IdentitiesOnly=yes -v -i ~/.ssh/key_to_the_kingdom [email protected]

• IdentitiesOnly disables trying other than the identity specified.
• -v to increase the client verbosity so one can see if the client is doing what is expected. (Is able to find and use the key on the client system, is able to negotiate the encryption algorithm, etc.)

(Not posting an example for the client.)

---

On the server (kingdom.gov in our example), we should see something like the following in the debug mode log:

Jul 14 12:46:39 kingdom.gov sshd[4665]: debug1: userauth-request for user king service ssh-connection method none [preauth]
Jul 14 12:46:39 kingdom.gov sshd[4665]: debug1: attempt 0 failures 0 [preauth]
Jul 14 12:46:39 kingdom.gov sshd[4665]: debug1: PAM: initializing for "king"
Jul 14 12:46:39 kingdom.gov sshd[4665]: debug1: PAM: setting PAM_RHOST to "75.73.78.71"
Jul 14 12:46:39 kingdom.gov sshd[4665]: debug1: PAM: setting PAM_TTY to "ssh"
Jul 14 12:46:39 kingdom.gov sshd[4665]: debug1: userauth-request for user king service ssh-connection method publickey [preauth]
Jul 14 12:46:39 kingdom.gov sshd[4665]: debug1: attempt 1 failures 0 [preauth]
Jul 14 12:46:39 kingdom.gov sshd[4665]: debug1: userauth_pubkey: test pkalg rsa-sha2-512 pkblob RSA SHA256:0hjXPXkM8d91W2D8bg3fcapifm5QJd7QV9wwOEMU1 [preauth]
Jul 14 12:46:39 kingdom.gov sshd[4665]: debug1: temporarily_use_uid: 112233/10 (e=0/0)
Jul 14 12:46:39 kingdom.gov sshd[4665]: debug1: trying public key file /home/king/.ssh/authorized_keys
Jul 14 12:46:39 kingdom.gov sshd[4665]: debug1: Could not open authorized keys '/home/king/.ssh/authorized_keys': Permission denied
Jul 14 12:46:39 kingdom.gov sshd[4665]: debug1: restore_uid: 0/0
Jul 14 12:46:39 kingdom.gov sshd[4665]: Failed publickey for king from 83.69.65.84 port 52756 ssh2: RSA SHA256:0hjXPXkM8d91W2D8bg3fcapifm5QJd7QV9wwOEMU1

In my case, the problem was in SELinux not allowing to use authorized_keys stored in the NFS home directory.

You may be asking: How can I check the permissions? What is the identity sshd uses to access the files?

Look at temporarily_use_uid: 112233/10 in the log above. There should be correct UID and primary GID for the user. In my case, these values were taken from the name service (LDAP) and were as expected.

If the client user identity is incorrect, look into the name service configuration and resolve this issue first.

The directory and file ownership and permissions were as expected (at least u=x for ~ and ~/.ssh directories, at least u=r for authorized_keys, if owned by the user). It was clear the reason must be in something like SELinux.

Let's check it:

getsebool use_nfs_home_dirs
  use_nfs_home_dirs --> off

For that case, the answer by Nadav Aharoni resolves the problem.

 setsebool -P use_nfs_home_dirs 1

Cleanup

• Restore LogLevel DEBUG in sshd_config to the previous value. (Or comment it out to restore the default.)
• systemctl restart sshd

Highstepper