paramiko, isn't talking to ssh-agent. same behavior in fabric

Question

Firstly I tried to get fabric working, but it kept asking me for a password.

So I'm trying to reduce the problem. Perhaps it would be a good POC to just create a SSH connection from Python. I discovered that fabric uses parmiko for it's SSH handling. Hmm. Ok, lets try to get an example working.

Here's what I wrote.

from ssh import *
import os 

print "SSH-AGENT VARS"

print "SSH_AGENT_PID: %s " % os.environ['SSH_AGENT_PID']
print "SSH_AUTH_SOCK: %s " % os.environ['SSH_AUTH_SOCK']

a = Agent()
keys=a.get_keys()
print keys.count("192.168.1.10")


client = SSHClient()
client.load_system_host_keys()
client.connect('192.168.1.10')

Resulting in the following error messages:

% ./ssh_test.py
SSH-AGENT VARS
SSH_AGENT_PID: 26557 
SSH_AUTH_SOCK: /tmp/ssh-pZHBElj26556/agent.26556 
0
Traceback (most recent call last):
  File "./ssh_test.py", line 18, in <module>
    client.connect('192.168.1.10')
  File "/usr/local/lib/python2.7/dist-packages/ssh/client.py", line 332, in connect
    self._auth(username, password, pkey, key_filenames, allow_agent, look_for_keys)
  File "/usr/local/lib/python2.7/dist-packages/ssh/client.py", line 493, in _auth
    raise saved_exception
ssh.PasswordRequiredException: Private key file is encrypted

ssh-agent is running in my session, I can SSH to that box, no problems, it doesn't prompt me for a password or anything.

I'm guessing paramiko isn't able to connect to the running ssh-agent for some weird reason.

Has anyone else had a problem like this? I'm using Ubuntu 11.10

I seem to remember trying Fabric a while back and having similar problems, perhaps it's been broken for a while?

I connect, just using the host name as the argument. This is as per the documentation.

http://www.lag.net/paramiko/docs/paramiko.SSHClient-class.html

connect(self, hostname, port=22, username=None, password=None, pkey=None, key_filename=None, timeout=None, allow_agent=True, look_for_keys=True, compress=False)

Dorthydortmund

I'm using fabric in Ubuntu 11.10 too, without any problems. Sounds like either fabric is not accessing the same key file as your console ssh, or you are using the wrong login. did you remember to set env.user = 'my_user' in your fabfile? – Butacaine

Tried that, same problem. I'm guessing that the problem lies with paramiko as opposed to fabric. Getting paramiko working first would be a good step. That's why I created the little test script above. – Dorthydortmund

Have you tried creating a branch new RSA key, making sure it has no passphrase, installing it on the server, and using it instead? – Butacaine

My RSA key does have a passphrase, but as the snippet indicates, I'm using ssh-agent, with which it is able to communicated but not able to use the private key. Rsync, and friends can obtain the private-key through ssh-agent - so I am expecting that paramiko could do the same· – Dorthydortmund

It's funny how this exception isn't even listed in the docs. – Mobilize

Did you try client.connect('192.168.1.10', password='your_passphrase')? I have a feeling this could help. – Mobilize

@LevLevitsky actually it is in the docs: lag.net/paramiko/docs/paramiko.PKey-class.html#from_private_key "if the private key file is encrypted, and password is None" – Butacaine

Exactly, I just didn't want to post much text before knowing that it works :) – Mobilize

Well, the OP doesn't rush to test it, so I posted it as an answer :) – Mobilize

But ssh-agent is running, and authenticated in my console. I can ssh to the server without providing a password. My question is how to achieve this with paramiko. I don't want to put the password in the code, that would negate the entire purpose of the exercise. – Dorthydortmund

Answer 1

Ok, so the first thing I discovered was that Paramiko is way out of date, and unmaintained.

It's now known as package ssh, at least under Ubuntu, and has a different maintainer (bitprophet)

• Here's a demo class that works exactly as described: https://raw.github.com/bitprophet/ssh/master/demos/demo.py

• It requires this file, for interactive prompts: https://github.com/bitprophet/ssh/blob/master/demos/interactive.py

Here's a sample session, using it:

$ ./ssh_demo.py
Hostname: 192.168.1.10
*** Host key OK.
Username [bryan]: root
Trying ssh-agent key eee5638f390e1698898984b10adfa9317 ... success!
*** Here we go!

Linux top.secret.com 2.9.37-1-amd64 #1 SMP Thu Nov 3 03:41:26 UTC 2011 x86_64
┌┌(root@top)-(10:44am-:-03/27)┌-¨-¨¨˙

That doesn't answer the question of why fabric isn't authenticating against the ssh-agent correctly thought. So the question remains open.

Update:

Thanks to Morgan's hint, I've gotten a little further with this problem. As he suggested, I enabled ssh logging by adding the following to the top of my fabfile.py

from fabric.api import *
import ssh
ssh.util.log_to_file("paramiko.log", 10)

I also monitored the server log. In doing so I discovered that the user which I specified was being disregarded and my local username used instead.

On the server:

tail -f /var/log/auth.log 
Mar 28 11:12:36 xxxxxxxxxxx sshd[17652]: Invalid user bryan from xxx.xxx.xxx.xxx

Locally:

tail -f paramiko.log    
DEB [20120328-11:39:29.038] thr=1   ssh.transport: starting thread (client mode): 0x8dfc66cL
INF [20120328-11:39:29.066] thr=1   ssh.transport: Connected (version 2.0, client OpenSSH_5.5p1)
DEB [20120328-11:39:29.093] thr=1   ssh.transport: kex algos:['diffie-hellman-group-exchange-sha256', 'diffie-hellman-group-exchange-sha1', 'diffie-hellman-group14-sha1', 'diffie-hellman-group1-sha1'] server key:['ssh-rsa', 'ssh-dss'] client encrypt:['aes128-ctr', 'aes192-ctr', 'aes256-ctr', 'arcfour256', 'arcfour128', 'aes128-cbc', '3des-cbc', 'blowfish-cbc', 'cast128-cbc', 'aes192-cbc', 'aes256-cbc', 'arcfour', '[email protected]'] server encrypt:['aes128-ctr', 'aes192-ctr', 'aes256-ctr', 'arcfour256', 'arcfour128', 'aes128-cbc', '3des-cbc', 'blowfish-cbc', 'cast128-cbc', 'aes192-cbc', 'aes256-cbc', 'arcfour', '[email protected]'] client mac:['hmac-md5', 'hmac-sha1', '[email protected]', 'hmac-ripemd160', '[email protected]', 'hmac-sha1-96', 'hmac-md5-96'] server mac:['hmac-md5', 'hmac-sha1', '[email protected]', 'hmac-ripemd160', '[email protected]', 'hmac-sha1-96', 'hmac-md5-96'] client compress:['none', '[email protected]'] server compress:['none', '[email protected]'] client lang:[''] server lang:[''] kex follows?False
DEB [20120328-11:39:29.093] thr=1   ssh.transport: Ciphers agreed: local=aes128-ctr, remote=aes128-ctr
DEB [20120328-11:39:29.093] thr=1   ssh.transport: using kex diffie-hellman-group1-sha1; server key type ssh-rsa; cipher: local aes128-ctr, remote aes128-ctr; mac: local hmac-sha1, remote hmac-sha1; compression: local none, remote none
DEB [20120328-11:39:29.183] thr=1   ssh.transport: Switch to new keys ...
DEB [20120328-11:39:29.224] thr=2   ssh.transport: Trying SSH agent key cda5638f390e166864444b1093b91017
DEB [20120328-11:39:29.272] thr=1   ssh.transport: userauth is OK
INF [20120328-11:39:53.310] thr=1   ssh.transport: Authentication (publickey) failed.
DEB [20120328-11:41:29.076] thr=1   ssh.transport: EOF in transport thread

Hmm, that's strange, I ran the command as: fab diskfree -H xxx.xxx.xxx.xxx -u root

But what is this?

$ cat ./fabfile.py
from fabric.api import *
import ssh
ssh.util.log_to_file("paramiko.log", 10)

env.user = 'bryan'

def host_type():
 run('uname -s')

def diskfree():
 run('df -h') 

Hmm

env.user = 'bryan'

Could that be the root of the problem? Could the ssh error messages just be misleading me?

I removed the line and it worked, so I guess so, is the answer.

Dorthydortmund

you might have a config line in your .ssh/config that you forgot about or figured was being honored, but I like to err on the side of caution and keep in the users there, or in the one of instances encode them into the hosts lists with list comprehensions. Glad this hint let you figure out the issue. – Janiecejanifer

SSH is unmaintained now, they recommend to use paramiko. github.com/bitprophet/ssh – Winograd

Answer 2

So from the paramiko code and yours when you do a.get_keys() that should return a list. I'd see what it returns. And it woudln't return something you can count like that, as it's returning the actual encrypted key bits. But anyhow, as you've moved onto ssh, and that works, let's move to Fabric.

You can get more logging by turning it on for the ssh lib by doing:

import ssh
ssh.util.log_to_file("paramiko.log", 10)

In your fabfile. This'll up all the logs and show more of what paramiko/ssh itself is doing which may assist you in debugging the issue further.

Janiecejanifer

Hey man, good hint. I'm going to put the extra information into the original posting. Turns out it's ignoring the specified user. – Dorthydortmund

Answer 3

I would try to specify the passphrase as the keyword password argument to connect().

As stated in the docs for SSHCLient.connect(), it uses any PKey it can find in the system if not provided with a specific one. The class methods from_private_key() and from_private_key_file() (I'm not sure which one of them is called, probably both) take an optional argument password. The docs say,

If the private key is encrypted and password is not None, the given password will be used to decrypt the key (otherwise PasswordRequiredException is thrown).

...which is probably what's happening in your case.

Mobilize

But ssh-agent is running, and authenticated in my console. I can ssh to the server without providing a password. My question is how to achieve this with paramiko. I don't want to put the password in the code, that would negate the entire purpose of the exercise. – Dorthydortmund

Answer 4

i had this issue and what worked for me what starting the SSH_AGENT:

eval $(ssh-agent)

and adding the SSH_KEY:

ssh-add ~/.ssh/id_rsa

Taishataisho

Answer 5

Perhaps you need to run, to add the key to the agent.

$ ssh-add ~/.ssh/id_dsa 

https://groups.google.com/forum/?fromgroups=#!topic/ansible-project/yRSMmlqKsAA

Urine

i think this is what you meant : ssh-add ~/.ssh/id_rsa – Taishataisho