SSH Agent Forwarding with Ansible

Question

I’m using Ansible 1.5.3 and Git with ssh agent forwarding (https://help.github.com/articles/using-ssh-agent-forwarding). I can log into the server that I am managing with Ansible and test that my connection to git is correctly configured:

ubuntu@test:~$ ssh -T [email protected]
Hi gituser! You've successfully authenticated, but GitHub does not provide shell access.

I can also clone and update one of my repos using this account so my git configuration looks good and uses ssh forwarding when I log into my server directly via ssh.

The problem: When I attempt the same test shown above using the Ansible command module. It fails with “Permission denied”. Part of the Ansible output (with verbose logging) looks like this:

failed: [xxx.xxxxx.com] => {"changed": true, "cmd": ["ssh", "-T", "[email protected]"], "delta": "0:00:00.585481", "end": "2014-06-09 14:11:37.410907", "rc": 255, "start": "2014-06-09 14:11:36.825426"}
stderr: Permission denied (publickey).

Here is the simple playbook that runs this command:

- hosts: webservers
  sudo: yes
  remote_user: ubuntu

  tasks:

  - name: Test that git ssh connection is working.
    command: ssh -T [email protected]

The question: why does everything work correctly when I manually log in via ssh and run the command but fail when the same command is run as the same user via Ansible?

I will post the answer shortly if no one else beats me to it. Although I am using git to demonstrate the problem, it could occur with any module that depends on ssh agent forwarding. It is not specific to Ansible but I suspect many will first encounter the problem in this scenario.

Schaumberger

Answer 1

There are some very helpful partial answers here, but after running into this issue a number of times, I think an overview would be helpful.

First, you need to make sure that SSH agent forwarding is enabled when connecting from your client running Ansible to the target machine. Even with transport=smart, SSH agent forwarding may not be automatically enabled, depending on your client's SSH configuration. To ensure that it is, you can update your ~/.ansible.cfg to include this section:

[ssh_connection]
ssh_args=-o ControlMaster=auto -o ControlPersist=60s -o ControlPath=/tmp/ansible-ssh-%h-%p-%r -o ForwardAgent=yes

Next, you'll likely have to deal with the fact that become: yes (and become_user: root) will generally disable agent forwarding because the SSH_AUTH_SOCK environment variable is reset. (I find it shocking that Ansible seems to assume that people will SSH as root, since that makes any useful auditing impossible.) There are a few ways to deal with this. As of Ansible 2.2, the easiest approach is to preserve the (whole) environment when using sudo by specifying the -E flag:

become_flags: "-E"

However, this can have unwanted side-effects by preserving variables like PATH. The cleanest approach is to only preserve SSH_AUTH_SOCK by including it in env_keep in your /etc/sudoers file:

Defaults    env_keep += "SSH_AUTH_SOCK"

To do this with Ansible:

- name: enable SSH forwarding for sudo
  lineinfile:
    dest: /etc/sudoers
    insertafter: '^#?\s*Defaults\s+env_keep\b'
    line: 'Defaults    env_keep += "SSH_AUTH_SOCK"'

This playbook task is a little more conservative than some of the others suggested, since it adds this after any other default env_keep settings (or at the end of the file, if none are found), without changing any existing env_keep settings or assuming SSH_AUTH_SOCK is already present.

Mousey

@AlouaniYounes It continued working for me when I upgraded to Ansible 2.4. – Mousey

@trevor-robison Oh, you just made my day. Adding the ssh_connection setting to my ansible.cfg works really well, so thank you! :) – Bladdernut

Sudo also allows keeping only certain environment variables through flags. Therefore -E in the become_flags ansible setting can be replaced with the more specific --preserve-env=SSH_AUTH_SOCK. This helps reduce side-effects from keeping the whole environment. – Gallous

Can you explain why each part is needed for the bit you added to ansible.cfg? – Hollander

Don't forget the common /etc/sudoers.dextension, which is somewhat superior to hacking the distro-provided /etc/sudoers file. – Nelidanelie

There's a problem here. Still doesn't work because SSH_AUTH_SOCK points to a socket that's most likely not accessible to the sudo'd user. It'd be more sensible to simply ssh to the destination as that user in the first place. – Nelidanelie

Answer 2

The problem is resolved by removing this line from the playbook:

sudo: yes

When sudo is run on the remote host, the environment variables set by ssh during login are no longer available. In particular, SSH_AUTH_SOCK, which "identifies the path of a UNIX-domain socket used to communicate with the agent" is no longer visible so ssh agent forwarding does not work.

Avoiding sudo when you don't need it is one way to work around the problem. Another way is to ensure that SSH_AUTH_SOCK sticks around during your sudo session by creating a sudoers file:

/etc/sudoers:

     Defaults    env_keep += "SSH_AUTH_SOCK"

Schaumberger

Setting sudo: no explicitly may be required. – Fidellas

If you go with SSH_AUTH_SOCK at least be aware of the security risk serverfault.com/a/371788/67801 – Bundestag

@SystematicFrank: The security risk in your link is for the chmod 777, not the SSH_AUTH_SOCK – Tipster

This worked for me. I automated it with: - name: enable SSH creds passthrough lineinfile: dest=/etc/sudoers regexp='^#?Defaults\Wenv_keep[^$]' line='Defaults env_keep += "SSH_AUTH_SOCK"' – Misti

Answer 3

Another answer to your question (with the exception that I am using Ansible 1.9) could be the following:

You may want to check your /etc/ansible/ansible.cfg (or the other three potential locations where config settings can be overridden) for transport=smart as recommended in the ansible docs. Mine had defaulted to transport=paramiko at some point during a previous install attempt, preventing my control machine from utilizing OpenSSH, and thus agent forwarding. This is probably a massive edge case, but who knows? It could be you!

Though I didn't find it necessary for my configuration, I should note that others have mentioned that you should add -o ForwardAgent=yes to your ssh_args setting in the same file like so:

[ssh_connection]
ssh_args=-o ForwardAgent=yes

I only mention it here for the sake of completeness.

Pram

I had to add the ssh_args to my ansible.cfg to get SSH Agent forwarding working at all. Kudos to you for being the only person to suggest it! I'm on OSX 10.10, running Ansible 1.9.1 – Anthracnose

If you specify ssh_args that will prevent ansible from supplying its own, including the args that setup persistent connections. This can cause a huge slowdown. The relevant arguments that would be added are -o ControlMaster=auto -o ControlPersist=60s -o ControlPath=/tmp/ansible-ssh-%h-%p-%r. – Degeneracy

Sometimes there is also problem with sudo_flags which are set to sudo_flags = -H -S -n by default and it can fail with "sudo: a password is required". In this case, you need to set sudo_flags = -H -S removing -n option. – Enceladus

I unterstand that using ssh_extra_args=-o ForwardAgent=yes prevents the slowdown (mentioned by qqx). – Cate

Answer 4

To expand on @j.freckle's answer, the ansible way to change sudoers file is:

- name: Add ssh agent line to sudoers
  lineinfile: 
    dest: /etc/sudoers
    state: present
    regexp: SSH_AUTH_SOCK
    line: Defaults env_keep += "SSH_AUTH_SOCK"

Insociable

But won't we need to run ansible twice? One to setup the ssh fwding and the other to actually get the connection with this env var set? – Directions

That sounds right. I always end up running and commenting out playbooks multiple times, ssh into boxes and check things manually. It's never been smooth sailing. – Insociable

To try to avoid having to run multiple times, I ended up putting this step in a script I run in a fresh server install. After all there are things to setup before I can setup my server with ansible... – Directions