Capistrano Deploy Failing on git:check - Permission denied (publickey)

Asked 6/1, 2018 at 4:51 Answered 27/1, 2021 at 9:42

Solved ruby-on-rails git capistrano3 ssh-agent

Local OS: Windows 10 Pro (using Git Bash as my terminal)

Staging Server OS: Ubuntu 16.04 LTS

I've been struggling to get a basic Capistrano deploy working on my staging server. I followed this guide to set up Capistrano.

The deployment process always fails on the git:check stage due to allegedly not having permission to access my repo on GitLab. I am sure my SSH Agent forwarding is working because I am able to SSH into my server and access GitLab with my SSH key. The SSH key is not stored anywhere on my server:

$ ssh [email protected]

deploy@MyServer:~$ ssh -T [email protected]
debug1: client_input_channel_open: ctype [email protected] rchan 2 win 65536 max 16384
debug1: channel 1: new [authentication agent connection]
debug1: confirm [email protected]
Welcome to GitLab, Alexander!debug1: channel 1: FORCE input drain

None of the questions about this issue have been working for me so far.

Here are my deploy files:

deploy.rb

set :application, "myapp"
set :branch, "master"
set :repo_url, "[email protected]:MyApp/myapp.git"

# Defaults to false
# Skip migration if files in db/migrate were not modified
set :conditionally_migrate, true

set :ssh_options, { forward_agent: true }

set :rvm_ruby_version, '2.2.6'

# Default deploy_to directory is /var/www/my_app_name
set :deploy_to, "/var/www/#{fetch(:application)}/"
set :deploy_user, "deploy"

# Tells Capistrano to store config/database.yml file inside a directory called /shared, which is meant for any files
# we want to persist between deploys
set :linked_files, fetch(:linked_files, []).push('config/database.yml', 'config/secrets.yml')

# Directories that are meant to persist between deploys, and they will also be stored inside /shared
set :linked_dirs, fetch(:linked_dirs, []).push('bin', 'log', 'tmp/pids', 'tmp/cache', 'tmp/sockets', 'vendor/bundle', 'public/system', 'public/uploads')

# The specs that should be run before deployment is allowed to continue
set :tests, []

# Delayed Job Config: https://github.com/AgileConsultingLLC/capistrano3-delayed-job
set :delayed_job_workers, 3

# Keep the last 5 deploys for rollback purposes
set :keep_releases, 5

namespace :deploy do
  after :restart, :clear_cache do
    on roles(:web), in: :groups, limit: 3, wait: 10 do
      # Here we can do anything such as:
      # within release_path do
      #   execute :rake, 'cache:clear'
      # end
    end
  end
end

staging.rb

set :stage, :staging
set :rails_env, :staging

role :app, %w{[email protected]}
role :web, %w{[email protected]}
role :db,  %w{[email protected]}

And here is the Capistrano debug log when I deploy:

$ bundle exec cap staging deploy --trace
** Invoke staging (first_time)
** Execute staging
** Invoke load:defaults (first_time)
** Execute load:defaults
** Invoke rvm:hook (first_time)
** Invoke passenger:rvm:hook (first_time)
** Invoke passenger:test_which_passenger (first_time)
** Execute passenger:test_which_passenger
** Execute passenger:rvm:hook
** Execute rvm:hook
** Invoke rvm:check (first_time)
** Execute rvm:check
rvm 1.29.3 (latest) by Michal Papis, Piotr Kuczynski, Wayne E. Seguin [https://rvm.io]
ruby-2.2.6
ruby 2.2.6p396 (2016-11-15 revision 56800) [x86_64-linux]
** Invoke bundler:map_bins (first_time)
** Invoke passenger:bundler:hook (first_time)
** Execute passenger:bundler:hook
** Execute bundler:map_bins
** Invoke deploy:set_rails_env (first_time)
** Execute deploy:set_rails_env
** Invoke deploy:set_linked_dirs (first_time)
** Execute deploy:set_linked_dirs
** Invoke deploy:set_rails_env
** Invoke deploy (first_time)
** Execute deploy
** Invoke deploy:starting (first_time)
** Execute deploy:starting
** Invoke deploy:check (first_time)
** Invoke git:check (first_time)
** Invoke git:wrapper (first_time)
** Execute git:wrapper
00:00 git:wrapper
      01 mkdir -p /tmp
    ✔ 01 [email protected] 0.286s
      Uploading /tmp/git-ssh-myapp-staging-localuser.sh 100.0%
      02 chmod 700 /tmp/git-ssh-myapp-staging-localuser.sh
    ✔ 02 [email protected] 0.277s
** Execute git:check
00:01 git:check
      01 git ls-remote [email protected]:MyApp/myapp.git HEAD
      01 Permission denied (publickey).
      01 fatal: Could not read from remote repository.
      01
      01 Please make sure you have the correct access rights
      01 and the repository exists.
cap aborted!
SSHKit::Runner::ExecuteError: Exception while executing as [email protected]: git exit status: 128
git stdout: Nothing written
git stderr: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
C:/Ruby22/lib/ruby/gems/2.2.0/gems/sshkit-1.15.1/lib/sshkit/runners/parallel.rb:15:in `rescue in block (2 levels) in execute'
C:/Ruby22/lib/ruby/gems/2.2.0/gems/sshkit-1.15.1/lib/sshkit/runners/parallel.rb:11:in `block (2 levels) in execute'

Caused by:
SSHKit::Command::Failed: git exit status: 128
git stdout: Nothing written
git stderr: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
C:/Ruby22/lib/ruby/gems/2.2.0/gems/sshkit-1.15.1/lib/sshkit/command.rb:99:in `exit_status='
C:/Ruby22/lib/ruby/gems/2.2.0/gems/sshkit-1.15.1/lib/sshkit/backends/netssh.rb:169:in `execute_command'
C:/Ruby22/lib/ruby/gems/2.2.0/gems/sshkit-1.15.1/lib/sshkit/backends/abstract.rb:141:in `block in create_command_and_execute'
C:/Ruby22/lib/ruby/gems/2.2.0/gems/sshkit-1.15.1/lib/sshkit/backends/abstract.rb:141:in `tap'
C:/Ruby22/lib/ruby/gems/2.2.0/gems/sshkit-1.15.1/lib/sshkit/backends/abstract.rb:141:in `create_command_and_execute'
C:/Ruby22/lib/ruby/gems/2.2.0/gems/sshkit-1.15.1/lib/sshkit/backends/abstract.rb:74:in `execute'
C:/Ruby22/lib/ruby/gems/2.2.0/gems/capistrano-3.10.1/lib/capistrano/scm/git.rb:77:in `git'
C:/Ruby22/lib/ruby/gems/2.2.0/gems/capistrano-3.10.1/lib/capistrano/scm/git.rb:38:in `check_repo_is_reachable'
C:/Ruby22/lib/ruby/gems/2.2.0/gems/capistrano-3.10.1/lib/capistrano/scm/tasks/git.rake:19:in `block (4 levels) in eval_rakefile'
C:/Ruby22/lib/ruby/gems/2.2.0/gems/sshkit-1.15.1/lib/sshkit/backends/abstract.rb:93:in `with'
C:/Ruby22/lib/ruby/gems/2.2.0/gems/capistrano-3.10.1/lib/capistrano/scm/tasks/git.rake:18:in `block (3 levels) in eval_rakefile'
C:/Ruby22/lib/ruby/gems/2.2.0/gems/sshkit-1.15.1/lib/sshkit/backends/abstract.rb:29:in `instance_exec'
C:/Ruby22/lib/ruby/gems/2.2.0/gems/sshkit-1.15.1/lib/sshkit/backends/abstract.rb:29:in `run'
C:/Ruby22/lib/ruby/gems/2.2.0/gems/sshkit-1.15.1/lib/sshkit/runners/parallel.rb:12:in `block (2 levels) in execute'
Tasks: TOP => deploy:check => git:check
The deploy has failed with an error: Exception while executing as [email protected]: git exit status: 128
git stdout: Nothing written
git stderr: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
** Invoke deploy:failed (first_time)
** Execute deploy:failed


** DEPLOY FAILED
** Refer to log/capistrano.log for details. Here are the last 20 lines:


 DEBUG [6b6ba2d0] Finished in 0.471 seconds with exit status 0 (successful).

 DEBUG [c6e2d7dc] Running ~/.rvm/bin/rvm 2.2.6 do ruby --version as [email protected]

 DEBUG [c6e2d7dc] Command: ~/.rvm/bin/rvm 2.2.6 do ruby --version

 DEBUG [c6e2d7dc]       ruby 2.2.6p396 (2016-11-15 revision 56800) [x86_64-linux]

 DEBUG [c6e2d7dc] Finished in 0.608 seconds with exit status 0 (successful).

  INFO [fd5500a8] Running /usr/bin/env mkdir -p /tmp as [email protected]

 DEBUG [fd5500a8] Command: /usr/bin/env mkdir -p /tmp

  INFO [fd5500a8] Finished in 0.286 seconds with exit status 0 (successful).

 DEBUG Uploading /tmp/git-ssh-myapp-staging-localuser.sh 0.0%

  INFO Uploading /tmp/git-ssh-myapp-staging-localuser.sh 100.0%

  INFO [f33d4873] Running /usr/bin/env chmod 700 /tmp/git-ssh-myapp-staging-localuser.sh as [email protected]

 DEBUG [f33d4873] Command: /usr/bin/env chmod 700 /tmp/git-ssh-myapp-staging-localuser.sh

  INFO [f33d4873] Finished in 0.277 seconds with exit status 0 (successful).

  INFO [86d3cd5a] Running /usr/bin/env git ls-remote [email protected]:MyApp/myapp.git HEAD as [email protected]

 DEBUG [86d3cd5a] Command: ( export GIT_ASKPASS="/bin/echo" GIT_SSH="/tmp/git-ssh-myapp-staging-localuser.sh" ; /usr/bin/env git ls-remote [email protected]:MyApp/myapp.git HEAD )

 DEBUG [86d3cd5a]       Permission denied (publickey).

 DEBUG [86d3cd5a]       fatal: Could not read from remote repository.



Please make sure you have the correct access rights

and the repository exists.

I've tried manually running the commands that Capistrano attempts both locally and on my remote server, and they all succeed. I'm really stuck, and any help would be hugely appreciated!

Moye answered 6/1, 2018 at 4:51 Comment(8)

What are you mean by say The SSH key is not stored anywhere on my server ? – Liu 6/1, 2018 at 5:5

@Зелёный I mean that I don't have the SSH key anywhere on my server; it is only possible for me to connect to GitLab successfully on my remote server because I am using SSH Agent forwarding. – Moye 6/1, 2018 at 5:6

have you tried to setup it in the ~/.ssh/config on the server ? – Liu 6/1, 2018 at 5:9

@Зелёный I followed this guide from GitHub on setting up SSH Agent forwarding, and I'm not sure what I need to set up under ~/.ssh/config on the server. Can you please clarify? – Moye 6/1, 2018 at 5:18

This might be silly, but could you try it without set :ssh_options, { forward_agent: true }? – Blanton 6/1, 2018 at 13:3

@Blanton Thanks for the suggestion. I've tried it without ssh_options, and it still doesn't work unfortunately. – Moye 6/1, 2018 at 18:55

It looks like Capistrano is behaving differently than normal SSH for some reason. You have successfully gotten SSH Agent working for normal ssh. I wonder whether the context in which you are running Capistrano doesn't have the keys loaded. ssh-add? – Blanton 6/1, 2018 at 20:36

ssh-add -l seems to suggest that the keys are loaded both locally and when I SSH into the server. I believe the as in this exception shows the user it is attempting to run the commands under, but I could be wrong: SSHKit::Runner::ExecuteError: Exception while executing as [email protected]: git exit status: 128 – Moye 6/1, 2018 at 21:52

Thanks to everyone who answered, I've managed to find a solution! The main culprit was Git Bash, which, for whatever reason, was not changing the permissions on my ~/.ssh directory to 0700 when I ran chmod 700 ~/.ssh. This prevented SSH agent forwarding from working when Capistrano was deploying but not when I manually SSH'd into my server. I decided to try using Bash on Ubuntu on Windows (BUW) instead of Git Bash, and sure enough, my deploy worked! I copied over the same exact configuration and keys from Git Bash over to BUW. The only difference is that I was able to change the permissions on BUW's ~/.ssh directory to 0700. With that said, here is the solution to my problem:

1. Create a deploy key and add it to GitLab

As @Onur and @grizzthedj, and @Gokul M indicated, I needed to create a deploy key for GitLab and authorize it on my server. Here's how I did that:

Generate a new SSH key on my local machine: ssh-keygen -t rsa -b 4096
Copy the output of the public key: cat ~/.ssh/id_rsa.pub
SSH into my server
Add the public key to the end of ~/.ssh/authorized_keys
Back on my local machine, open up a browser, log into GitLab, go to my repository page, and paste the public key in Settings > Repository > Deploy Keys

2. Use BUW instead of Git Bash

On my local machine, I set up BUW to start the SSH agent on session load using the instructions from this SO answer.
Change the permissions on ~/.ssh: chmod 700 ~/.ssh
Start the SSH agent and add my deploy key to it:
```
eval $(ssh-agent -s)
ssh-add ~/.ssh/id_rsa
```

I've removed the set :ssh_options line from my deploy.rb file because it works just fine without it.

And that's it! It looks like I'll be deploying using BUW from now on.

Moye answered 11/1, 2018 at 6:39 Comment(1)

Official documentation: capistranorb.com/documentation/getting-started/… – Dwelling 13/2, 2020 at 11:14

Add your public key to the list of deploy keys in your repository setting by following the below steps:

Local machine setup:

Check if your local system has ~/.ssh/id_rsa.pub key file. If not, create a new one:
```
$ ssh-keygen -t rsa
```
Add the newly created public key ~/.ssh/id_rsa.pub to the repository's deployment(access) keys in settings:
- Bitbucket :
  - https://confluence.atlassian.com/bitbucket/set-up-ssh-for-git-728138079.html#SetupSSHforGit-Step4.InstallthepublickeyonyourBitbucketaccount
- Github :
  - https://developer.github.com/v3/guides/managing-deploy-keys/#deploy-keys
- GitLab :
  - https://docs.gitlab.com/ce/ssh/README.html#deploy-keys
Load the keys to ssh-agent:

Check whether ssh-agent is running, If not, start the ssh agent
```
$ ssh-agent /bin/bash
```
Add the id_rsa key to the agent:
```
$ ssh-add ~/.ssh/id_rsa
```
Note: Sometimes, this step(Step - 3) needs to be done before each deployment if you receive "Access denied to the repository" error during deployment.
Add your local SSH Key to deployment server Authorized Keys file (remember to replace the port number with your customized port number):
```
$ cat ~/.ssh/id_rsa.pub | ssh -p port_num user@server_ip 'cat >> ~/.ssh/authorized_keys'
```

Ref: https://www.digitalocean.com/community/tutorials/deploying-a-rails-app-on-ubuntu-14-04-with-capistrano-nginx-and-puma

deploy.rb

Try to change the ssh_options in deploy.rb file as below:

set :ssh_options, { forward_agent: true, user: "deploy", auth_methods: ['publickey'], keys: %w(~/.ssh/privatekey.pem) }

Replace ~/.ssh/privatekey.pem with the path to your SSH private key file path.

Deploy:

Just run cap staging deploy to deploy to the server.

Replace staging in cap deploy command with the environment as needed.

Chafee answered 6/1, 2018 at 13:10 Comment(4)

Thanks for your answer. I've used these options, and I'm receiving the same error message. Should the value of the user option be the name of the user on my server or the username on GitLab? Neither is working. – Moye 6/1, 2018 at 19:4

It should be name of the user in your server. – Chafee 7/1, 2018 at 5:58

@Moye I have added some more detailed steps please check whether it solves. Try adding the public key to repository's deployment(access) keys. – Chafee 9/1, 2018 at 4:14

I think you missed the Step - 2. – Chafee 9/1, 2018 at 4:41

1. Create a deploy key and add it to GitLab

As @Onur and @grizzthedj, and @Gokul M indicated, I needed to create a deploy key for GitLab and authorize it on my server. Here's how I did that:

Generate a new SSH key on my local machine: ssh-keygen -t rsa -b 4096
Copy the output of the public key: cat ~/.ssh/id_rsa.pub
SSH into my server
Add the public key to the end of ~/.ssh/authorized_keys
Back on my local machine, open up a browser, log into GitLab, go to my repository page, and paste the public key in Settings > Repository > Deploy Keys

2. Use BUW instead of Git Bash

On my local machine, I set up BUW to start the SSH agent on session load using the instructions from this SO answer.
Change the permissions on ~/.ssh: chmod 700 ~/.ssh
Start the SSH agent and add my deploy key to it:
```
eval $(ssh-agent -s)
ssh-add ~/.ssh/id_rsa
```

I've removed the set :ssh_options line from my deploy.rb file because it works just fine without it.

And that's it! It looks like I'll be deploying using BUW from now on.

Moye answered 11/1, 2018 at 6:39 Comment(1)

Official documentation: capistranorb.com/documentation/getting-started/… – Dwelling 13/2, 2020 at 11:14

Based on your cap deploy output, it looks like SSH connectivity from your laptop to your server is not the issue.

00:00 git:wrapper
      01 mkdir -p /tmp
    ✔ 01 [email protected] 0.286s
      Uploading /tmp/git-ssh-myapp-staging-localuser.sh 100.0%
      02 chmod 700 /tmp/git-ssh-myapp-staging-localuser.sh
    ✔ 02 [email protected] 0.277s

This issue is when capistrano tries to run git ls-remote [email protected]:MyApp/myapp.git HEAD over SSH.

You need to add the public key to your SSH Keys in GitLab, since SSH agent forwarding requires installation of public keys on all target systems. Gitlab, in your case, is a target system.

cat ~/.ssh/id_rsa.pub    # Copy the contents of your public key(filename may be different)

You also need to specify the user that you created the SSH key with in ssh_options.

set :ssh_options, {
    forward_agent: true,
    user: 'deploy',
}

Colombia answered 8/1, 2018 at 19:36 Comment(9)

Thanks for your answer. With SSH Agent forwarding, if I already have the key working on my local environment, shouldn't the agent have access to the keys once I SSH into the server? I've put the public key in my authorized_keys file on the server. – Moye 9/1, 2018 at 3:48

SSH connectivity between your local env and server is not the issue. It is connectivity between your server and GitLab that is the problem. I have update my answer(see EDIT part) – Colombia 9/1, 2018 at 14:52

Where are you running capistrano from when you deploy to staging? From your local workstation, or is it run on another server? – Colombia 10/1, 2018 at 17:10

I am running cap staging deploy from my local workstation. – Moye 10/1, 2018 at 17:13

If you SSH into your staging server and run git ls-remote [email protected]:MyApp/myapp.git HEAD as deploy user, does it succeed? – Colombia 10/1, 2018 at 17:18

Yes, it does indeed succeed. – Moye 10/1, 2018 at 17:24

Is ssh-agent running on your staging server? Also, have you checked the output of sshd_config on your staging server to ensure that AllowAgentForwarding is allowed? – Colombia 10/1, 2018 at 17:30

Yes, it is running on my staging server after I SSH into it. I've set AllowAgentForwarding yes in my sshd_config. – Moye 10/1, 2018 at 18:38

Did you add the public key to Gitlab? SSH agent forwarding does require installation of public keys on all target systems. Gitlab, in this case, is a target system. – Colombia 10/1, 2018 at 18:53

You need to define the public keys of the remote servers to the repository as well. Apparently your computer can access the repo but the servers can't.

To get the default public key, you can look into ~/.ssh/id_rsa.pubor you can generate one using ssh-keygen -t rsa -b 4096 -C "[email protected]" Don't forget to initialize the public key with using ssh-add in servers too.

Hope it helps

Embryotomy answered 10/1, 2018 at 19:17 Comment(0)

For those in the future with this issue looking for an alternative:

Most searches regarding this issue recommend adding SSH keys. If you are sure that they are already added, and ssh -T [email protected] (or whichever git service you're using) is showing you as logged in, and agent forwarding is set up, an alternative that has worked for me on Windows when having the same problem is to install PuTTY. After installing:

Open PuTTYGen > Conversions > Import Key (if you have a key you want to use; otherwise, generate a new one)
Run Pageant (included with PuTTY)
Find Pageant in the system tray, right-click, and "Add key"
Add the aforementioned key
Run cap deploy as before

I found this method out in a different SO article regarding response length errors when using Capistrano on Windows, so ostensibly it works in those situations, also. This is probably because net-ssh on Windows requires Pageant to work.

Daliadalila answered 19/9, 2018 at 20:47 Comment(0)

I had a similar issue, for me all I had to do was to add my SSH key to GitHub. The documentation for this process can be found here: https://docs.github.com/en/github/authenticating-to-github/adding-a-new-ssh-key-to-your-github-account

Augmenter answered 27/1, 2021 at 9:42 Comment(0)

Hot tags

Godot Unity Godot Help Programming Godot 4.X GUI GDScript 3D 2D Physics CSharp Godot 3.X VR XR Projects C++

Recommended topics

Hot tags