Rsync to Google Compute engine Instance from Jenkins

Question

I want to send files from Jenkins to to my instance in Google Compute engine instance I added a build in my config in jenkins :

rsync -vrzhe "ssh -i /var/lib/jenkins/.ssh/google_compute_engine -o UserKnownHostsFile=/dev/null -o CheckHostIP=no -o StrictHostKeyChecking=no" . login@Host:/var/www

And I get this error :

Checking out Revision 59cf9dd819fe2168c4c40f716707d58b2b99e251 (refs/remotes/origin/master)
> git config core.sparsecheckout # timeout=10
> git checkout -f 59cf9dd819fe2168c4c40f716707d58b2b99e251
> git rev-list 59cf9dd819fe2168c4c40f716707d58b2b99e251 # timeout=10
[Platform] $ /bin/sh -xe /tmp/hudson4502433356962914860.sh
+ rsync -vrzhe 'ssh -i /var/lib/jenkins/.ssh -o UserKnownHostsFile=/dev/null -o 

---

CheckHostIP=no -o StrictHostKeyChecking=no' . login@Host:/var/www
   StrictHostKeyChecking=no' . login@host:/var/www
   ssh: connect to host host port 22: Connection timed out
   rsync: connection unexpectedly closed (0 bytes received so far) [sender]
   rsync error: unexplained error (code 255) at io.c(601) [sender=3.0.7]
   Build step 'Exécuter un script shell' marked build as failure
   Finished: FAILURE

Any idea

Imprest

Answer 1

The problem was with my public key so to solve the problem you need to :

1 - Setting up your ssh keys : run

gcloud compute ssh example-instance

2 - cp your .ssh/google_compute_engine.pub content into the GCE VM authorized_key

3 - restart VM instance

Thanks for your help

Imprest

At least you didn't have any incompatibility between rsync and ssh, as I feared initially. +1 – Westberg

Answer 2

first configure .ssh config file by (assuming you installed gcloud sdk):

gcloud compute config-ssh

It will tell you that "now you can use ssh/scp with your instances by running

ssh your-instance

your-instance is in the form of "instance.zone.project", usually.

Now you can rsync:

rsync -ave ssh your-local-dir your-instance:~/your-destination

Done.

You can mention the user if you like so:

rsync -ave ssh your-local-dir your-user@your-instance:~/your-destination

It worked for me. Juts do not forget to replace "your-instance" (and your-user) with the correct one. You can get it through "gcloud compute config-ssh" or "gcloud compute config-ssh --dry-run" or go to your cloud.google.com then compute engine then vm instances then from connect choose view gcloud command. All will show your instance name in the form of "instance.zone.project."

I hope it will help someone in future. :)

Say

You could mention about the -z flag too which does speed up things by enabling compression in most workflows. – Beshrew

Do not forget to install rsync on the server! sudo apt-get install rsync. Otherwise it won't work. If you use the default debian image it's not installed. – Redhanded

Answer 3

Maybe a little late, but you can use the gcloud compute ssh command directly instead of finding the google ssh keys.

First, you have to make a script that will hide the rsync ssh command args from gcloud:

cat >./gcloud-compute-ssh <<EOF
#! /bin/sh
host="$1"
shift
exec gcloud compute ssh "$host" -- "$@"
EOF

chmod a+x ./gcloud-compute-ssh

Then you can rsync -e to your heart's content:

rsync -e ./gcloud-compute-ssh my-dir my-instance:/my-dir

Carbone

Have you tried it by yourself? It does not work for me. – Say

Of course! Can you describe what didn't work, perhaps with a terminal log? – Carbone

Sorry, after a little twik in my .ssh file, it started working. Thanks. – Say

Answer 4

Your rsync command above uses the -i option for ssh. The argument to -i option should be path to ssh key file, not the directory where the key file is.

Motte

Yes of course -i should be the path for the file i will edit my post but that doesn't solve my problem – Imprest

Answer 5

As mentioned in "How to solve rsync error: error in rsync protocol data stream (code 12) at io.c(600) on windows", check which ssh get to be actually used, and if that ssh version is compatible with with the one from rsync.

Specifying an absolute path for the ssh executable can help:

rsync -e /usr/bin/ssh .... 

This thread also suggests to purge ~/.ssh/know_hosts in case the server has changed and now has a different key.

Westberg

@RjaibiMejdi What OS are you on?, what version of rsync and ssh are you using? – Westberg

Ubuntu server to Ubuntu server / OpenSSH_5.5p1 / rsync version 3.0.7 protocol version 30 – Imprest

I mean what exact precise complete version number? What does uname -a return? What rsync -v return? – Westberg

Linux ns61740.ovh.net 3.2.13-grsec-xxxx-grs-ipv6-64 #1 SMP Thu Mar 29 09:48:59 UTC 2012 x86_64 GNU/Linux – Imprest

Ok, the thread I mentioned in the answer seemed to suggest a potential incompatibility between certain versions of rsync and ssh, hence my previous question. – Westberg

Answer 6

In 2024, you can follow step:

gcloud compute config-ssh

This cli will generate config in ~/.ssh/config, and return something like below:

You should now be able to use ssh/scp with your instances.
For example, try running:

 $ ssh [instanceName].[zone].[projectName]

And then you can use this host to use with rsync:

rsync -avzP ~/local/file [instanceName].[zone].[projectName]:/host/file

Quixotic