How do I copy a directory to a remote machine using Fabric?

Question

I have a directory on my local machine that I would like to copy to a remote machine (and rename it) using Fabric. I know I can copy file using put(), but what about a directory. I know it's easy enough using scp, but I would prefer to do it from within my fabfile.py if possible.

Demott

Answer 1

You can use put for that as well (at least in 1.0.0):

local_path may be a relative or absolute local file or directory path, and may contain shell-style wildcards, as understood by the Python glob module. Tilde expansion (as implemented by os.path.expanduser) is also performed.

See: http://docs.fabfile.org/en/1.0.0/api/core/operations.html#fabric.operations.put

---

Update: This example works fine (for me) on 1.0.0.:

from fabric.api import env
from fabric.operations import run, put

env.hosts = ['[email protected]']

def copy():
    # make sure the directory is there!
    run('mkdir -p /home/frodo/tmp')

    # our local 'testdirectory' - it may contain files or subdirectories ...
    put('testdirectory', '/home/frodo/tmp')

# [[email protected]] Executing task 'copy'
# [[email protected]] run: mkdir -p /home/frodo/tmp
# [[email protected]] put: testdirectory/HELLO -> \
#     /home/frodo/tmp/testdirectory/HELLO
# [[email protected]] put: testdirectory/WORLD -> \
#     /home/frodo/tmp/testdirectory/WORLD
# ...

Berglund

Thanks. I'm getting an exception (Is a directory) any chance of an example? – Demott

@gaviscon_man: Added a (tested) example, but really it's just vanilla fab, no tricks. You'll get errors, if the target directories aren't in place already - so I included a simple mkdir -p before the put. (But other subdirectories, which are below the testdirectory will automatically created on the remote machine). – Berglund

put is working. Will it support copying of folder with compress at source machine and decompress at remote machine. – Absurdity

Answer 2

I would also look at the Project Tools module: fabric.contrib.project Documentation

This has an upload_project function which takes a source and target directory. Even better, there is an rsync_project function that uses rsync. This is nice because it only updates the files that have changed and it accepts extra args like "exclude" which is nice for doing things like excluding your .git directory.

For example:

from fabric.contrib.project import rsync_project

def _deploy_ec2(loc):

    rsync_project(local_dir=loc, remote_dir='/var/www', exclude='.git')

Soundless

fabric.contrib.project docs for latest version: docs.fabfile.org/en/latest/api/contrib/project.html – Underfeed

way better than put/get. also works perfectly for fetching user uploads from live websites, for example (upload=False, it's not obvious that it works in both ways). – Economizer

i had to wrap the excluded dir in a list to get this to work: exclude=['.git'] – Strained

This has been transfered to the patchwork library now: fabric-patchwork.readthedocs.io/en/latest/api/transfers.html See the other answer for details: https://mcmap.net/q/122011/-how-do-i-copy-a-directory-to-a-remote-machine-using-fabric – Noway

Answer 3

For those using Fabric 2, put can no longer upload directories, only files. Also, rsync_project is no longer part of the main Fabric package. The contrib package has been removed, as explained here. Now, rsync_project has been renamed to rsync, and you need to install another package in order to be able to use it:

pip install patchwork

Now, assuming you already have created a connection to your server:

cxn = fabric.Connection('username@server:22')

You can use rsync as below:

import patchwork.transfers
patchwork.transfers.rsync(cxn, '/my/local/dir', target, exclude='.git')

Please refer to the fabric-patchwork documentation for more information.

Dolorous

is there a way to provide automatic password to rsync? – Isotron

@Isotron Yes, using connect_kwargs. For example: cxn = fabric.Connection('username@server:22', connect_kwargs=dict(password='yourpass')) – Dolorous

@TGO Can you share any example on using put option to transfer file? – Millrace

@Millrace put can not upload in Fabric 2. If you're using Fabric 1, then please refer to the accepted answer. With Fabric 2 I use the example provided in the answer, using rsync. – Dolorous