How to do gsutil cp -R while ignoring files like .git, .gitignore?

Question

I am trying to automate the process of syncing my web assets with Google Cloud Storage. I basically need to copy everything in my development directory up to the cloud. However, I need to ignore the .git directory and some other irrelevant files.

I can't just do a gsutil cp -R . <dest> because that takes absolutely everything, including .git. I tried find . | fgrep git | gsutil cp -I <dest> but that flattens all directories and puts them in root!

Is there a way I can solve this with gsutil or do I have to do a loop in script which uploads all directories (except .git) with -R and then uploads individual files in current directory?

Telugu

Possible duplicate of rsync will not exclude hidden files in gsutil 4.15 – Pentateuch

Answer 1

You have two options:

A) Remove the git files after they are uploaded:

gsutil rm gs://bucket/\*.git\*

B) Use find to exclude git files:

find . -not -path '*/.git' -type f -printf '%P\n' | xargs -I '{}' gsutil cp '{}' gs://bucket/'{}'

Source: https://groups.google.com/forum/#!topic/gsutil-discuss/zoHhkTPhiNc

It would've been much easier if gsutil implemented rsync, this would've been easier with their --exclude flag.

Friary

This answer is outdated, see cloud.google.com/storage/docs/gsutil/commands/rsync – Tow

Answer 2

You could use a command like:

gsutil rsync -x '\.git.*' dev_dir gs://your-bucket

See Google Storage - rsync - Synchronize content of two buckets/directories

Sawfly

This is an appropriate answer. Not sure why it got downvoted. – Gauffer

This has the downside of not being able to use -z js,css,json,html,svg to put these files in GCS compressed with gzip. This makes the storage cost slightly smaller, but the key value of it is that without this the files will be served from the GCS bucket without compression. – Divest

Answer 3

Your web assets are text files - JS, CSS, etc. - which you want to serve compressed, don't you?

Then you need to be aware that GCS requires you to upload such files compressed, to serve them compressed!

Therefore you want to use the -z parameter of gsutil to compress such files.

Expanding on Aziz Saleh's answer then, you probably want to do this:

path '*/.git' -type f -printf '%P\n' | xargs -I '{}' gsutil cp -z js,css,json,html,htm,xml '{}' gs://bucket/'{}'

(You cannot use -z parameter with the Mike Schwartz's gsutil rsync solution.)

Read more about gsutil cp here and about "Transcoding of gzip-compressed files" here.

Divest