How to perform logging with gsutil rsync

Question

What's the proper way to log any errors or warnings when performing a quiet rsync?

This is what I currently run from my crontab:

gsutil -m -q rsync -r -C /mount1/share/folder gs://my-bucket-1/folder/ > /mount2/share/folder/gsutil.log

Since the log file is always completely empty and I'm uploading terabytes of data I'm starting to think that maybe even errors and warnings are being supressed.

Styles

Errors are still reported with -q but they should get sent to stderr instead of stdout. Can you redirect stderr as well? In general though, you shouldn't receive many errors, if any. – Chau

Isn't it just too good to be true not to get any errors? ;) How about command 2>&1 >> gsutil.log. Is that what you mean? It's hard to know if it is really working since there is absolutely nothing being logged to the file. – Styles

If you want progress information, why not remove -q? – Chau

If possible, I only want errors and warnings logged since I'm transferring a large amount of files. The log would get enormous if everything (INFO level) would get logged. – Styles

I simply removed the -q option, and noticed that nothing gets logged to that file I'm piping output to. All INFO level messages are printed in my terminal. I wonder what I'm doing wrong... – Styles

I think you've got your redirect set up wrong. You want command >> gsutil.log 2>&1. – Chau

Ah, yes – you are right about command >> gsutil.log 2>&1. Thank you. Still not getting anything in the log when using -q though... but I guess that's a good thing! :) – Styles

Answer 1

After having realized that this is related to how you pipe stdout and/or stderr to files in general, the answer really lies within this existing thread: How to redirect both stdout and stderr to a file

So a simple solution to log as much as possible into one single log file could be something like:

gsutil -d rsync [src] [dst] &> [logfile]

...where -d enables debug output. I found this to be the only way to show files which were affected by an error such as CommandException: 3 files/objects could not be copied. Please note that -d exposes authentication credentials.

Styles

is this command allows the command to execute even if SSH connection breaks ? – Chrysler

working eg: gsutil -m rsync -r /mnt/disks/stutzen.co/ gs://stutzen.me/sbkp/vm10-vm/ &> /tmp/ry1.txt & – Chrysler

-d is for Delete extra files under dst_url not found under src_url, use -D instead. – Goldeneye

This is debugging. Not logging. It's really just not useful. -d after the rsync is for delete. -d before the rsync is debug. Just confusing and not useful. It should just actually log the relevant/important information. – Gutter