Handle gsutil ls and rm command errors if no files present

Question

I am running the following command to remove files from a gcs bucket prior to loading new files there.

gsutil -m rm gs://mybucket/subbucket/*

If there are no files in the bucket, it throws the "CommandException: One or more URLs matched no objects".

I would like for it to delete the files if exists without throwing the error.

There is same error with gsutil ls gs://mybucket/subbucket/*

How can I rewrite this without having to handle the exception explicitly? Or, how to best handle these exceptions in batch script?

Lifton

Answer 1

Try this:

gsutil -m rm gs://mybucket/foo/* 2> /dev/null || true

Or:

gsutil -m ls gs://mybucket/foo/* 2> /dev/null || true

This has the effect of suppressing stderr (it's directed to /dev/null), and returning a success error code even on failure.

Wheelhorse

Thanks. I did change it to following in windows batch script. gsutil -m rm gs://mybucket/foo/* 2>nul || true – Lifton

Answer 2

You might not want to ignore all errors as it might indicate something different that file not found. With the following script you'll ignore only the 'One or more URLs matched not objects' but will inform you of a different error. And if there is no error it will just delete the file:

gsutil -m rm gs://mybucket/subbucket/* 2> temp
if [ $? == 1 ]; then
    grep 'One or more URLs matched no objects' temp
    if [ $? == 0 ]; then
        echo "no such file"
    else
        echo temp
    fi
fi
rm temp

This will pipe stderr to a temp file and will check the message to decide whether to ignore it or show it.

And it also works for single file deletions. I hope it helps.

Refs:

How to grep standard error stream
Bash Reference Manual - Redirections

Jarman

The error changed to "CommandException: 1 files/objects could not be removed." – Phia

Answer 3

You may like rsync to sync files and folders to a bucket. I used this for clearing a folder in a bucket and replacing it with new files from my build script.

gsutil rsync -d newdata gs://mybucket/data - replaces data folder with newdata

Ting

gsutil rsync -d -r newdata gs://mybucket/data might be what you want in order to sync recursively – Saucedo