Error "No URLs matched" When copying Google cloud bucket data to my local computer?

Question

I am trying to download a folder which is inside my Google Cloud Bucket, I read from google docs gsutil/commands/cp and executed below the line.

gsutil cp -r appengine.googleapis.com gs://my-bucket

But i am getting the error

CommandException: No URLs matched: appengine.googleapis.com

Edit

By running below command

gsutil cp -r gs://logsnotimelimit .

I am getting Error

IOError: [Errno 22] invalid mode ('ab') or filename: u'.\logsnotimelimit\appengine.googleapis.com\nginx.request\2018\03\14\14:00:00_14:59:59_S0.json_.gstmp'

Teammate

Answer 1

What is the appengine.googleapis.com parameter in your command? Is that a local directory on your filesystem you are trying to copy to the cloud bucket?

The gsutil cp -r appengine.googleapis.com gs://my-bucket command you provided will copy a local directory named appengine.googleapis.com recursively to your cloud bucket named my-bucket. If that's not what you are doing - you need to construct your command differently.

I.e. to download a directory named folder from your cloud bucket named my-bucket into the current location try running gsutil cp -r gs://my-bucket/folder .

-- Update: Since it appears that you're using a Windows machine (the "\" directory separators instead of "/" in the error message) and since the filenames contain the ":" character - the cp command will end up failing when creating those files with the error message you're seeing.

Impound

Can you please have a relook at my updated post. Because now i am getting a new error. – Teammate

Are you there ?? – Teammate

The gsutil cp -r gs://logsnotimelimit . command looks fine assuming you have the rights to read from logsnotimelimit and write locally which it seems you do. I just tried running a similar one right now and it worked correctly as expected copying the entire bucket into the current directory. Not sure why it's not working for you, but you may want to make sure you're running the latest gcloud suite via the gcloud components update command, also try adding the -m param to run in multiple threads to see if the issue might be in this one file, i.e. gsutil -m cp -r gs://logsnotimelimit . – Impound

Hi Sudhanshu, I am not expert on Google Storage but is it possible the complex characters (colons, full stops etc) in the file name are causing confusion and giving the "bad filename" error? On the link here there are some guidelines on filenaming. This person had a similar problem and it turned out to be filename related. – Yasukoyataghan

@MihailRussu I first updated my gcloud components and then tried this command gsutil -m cp -r gs://logsnotimelimit . but still i am getting the same error. – Teammate

@Yasukoyataghan thanks for helping me out of here, the naming is default naming used by the app engine. e.g logsnotimelimit/appengine.googleapis.com/stdout/2018/03/14/14:00:00_14:59:59_S0.json – Teammate

@SudhanshuGaur are you on a Windows machine by any chance (or using a Windows filesystem)? Since the directory separator shown in your error message appears to be \ instead of / it's probably safe to assume that it is correct in which case @Yasukoyataghan is likely right as you cannot have files with the : character in their names on a Windows filesystem. Give it a try on a linux/macos machine and it will probably work. – Impound

Yeah it solved the problem, it was because of name of json files were having : in them (y) thanks a lot. You can update your answer so that i can accept it (y). – Teammate

@SudhanshuGaur ok, great! Update posted. – Impound

Answer 2

Just wanted to help people out if they run into this problem on Windows. As administrator:

• Open C:\Program Files (x86)\Google\Cloud SDK\google-cloud-sdk\platform\gsutil\gslib\utils
• Delete copy_helper.pyc
• Change the permissions for copy_helper.py to allow writing
• Open copy_helper.py
• Go to the function _GetDownloadFile
• On line 2312 (at time of writing), change the following line

download_file_name = _GetDownloadTempFileName(dst_url)

to (for example, objective is to remove the colons):

download_file_name = _GetDownloadTempFileName(dst_url).replace(':', '-')

• Go to the function _ValidateAndCompleteDownload
• On line 3184 (at time of writing), change the following line

final_file_name = dst_url.object_name

to (for example, objective is to remove the colons):

final_file_name = dst_url.object_name.replace(':', '-')

• Save the file, and rerun the gsutil command
• FYI, I was using the command gsutil -m cp -r gs://my-bucket/* . to download all my logs, which by default contain : which does not bode well for Windows files!

Hope this helps someone, I know it's a somewhat hacky solution, but seeing as you never need (should have) colons in Windows filenames, it's fine to do and forget. Just remember that if you update the Google SDK you'll have to redo this.

Tunny

Answer 3

This is also gsutil's way of saying file not found. The mention of URL is just confusing in the context of local files.

Superfuse

Txs @Superfuse waster time not correctly understanding error – Interlock

I thought I had to generate a URL for the file in the GCP bucket or something. – Rome

Answer 4

I got same issue and resolved it as below.

1. Open a cloud shell, and copy objects by using gsutil command.

gsutil -m cp -r gs://[some bucket]/[object] .

2. On the shell, zip those objects by using zip command.

zip [some file name].zip -r [some name of your specific folder]

3. On the shell, copy the zip file into GCS by using gsutil command.

gsutil cp [some file name].zip gs://[some bucket] .

4. On a Windows Command Prompt, copy the zip file in GCS by using gsutil command.

gsutil cp gs://[some bucket]/[some file name].zip .

I wish this information helps someone.

Melaniemelanin

Answer 5

Be careful, in this command, the file path is case sensitive. You can check if it is not a capitalized letter issue.

Supine

Answer 6

Adapting eugene 's answer for 2023 for gsutil-5.25 installed on windows with scoop install gcloud

I added to the file <SCOOP>\apps\gcloud\current\platform\gsutil\gslib\utils\copy_helper.cpython-39.pyc the replace line and the log

def PerformCopy(...):
"""Performs copy from src_url to dst_url, handling various special cases.
...
"""
logger.info("copy to dst_url %s",dst_url.object_name)
dst_url.object_name = dst_url.object_name.replace(':', '-')

Reciprocity