`gsutil cp` command throwing "OSError: The filename, directory name, or volume label syntax is incorrect"

Question

I've followed the gsutil install guide following the Google Cloud instructions and I've updated GCloud components to the latest versions. I've just recently installed and configured GCloud to work with my credentials and project. First, I've used the following command to export a single collection I need from my Firestore :

gcloud firestore export gs://my-project-id.appspot.com --collection-ids=theCollectionINeed

Now I see this export is in my Firebase Console Storage section as a folder. Being the project Owner, I'd like to get this export into my local system. For that, I see I need to use gsutil to be able to copy it. Reading the instructions to download the object from your bucket, I've tried with the following command, but I got this error:

$ gsutil cp -r gs://m-project-id.appspot.com/2020-05-22T02:01:06_86154 .
Copying gs://lucky-level-dev-6ac34.appspot.com/2020-05-22T02:01:06_86154/2020-05-22T02:01:06_86154.overall_export_metadata...
OSError: The filename, directory name, or volume label syntax is incorrect.

I'm running this command using cmd in a Windows 10 environment. I'd like to be able to download this folder from the cloud to my local drive.

Update

After being, I tried to change the bucket folder (object prefix) as suggested:

gsutil mv gs://my-project-id.appspot.com/2020-05-22T02:01:06_86154 gs://my-project-id.appspot.com/2020-06-23_someFolder

But trying again now throws me a new error:

gsutil cp -r gs://my-project-id.appspot.com/2020-05-22_someFolder .
Copying gs://my-project-id.appspot.com/2020-05-22_someFolder/2020-05-22T02:01:06_86154.overall_export_metadata...
OSError: Invalid argument.9.0 B]

Surely I need to change the name of the file too?

Yester

I suspect the issue is that GCS object prefix includes colons and these may cause issues for Windows. I don't have Windows. Please try gsutil mv gs://m-project-id.appspot.com/2020-05-22T02:01:06_86154 gs://m-project-id.appspot.com/X to rename the object and then try gsutil cp -r gs://m-project-id.appspot.com/X . again. – Adebayo

@Adebayo Thanks for the suggestion. Now I've surpassed that error; sadly now I got the following: OSError: Invalid argument.9.0 B] I'll update my question. Any ideas? – Yester

@Adebayo I also had to rename the inner file that also included colons in its name. Thanks again! – Yester

You may wish to file a bug with Google. The command should work on supported OSs and clearly does not with Windows. issuetracker.google.com – Adebayo

For anyone who is interested, this issue is fixed in gcloud storage (github.com/GoogleCloudPlatform/gsutil/issues/1513) – Sissie

Answer 1

When doing the cp -r gs:/bucket_name/folder . Gsutil will try to create a folder with "bucket_name" name in the current location, Windows doesn't allow the name of folders to have some special characters including the ':'

You can rename the bucket folder (object prefix) with the command @DazWilkin suggested gsutil mv gs://m-project-id.appspot.com/2020-05-22T02:01:06_86154 gs://m-project-id.appspot.com/new_folder_name and then try again with the new folder name.

Also check that all filenames inside the directory structure don't have the ':' character or any other special one.

Moureaux

Thanks for the suggestion. That really seemed to be the problem, sounds logic but now I've got the following: OSError: Invalid argument.9.0 B] Any ideas? – Yester

I also had to rename the inner file that also had colons and the problem was solved. Please, complement your answer with this addition to get a full answer in case someone don't read the comments. Thanks! – Yester

Ok I am glad this has been solved! I have updated my answer. – Moureaux

Answer 2

(Posted solution from question author to place it in the answer section).

All I had to do was rename the inner file that also had colons in its name. Renaming it solved the problem!

Landed

Do you really think this answers the question by itself?? Don't just over moderate... – Yester

@Metafaniel: it is (partial) answer material, so it needs to go in the answer space. I am happy to delete it if you think it is trivial, or close the question as Typo/Unrepro if the answers are unlikely to be useful to future readers, but I do not think this material belongs in the question. Alternatively you can post this answer yourself (minus the attribution statement) as a self-answer and I can delete the CW copy. – Landed

Finally, you could ask a question over at Meta Stack Overflow if you want to get second opinions though. – Landed

Answer 3

In the case of a gcloud firestore export the issue can be avoided by specifying a directory:

# BAD this will auto generate a directory windows doesnt like
gcloud firestore export gs://my-bucket

# GOOD specify a directory
gcloud firestore export gs://my-bucket/firestore-backup

# copy as usual
gsutil -m cp -r "gs://my-bucket/firestore-backup" .

Roshelle