Rename a folder in GCS using gsutil

Question

I can't rename an existing folder in GCS. How do I do this?

As per the documentation, this should be:

gsutil mv gs://my_bucket/olddir gs://my_bucket/newdir

However, what happens is that olddir is placed under newdir, i.e the directory structure is like this (after the call to gsutil mv):

my_bucket
    newdir
        olddir

instead of (what I would expect)

my_bucket
    newdir

I've tried all four combinations of putting trailing slashes or not, but none of them worked.

Chalybite

What is the output from the gsutil command when you run it and end up with the unwanted nested folder structure? Normally you would see a Copying... and Removing... line recursively for all the files contained under your olddir. – Galleon

Possible duplicate of: #45897212 ...which confirms my answer - while I even provided a viable solution approach to the problem, instead of just telling "it's not possible". – Orthopterous

@yoape: Yes, I see lots of "Copying gs://[..]" and "Removing gs://[..]" lines. – Chalybite

Generally speaking, in object storage systems like GS and AWS S3, a folder is not a real first-class item the way a file is. GS seems to have fixed this bug, but in object management you should design around files, not folders. – Schmo

The issue (bug) is fixed now and it works well. The only problem is that it doesn't delete the old directory, but the files from the old directory are moved to the newly created directory. A tip: use -m if you have a lot of files - this way the moving can be done in parallel, meaning much faster. – Permalloy

Answer 1

This is a confirmed bug in GCS, see https://issuetracker.google.com/issues/112817360

It actually only happens, when the directory name of newdir is a substring of olddir. So the gsutil call from the question actually works, but the following one would not:

gsutil mv gs://my-organization-empty-bucket/dir_old gs://my-organization-empty-bucket/dir

Chalybite

According to the issue tracker this has been fixed since gsutil v4.43. I'm using v4.51 and it worked like a charm. – Soche

Answer 2

I reproduced your case by having a bucket with a folder named olddir of which I want to move the content to newdir folder.

the following command:

 gsutils mv gs://<bucketname>/olddir gs://<bucketname>/newdir

moved the whole content of folder to the newly created newdir folder.

Olddir and newdir folders were then at the same level, in the bucket root. after that I just had to remove the folder called olddir.

Objects in a bucket cannot be renamed.

The gsutil mv command does not remove the previous folder object like the mv comand would do in Unix CLI.

I guess that if you have tried moving folders several times by using "/" characters placed differently, the structure and hierarchy of the folders will have changed after issuing the initial command.

Please try again from the beginning.

Bear in mind that once you have a subfolder inside a folder, objects will have to be moved one by one using the full path.

Buddleia

I double-checked everything (in a new empty bucket, with new folders/files) and I still see the wrong behaviour. This is very odd, seems not like a usage problem, but like a bug in GCS. – Chalybite

pastebin.com/iksntwwE In case you're interested in the actual commands I used. – Chalybite

It is because in your case the folder gs://my-organization-empty-bucket/test_data already exist when you run the mv command, so it copies the data_b folder inside the folder named test_data. If the destination test_data folder would not exist it would create it and move the content of data_b folder inside the test_data folder. – Buddleia

According to the script output: gs://my-organization-empty-bucket/test_data/text.txt $ gsutil rm -r gs://my-organization-empty-bucket/test_data Removing gs://my-organization-empty-bucket/test_data/text.txt#1534770322877415... / [1 objects] Operation completed over 1 objects. the file inside the folder was deleted but not the folder itself. when I reproduce on my side I have operation completed on 2 objects, the file and the folder. – Buddleia

The concept of a folder does not exist in GCS. When the last file is deleted, it also deletes the folder. Also, I'm calling rm -r on the folder. That aside, the bug also happens if I don't create the directory before. Apparently it's enough if the new directory name is a substring of the old directory name ... – Chalybite

@Chalybite Hi! I'm with GCP Support. I saw that you posted in the Public Issue Tracker. Any comments on the issue will be posted over there. Feel free to, later on, post anything from there in this question, for future reference of the community :) – Crampon