How to create an empty folder on Google Storage with Google API?

Question

How to create an empty folder on Google Storage with Google API? (Assume that / is path separator.)

Scriabin

checkout google tutorial regarding to uploading images to google storage. codelabs.developers.google.com/codelabs/cloud-nodejs/#7 – Hoi

Answer 1

Google Cloud Storage does not have folders or subdirectories. However, there is some support for emulating them. gsutil's How Subdirectories Work is a good read for some background.

Google Cloud Storage objects are a flat namespace, but many tools, including gsutil and the Google Cloud Storage UI, create an illusion of a hierarchical file tree.

There are two widely used conventions for creating the illusion of an empty subdirectory:

1. (recommended) Create an object that ends in a trailing slash. For example, to create a subdirectory called foo at the root of a bucket, you would create an empty object (size 0) called foo/.

2. (legacy) Create an object with _$folder$ appended to the name. For example, to create a subdirectory called foo at the root of a bucket, you would create an empty object (size 0) called foo_$folder$.

Note that most tools and utilities are using method 1 now. Method 2 is less frequently used.

Rectum

This answer is over a year old, so it's possible that things have changed. If not, can someone tell me exactly how (preferably using gsutil) I can create an empty object with a name ending with '/' ? I'm hoping to create explicit folders for use with gcsfuse. Thanks. – Toilet

if I create an empty file locally, and try to upload it like this: "gsutil cp dummy_file gs://<bucket-name>/<folder-name>/" it just creates an empty object at gs://<bucket-name>/<folder-name>/dummy_file ... and if I try to copy it to an object called gs://<bucket-name>/<folder-name>_$folder$ I get an object called gs://<bucket-name>/<folder-name>_$ @Rectum – Toilet

The info is current: there's still no way to create a directory placeholder object with gsutil. – Rectum

ok, good to know -- can you provide an example of how to do it using an API/endpoint? (either raw https or python, if possible) – Toilet

Currently GCS allows create folders (including nested) in buckets. – Typhogenic

Answer 2

@SheRey - looking at folders created via the GCS web interface, the Content-Type is set to application/x-www-form-urlencoded;charset=UTF-8 but it doesn't really matter. Here's what worked for me in python:

# pip install google-cloud-storage

from google.cloud import storage

gcs_client = storage.Client(project='some_project')
bucket = gcs_client.get_bucket('some_bucket')
blob = bucket.blob('some/folder/name/')

blob.upload_from_string('', content_type='application/x-www-form-urlencoded;charset=UTF-8')

Haematogenesis

Thanks and congratulations, you've accomplished what Google deemed to be "Infeasible" (issuetracker.google.com/issues/148487275). – Demos

Don't you need to create all the intermediate directories this way? First some, than some/folder, than some/folder/name? – Steradian

@AntonDaneyko no, that is not neccesary. – Positive

Answer 3

Thank you for the Question and the chosen Best answer. Here is a code snippet that I wrote: Python Method:

def create_folder(bucket_name, destination_folder_name):
    storage_client = storage.Client()
    bucket = storage_client.get_bucket(bucket_name)
    blob = bucket.blob(destination_folder_name)

    blob.upload_from_string('')

    print('Created {} .'.format(
        destination_folder_name))

main code that calls the method:

folder = create_folder(bucket_name, 'test-folder/')

Microwave

A lot has changed over the years and this is now the best answer. – Arkwright

Answer 4

Node.js + @google-cloud/storage@^2.5.0:

Only you need do is to assign a destination like: <folder>/<file Name> pattern.

For below example, I am using uuid as my folder name to simulate each user has a folder to store their own files.

 it('should upload file and create a folder correctly', async () => {
    const myStorage = new Storage({ keyFilename: path.resolve(__dirname, '../../../.gcp/cloud-storage-admin.json') });
    const bucket = myStorage.bucket('ez2on');
    const fileName = 'mmczblsq.doc';
    const filePath = path.resolve(__dirname, `../../../tmp/${fileName}`);
    const uuid = faker.random.uuid();

    await bucket.upload(filePath, {
      destination: `${uuid}/${fileName}`,
      gzip: true,
      metadata: {
        cacheControl: 'public, max-age=31536000'
      }
    });
  });

The result is:

Here is the API docs for @google-cloud/storage: https://googleapis.dev/nodejs/storage/latest/Bucket.html#upload

Go + cloud.google.com/go/storage

package main

import (
    "cloud.google.com/go/storage"
    "context"
    "fmt"
    "github.com/google/uuid"
    "google.golang.org/api/option"
    "io"
    "log"
    "os"
)

func main() {
    ctx := context.Background()
    opts := option.ClientOption(
        option.WithCredentialsFile(os.Getenv("CredentialsFile")),
    )

    client, err := storage.NewClient(ctx, opts)
    if err != nil {
        log.Fatalf("%v", err)
    }
    filename := "mmczblsq.doc"
    filepath := fmt.Sprintf("./tmp/%s", filename)
    file, err := os.Open(filepath)
    if err != nil {
        log.Fatalf("%v", err)
    }
    defer file.Close()

    uuidIns, err := uuid.NewUUID()
    if err != nil {
        log.Fatalf("%v", err)
    }
    object := fmt.Sprintf("%s/%s", uuidIns, filename)
    log.Printf("object name: %s", object)
    wc := client.Bucket("ez2on").Object(object).NewWriter(ctx)
    if _, err := io.Copy(wc, file); err != nil {
        log.Fatalf("%v", err)
    }
    if err := wc.Close(); err != nil {
        log.Fatalf("%v", err)
    }
}

Output of stdout:

☁  upload [master] ⚡  CredentialsFile=/Users/ldu020/workspace/github.com/mrdulin/nodejs-gcp/.gcp/cloud-storage-admin.json go run main.go
2019/07/08 14:47:59 object name: 532a2250-a14c-11e9-921d-8a002870ac01/mmczblsq.doc

Check the file in google cloud platform console:

Un

Answer 5

if you are using java SDK try this.

Blob folderCreated = bucket.create(folder_name + "/", "".getBytes());

1. folder_name should end with "/"
2. upload empty bytes

Schulte

Answer 6

I got through by "creating" the folder when naming the object to be written.

storage_client = storage.Client()
bucket = storage_client.bucket(bucket_name)

blob = bucket.blob("new_folder_name/destination_blob_name")

blob.upload_from_filename(source_file_name)

Freddyfredek

Answer 7

I was trying to create a unique subdirectory on Google Storage bucket from CircleCI instance. Here's how i did it using gsutil. I created a dummy file. I also appended my folder name to a destination bucket name with slash.

Usage: gsutil cp OBJECT_LOCATION gs://DESTINATION_BUCKET_NAME/

sudo gsutil cp ~/some-path/dummyFile.txt gs://my-bucket/unique-folder-name/

Ombudsman

Answer 8

This seemed to work for me for node

import os = require('os');

const tempFilePath = path.join(os.tmpdir(), 'temp.txt');
fs.writeFileSync(tempFilePath, '');
await admin.storage().bucket().upload(tempFilePath, {
  destination: `other-folders/new-folder/`,
});

Norther

Answer 9

Using node.js sdk

const { Storage } = require('@google-cloud/storage');
const storage = new Storage();

async function addEmptyFolder() {
  return storage.bucket('my-bucket')
    .file('folderA/folderB/')
    .save('');
}

Aether