Delete multiple objects fron google cloud storage using node.js

Question

I need to delete multiple objects from google cloud storage. I have deleted one object at a time.

This is my code:

var gcloud = require('gcloud')({
  projectId: "sampleProject1"
});
var gcs = gcloud.storage();
var myBucket = gcs.bucket('sampleBucket1');
var file = myBucket.file('1.png');

file.delete(function (err, apiResponse) {
  if (err) {
    console.log(err);
  }
  else {
    console.log("Deleted successfully");
  }
});

But I need to delete multiple objects simultaneously. Is it possible or not?

Nations

Answer 1

We do have bucket#deleteFiles that will handle throttling the requests for you. You can use the prefix option to target multiple images by a naming convention, like:

bucket.deleteFiles({ prefix: 'image-' }, callback);

If that doesn't work, we also have a guide that shows how you can do the throttling logic yourself. See "My requests are returning errors instructing me to retry the request": https://googlecloudplatform.github.io/gcloud-node/#/docs/v0.29.0/guides/troubleshooting

Edit to elaborate on how to do the throttling using async:

var async = require('async');
var PARALLEL_LIMIT = 10;

function deleteFile(file, callback) {
  file.delete(callback);
}

async.eachLimit(filesToDelete, PARALLEL_LIMIT, deleteFile, function(err) {
  if (!err) {
    // Files deleted!
  }
});

Dwt

Answer 2

There is no way to atomically delete multiple objects from GCS at the same time.

However, you can issue multiple requests with a single call, which returns the status of each individual operation, using the batch API.

Cloth

Answer 3

I created an array of all the objects that I wanted to delete and then iterated over the array, executing the "delete" function specified on the google cloud storage documentation.

Delete GCS object

const storage = new Storage({keyFilename: 'google-credentials.json'});
const imagesToDelete = ['fileName1', 'fileName2', 'fileName3'];
    
    imagesToDelete.map(async (image) => {
      await storage
        .bucket('yourbucketName')
        .file(image)
        .delete();
    });
 

Tamis

It would really be better to await a Promise.all() call with the mapped array. – Fante

Answer 4

    var gcloud = require('gcloud')({
    projectId: "sampleProject1"
     });
    var gcs = gcloud.storage();
   var myBucket = gcs.bucket('sampleBucket1');
   var collection = gcs.collection("Add file for delete");  
   collection.insert({'1.png'},{'2.png'});

   collection.delete(function (err, apiResponse) {
   if (err) {
    console.log(err);
    }
   else {
    console.log("Deleted successfully");
   }
   });

Ruphina

The code might work, but please explain a little bit about what it does exactly so that OP can understand more clearly why this code should be used to achieve his goal. Just edit your answer a bit and add some commentary there. – Mistrustful

Use collection like(list,dic,map etc) – Ruphina

Where did gcs.collection come from? We don't have that method on our API: googlecloudplatform.github.io/gcloud-node/#/docs/v0.30.1/… :) – Dwt