Google Cloud Storage Search Files

Question

I'm developing an application that will serve a lot of file. Lets say it's a car company. Each car has a folder with documents/files for that car. These files are uploaded into categories, represented as "folders" in GStorage. 2 files for 2 different cars could look like this:

car1/receipt/2016-02-02_payment.pdf
car1/pictures/tires.png

car2/receipt/2016-01-02_payment.pdf

Lets say I want to list all receipts in my application. How would I do this? Since the search capabilities in Google Cloud Storage is very limited my current proposed solution is to have a mirrored database table with all of the files in it, when i want to access the files i generate the URL for the user. It would be very nice is GStorage had a way to search for */ files.

Phyle

Answer 1

In your example, you could achieve this using wildcards with gsutil to list your bucket contents:

gsutil ls gs://your-bucket/car*/receipt

Chipboard

Would this work using PHP? I need to bring it to a web front-end – Phyle

You can make API calls using prefix listing and a delimiter ("/" is the default). You'd have to implement matching logic similar to gsutil's to produce the desired result. First, list prefixes starting with "car", then make separate list calls on the results, extending them with the prefix "receipt". – Chipboard

How do I only fetch prefixes though? It keeps listing tons of unrelated files each time I use a delimiter. Thank you so much for your time by the way! – Phyle

You can use the 'fields' parameter to filter out object names (See github.com/GoogleCloudPlatform/google-cloud-php/blob/master/src/…). Drop 'items/name' and include only the 'prefixes' and 'pageToken' fields in your first list call. – Chipboard

You are an angel, then I'll just have to make separate calls for each prefix. Although if I remember correctly I should be able to make a batchcall, but maybe that's only with gsutil. A separate call for each prefix is likely to take up a lot of time. I'm going to bed in a few, will have to look into that and your answer tomorrow :) if you know anything feel free to drop another comment! – Phyle

Answer 2

gsutil supports * and ? wildcards only for files. To include folders in the wildcard target you need to double the * or ? sign. For instance, gsutil ls gs:///**.txt will list all the text files in all subdirectories. The wildcard page offers more details.

Viewless

Answer 3

To list or search for all the text files in Google Cloud Storage buckets, you can use either of the following commands:

1. For a specific bucket:

   gcloud storage ls gs://<bucket>/**.txt

2. To search across all buckets:

   gcloud storage ls gs://*/**.txt

See the wildcard page for more details: https://cloud.google.com/storage/docs/wildcards

Newmark