How do I identify the Google Cloud Storage URI from my Google Developers Console?

Question

When I attempt load data into BigQuery from Google Cloud Storage it asks for the Google Cloud Storage URI (gs://). I have reviewed all of your online support as well as stackoverflow and cannot find a way to identify the URL for my uploaded data via the browser based Google Developers Console. The only way I see to find the URL is via gsutil and I have not been able to get gsutil to work on my machine.

Is there a way to determine the URL via the browser based Google Developers Console?

Jardena

Could you post (maybe a separate question) the problems you encountered trying to set up gsutil? – Jacquettajacquette

yes, how did you upload your data? If it's small enough you can upload it straight to BigQuery – Darrendarrey

I uploaded data via the google developers console. The suggestion to try gs://bucket/file name worked. This was very helpful. – Jardena

Answer 1

The path should be gs://<bucket_name>/<file_path_inside_bucket>.

Bach

Ridiculous that I had to find this answer on SO rather than in Google's own docs. Thank you @nmore! – Anastasius

It's regrettable that this question did not get favorited – Liturgical

@StanJames it's normal picture – Sandbox

Answer 2

To answer this question more information is needed. Did you already load your data into GCS?

If not, the easiest would be to go to the project console, click on project, and Storage -> Cloud Storage -> Storage browser.

You can create buckets there and upload files to the bucket.

Then the files will be found at gs://<bucket_name>/<file_path_inside_bucket> as @nmore says.

Darrendarrey

Answer 3

Couldn't find a direct way to get the url. But found an indirect way and below are the steps:

1. Go to GCS
2. Go into the folder in which the file has been uploaded
3. Click on the three dots at the right end of your file's row
4. Click rename
5. Click on gsutil equivalent link
6. Copy the url alone

Eastwards

Answer 4

Follow the following steps :
1. Go to GCS
2. Go into the folder in which the file has been uploaded
3. On the top you can see overview option
4. You can see there will be Link URL and link for GSUtil

Hostility

Answer 5

Retrieving the Google Cloud Storage URI To create an external table using a Google Cloud Storage data source, you must provide the Cloud Storage URI.

The Cloud Storage URI comprises your bucket name and your object (filename). For example, if the Cloud Storage bucket is named mybucket and the data file is named myfile.csv, the bucket URI would be gs://mybucket/myfile.csv. If your data is separated into multiple files you can use a wildcard in the URI. For more information, see Cloud Storage Request URIs.

BigQuery does not support source URIs that include multiple consecutive slashes after the initial double slash. Cloud Storage object names can contain multiple consecutive slash ("/") characters. However, BigQuery converts multiple consecutives slashes into a single slash. For example, the following source URI, though valid in Cloud Storage, does not work in BigQuery: gs://[BUCKET]/my//object//name.

To retrieve the Cloud Storage URI:

Open the Cloud Storage web UI.

CLOUD STORAGE WEB UI

Browse to the location of the object (file) that contains the source data.

At the top of the Cloud Storage web UI, note the path to the object. To compose the URI, replace gs://[BUCKET]/[FILE] with the appropriate path, for example, gs://mybucket/myfile.json. [BUCKET] is the Cloud Storage bucket name and [FILE] is the name of the object (file) containing the data.

Circumfuse

You took several paragraphs to answer a question that could have been answered in one and it seems like you simply copied and pasted the answer from elsewhere – Liturgical

Answer 6

If you need help on subdirectories, check this out on https://cloud.google.com/storage/docs/gsutil/addlhelp/HowSubdirectoriesWork

And https://cloud.google.com/storage/images/gsutil-subdirectories-thumb.png, if you need to see how gsutil provides a hierarchical view of objects in a bucket.

Pronto

Answer 7

In addition to nmore's answer:

def blobInternalUri(blob):
    return f"gs://{blob.bucket.name}/{blob.name}"

It is unclear to me as well why this is not a method of the Blob class.

Vladimir

Answer 8

This will be the full path

https://storage.googleapis.com/<bucketname>/<file path>

Organogenesis