I have a BigQuery table I stream updates to from Pub/Sub topic via push subscription writing to it with the topic schema. At some point, I backed up the table with: CREATE TABLE `my-dataset.my-tabl...

I was trying to figure out key differences between using GCP Vertex AI feature store and Saving preprocessed features to BigQuery and loading whenever it gets necessary. I still cannot understand w...

I have a fairly large table in big query ( app. 9M rows) and I would like to read it via pandas. I've tried reading and using the [pd.read_gbq()][1] function, which works fine on small tables. ...

Is it possible to count number of occurrences of a specific log message over a specific period of time from GCP Stackdriver logging? To answer the question "How many times did this event occur duri...

I'm working on a script where I'm sending a dataframe to BigQuery: load_job = bq_client.load_table_from_dataframe( df, '.'.join([PROJECT, DATASET, PROGRAMS_TABLE]) ) # Wait for the load job to c...

I am trying to upload a Dataframe to a BigQuery Table using client.load_table_from_dataframe. And I am also supplying a schema using the job_config = bigquery.LoadJobConfig( schema=[xxxxxx,xxxxxx,...

This must be a super trivial issue, but i've updated my windows virtual machine with; pip install --upgrade google-cloud-storage However, when I run the script I still receive the following erro...

This may be a dummy question but I cannot seem to be able to run python google-clood-bigquery asynchronously. My goal is to run multiple queries concurrently and wait for all to finish in an async...

In BigQuery's documentation , it says that the function returns the approximate result for COUNT(DISTINCT expression). However, I would like to know what's the variance of the result from using APP...

How can I assign surrogate keys when inserting records in a BigQuery table? Something like using Sequence to generate unique values or NextVal ?

I want to run BigQuery queries with thousands of rows of total results, but I only want to retrieve a page of 100 results at a time (using the maxResults and pageToken parameters). The BigQuery AP...

Supose I have the following schema: [ { 'name': 'id', 'type': 'INTEGER' } { 'name': 'record', 'type': 'RECORD', 'fields': [ { 'name': 'repeated', 'type': 'STRING', 'mode': 'REPEATED' ...

Did any one know the how to use the Python Google Cloud BigQuery API to calculate the cost of a query?

I have an array column and I would like to get first N elements of it (keeping an array data type). Is there a some nice way how to do it? Ideally without unnesting, ranking and array_agg back to a...

I have a table with >1M rows of data and 20+ columns. Within my table (tableX) I have identified duplicate records (~80k) in one particular column (troubleColumn). If possible I would like to ret...

I get an error (in red, in picture below) whilst creating pubsub Bigquery subscription. Error shows up in subscription creation view: First question - why is this error appears in the first place?...

We need to create descriptive aliases for fields. Ideally we would like to create views with alias with a space. Is this possible? How can we do this? Example: SELECT word, word_count "Word Coun...

I am working with GA4 data in Bigquery and am attempting to validate against the numbers I see in the Google Analytic UI. In each case, the BigQuery count of events is a little under double that sh...

Given: json string that represents array field selected as a result of query. How to find out length of JSON array field in Big Query?

Is there a way to Select * except [x,y,z column names] in BigQuery? I see some solutions for MySQL but not sure if it applies to BQ. Thank you.

I am new to Google Cloud and BigQuery I have a cloud instance with 2 projects. One is a Compute Engine project and another is a BigQuery project. I can successfully get an Access Token and also a ...

is there any way how to cancel a running query? i use the web interface. first i ran a series of tests on tables of 10k and than 20k rows and the response was in seconds. but than i ran the triple...

How do you get the creation time for a table in the dataset? bq show my_project:my_dataset.my_table gives you Table my_project:my_dataset.my_table Last modified Schema Total Rows Total Bytes E...

I'm using Google bigquery to analyze the GDELT GKG 2.0 dataset and would like to better understand how to query based on themes (or V2Themes). The docs mention a 'Category List' spreadsheet but so ...

BigQuery with Legacy SQL has a pretty convenient QUANTILES function to quickly get a histogram of values in a table without specifying the buckets by hand. I can't find a nice equivalent in aggreg...