How to get detailed Big Query error by using PYTHON

Question

I'm looking to see this information as python message:

But currently, I can see only first/second

This is what I'm currently using

from google.api_core.exceptions import BadRequest

if __name__ == '__main__':
      try:
          upload('XXX','XXX')
      except BadRequest as e: 
          print('ERROR: {}'.format(str(e)))

UPLOAD:

def upload(FILE_NAME, TABLE_ID):

    client = bigquery.Client()

    dataset_ref = client.dataset(config.DATASET_ID )
    table_ref = dataset_ref.table(TABLE_ID)
    job_config = bigquery.LoadJobConfig()
    job_config.source_format = bigquery.SourceFormat.NEWLINE_DELIMITED_JSON
    job_config.autodetect = False

    with open(FILE_NAME, 'rb') as source_file:
        job = client.load_table_from_file(
            source_file,
            table_ref,
            location='EU',  # Must match the destination dataset location.
            job_config=job_config)  # API request

    job.result()  # Waits for table load to complete.

Besom

Please add code for upload() method – Fayefayette

@Fayefayette Done, please have a look – Besom

@Fayefayette ANy suggestions?? – Besom

@FelipeHoffa Do you have any suggestions? – Besom

Have you tried other Exception classes? The 3rd error might not be a BadRequest exception? google-cloud-python.readthedocs.io/en/0.32.0/_modules/google/… – Stithy

@Stithy I have but no luck - google-cloud-python.readthedocs.io/en/0.32.0/core/… – Besom

Answer 1

This error is not raised as an exception. You can find it in the errors property of the LoadJob object. Here is an example:

try:
    job.result()
except BadRequest as e:
    for e in job.errors:
        print('ERROR: {}'.format(e['message']))

Output:

ERROR: Error while reading data, error message: JSON table encountered too many errors, giving up. Rows: 1; errors: 1. Please look into the errors[] collection for more details.
ERROR: Error while reading data, error message: JSON processing encountered too many errors, giving up. Rows: 1; errors: 1; max bad: 0; error percent: 0
ERROR: Error while reading data, error message: JSON parsing error in row starting at position 0: No such field: SourceSystem.

Newlywed

Great, thank you. Only a nitpick, you could delete the as e because a) the exception is not used, and b) e is redefined in the loop – Neurogenic