"bq" command line tool throws CERTIFICATE_VERIFY_FAILED

Question

Update (2019-02-07): the issue has now been fixed, so if you're still running into this, try gcloud components update.

---

At some point during the past few monthts, my bq tool stopped working. Even a simple thing shows this error:

$ bq show
BigQuery error in show operation: Cannot contact server. Please try again.
Traceback: Traceback (most recent call last):
File "/opt/google-cloud-sdk/platform/bq/bigquery_client.py", line 685, in BuildApiClient
response_metadata, discovery_document = http.request(discovery_url)
File "/opt/google-cloud-sdk/platform/bq/third_party/oauth2client_4_0/transport.py", line 176, in new_request
redirections, connection_type)
File "/opt/google-cloud-sdk/platform/bq/third_party/oauth2client_4_0/transport.py", line 283, in request
connection_type=connection_type)
File "/opt/google-cloud-sdk/platform/bq/third_party/httplib2/__init__.py", line 1626, in request
(response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "/opt/google-cloud-sdk/platform/bq/third_party/httplib2/__init__.py", line 1368, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "/opt/google-cloud-sdk/platform/bq/third_party/httplib2/__init__.py", line 1288, in _conn_request
conn.connect()
File "/opt/google-cloud-sdk/platform/bq/third_party/httplib2/__init__.py", line 1082, in connect
raise SSLHandshakeError(e)
SSLHandshakeError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:726)

I've tried the following:

• sudo gcloud components update (version 221.0.0).
• sudo pacman -Syu (system update) to get the latest set of SSL certificates. This is Arch Linux, so pretty much always bleeding edge.
• sudo gcloud components reinstall.
• Uninstalling google-cloud-sdk, wiping out remaining /opt/google-cloud-sdk and reinstalling entirely from AUR.
• Adding --httplib2_debuglevel=3 (valid values are not documented, found the value 3 here). This does not give any extra output.
• Adding one of --ca_certificates_file=/etc/ca-certificates/extracted/tls-ca-bundle.pem, --ca_certificates_file=/etc/ca-certificates/extracted/ca-bundle.trust.crt and --ca_certificates_file=/etc/ssl/certs/ca-certificates.crt one of which must surely be the bundle of root certificates on my system. The last one of these is used by curl, which can talk to www.googleapis.com just fine.
• Poking at the source code to discover that /opt/google-cloud-sdk/platform/bq/third_party/httplib2/cacerts.txt is the cert bundle used by default. If I try this one with curl --cacert ..., it still works.
• Setting the GOOGLE_APPLICATION_CREDENTIALS environment variable in this shell. As expected, this also doesn't make a difference; the SSL error occurs before bq has even had a chance to begin the OAuth handshake.
• Adding --disable_ssl_validation. This "works" but is obviously not secure.

Anyone else seeing this, or have ideas how to debug/solve?

Bornstein

I wonder if this is something related to Python OpenSSL itself. If you try to make SSL connections using requests to any secured url, does it work? Maybe updating pyopenssl and http2 might change something as well. – Nopar

I got same error today, after updating to Ubuntu 18.10, temporarily using --disable_ssl_validation though its not recommended – Unweighed

Probably it has something to do with new Ubuntu 18.10? Since I also have it installed. – Koel

Answer 1

I'm seeing the exact same issue using Arch Linux as well.

When you issue a bq command on the command line however, I'm pretty sure that the certificate file at /opt/google-cloud-sdk/platform/bq/third_party/httplib2/cacerts.txt is not used, because the flag --ca_certificates_file=/etc/ssl/certs/ca-certificates.crt will is put into the flags automatically in the application bootstrap process. On Arch Linux, this file is a symlink to /etc/ca-certificates/extracted/tls-ca-bundle.pem.

I've tried using curl and openssl s_client with this CA bundle against the API URL being called, which is

https://www.googleapis.com/discovery/v1/apis/bigquery/v2/rest

and it works just fine.

My assumption is, that this is not an issue with missing or expired certificates. My pyopenssl package is at version 18.0.0, so I'm at the newest version here. However, I think this issue is caused by unsupported ciphers or algorithms in the TLS handshake process.

Barabbas

Thank you for confirming I'm not crazy. Must be something Arch-related then... somehow. Did you install google-cloud-sdk from AUR as well? – Bornstein

Never mind, I tried from the official tarball and it also fails. I filed an issue with gcloud: issuetracker.google.com/issues/117948931 – Bornstein

Answer 2

There's a public issue tracker with a similar behavior that you're having. I suggest starring it to keep updated about it as well providing your scenario.

If you're behind a corporate proxy, on comment #8 there's a scenario which the corporate proxy replaces the certificate, and the workaround is provided on comment #16

Hope it helps.

Reproachless

Thanks, but it looks like a different error. And I'm not behind a proxy. issuetracker.google.com/issues/115556782 looks closer to the mark, but is also eventually about proxies. – Bornstein