I am using the Google Cloud Stackdriver, logging from a GKE using a code snippet similar to:
import google.cloud.logging
logging_client = google.cloud.logging.Client()
# connects the logger to the root logging handler
cloud_logger = logging_client.logger('email_adaptor')
struct = {'message':'Processed Alarm', 'valid': True}
cloud_logger.log_struct(struct, severity='INFO')
I verified the page https://cloud.google.com/appengine/articles/deadlineexceedederrors for dealing with DeadlineExceededErrors but, it isn't clear what can cause google.api_core.exceptions.ServiceUnavailable.
The status for the Stackdriver has been Available on Google Cloud Status Page for the past few days. I have been getting such error about 12 times per hour.
Exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
return callable_(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 533, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking
raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "Deadline Exceeded"
debug_error_string = "{"created":"@1540904982.397797031","description":"Deadline Exceeded","file":"src/core/ext/filters/deadline/deadline_filter.cc","file_line":68,"grpc_status":14}"
>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/alarm-handler/helpers/email_utils.py", line 144, in forward_alarm_smtp
severity='INFO', production=not test_mode)
File "/home/alarm-handler/helpers/log.py", line 84, in log
cloud_logger.log_struct(cloud_struct, severity=severity)
File "/usr/local/lib/python3.6/dist-packages/google/cloud/logging/logger.py", line 336, in log_struct
client.logging_api.write_entries([entry_resource])
File "/usr/local/lib/python3.6/dist-packages/google/cloud/logging/_gapic.py", line 124, in write_entries
partial_success=partial_success)
File "/usr/local/lib/python3.6/dist-packages/google/cloud/logging_v2/gapic/logging_service_v2_client.py", line 341, in write_log_entries
request, retry=retry, timeout=timeout, metadata=metadata)
File "/usr/local/lib/python3.6/dist-packages/google/api_core/gapic_v1/method.py", line 139, in __call__
return wrapped_func(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/google/api_core/retry.py", line 260, in retry_wrapped_func
on_error=on_error,
File "/usr/local/lib/python3.6/dist-packages/google/api_core/retry.py", line 177, in retry_target
return target()
File "/usr/local/lib/python3.6/dist-packages/google/api_core/timeout.py", line 206, in func_with_timeout
return func(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/google/api_core/grpc_helpers.py", line 61, in error_remapped_callable
six.raise_from(exceptions.from_grpc_error(exc), exc)
File "<string>", line 3, in raise_from
google.api_core.exceptions.ServiceUnavailable: 503 Deadline Exceeded
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
return callable_(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 533, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking
raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "Deadline Exceeded"
debug_error_string = "{"created":"@1540905002.399793138","description":"Deadline Exceeded","file":"src/core/ext/filters/deadline/deadline_filter.cc","file_line":68,"grpc_status":14}"
Did anyone encounter such error in the past? What do you think can cause this error? I have been wondering if it is related to the amount of logging?