No data in Cloud Monitoring from user-defined metrics in Cloud Logging

Question

I have created 2 user-defined metrics in the Cloud Logging UI. Those metrics show up in Cloud Monitoring, but their graphs are perpetually showing "no graph data found". Are there any steps to troubleshoot this or are there other requirements to have the data from user-defined log metrics be visible in Cloud Monitoring?

Marinelli

Just to follow-up on this. New user-defined log metrics are now showing up in Stackdriver. I suspect there was a bug for some period of time that's not fixed, since nothing changed on my end. – Marinelli

New log entries might not match your metric's logs query. A logs-based metric gets data from matching log entries that are received after the metric is created. Logging does not backfill the metric from previous log entries. Reference: cloud.google.com/logging/docs/… – Hesterhesther

Answer 1

Were there matching log entries after you created the metric? The logs-based metrics start counting matching entries only after metric creation time.

If there were matching log entries after metric creation, did you wait a few minutes to see if there was data in your graphs? It takes a few minutes to update the logs-based metrics in Cloud Monitoring, so you may see log entries in Cloud Logging that are not yet counted in Cloud Monitoring.

If you did wait at a few minutes, was there any delay on your log ingestion? For this it would be good to know where the logs were coming from. If a log entry arrives late to Cloud Logging, it will appear in the Logs Viewer but will not be counted in the logs-based metrics. A log entry is considered late if it arrives more than two minutes after the timestamp included in the log entry. The number of late-arriving entries is recorded for each log in the system metric, logging.googleapis.com/dropped_log_entry_count.

Some of these steps are documented here: https://cloud.google.com/logging/docs/view/logs_based_metrics#troubleshooting

Storehouse

Answer 2

I assume you're using Cloud Monitoring v2beta custom metrics. I also assume that you have not only created the metrics themselves but also sent timeseries data into these metrics.

I'd start with listing the timeseries data using the API call to to "monitoring.projects.timeSeries.list" see if your data is really there, otherwise the Cloud Monitoring UI will display the metrics but won't have any data in them. You can use API Explorer to facilitate this test.

P.S. Custom metrics v2 are being depreciated these days and now are being replaced by v3. You might want to update your code to reflect these changes using this guide

Matty

Thanks, I'm not using any APIs directly. I created the user-defined log-based metric in the Cloud Logging UI, then created the alert policy in the Cloud Monitoring UI from that metric. I'll try poking around with the APIs to see what's happening at a lower level. – Marinelli

If this is the case, are you certain you've configured the correct "query" in the log based metric? Are you seeing entries matching the "query" in Cloud Logging? – Matty

Yup, if I click "edit metric" from the logging UI it takes me to the query and shows a steady stream of log data matching it. – Marinelli