Stackdriver Logs-Based Metrics - need sum over alignment period

Question

We have some stackdriver log entries that look something like this:

{
  insertId:  "xyz"  
  jsonPayload: {
    countOfApples:  100   
    // other stuff
    }
  // other stuff
}

We would like to be able to set up a log-based metric that tells us the total number of apples seen in the past 10 mins (or any alignment period) but I have, thus far, been unable to find a means of doing so despite reading through the documentation.

Attempt 1:

Filter for those log-entries where countOfApples is specified and create a Counter metric with countOfApples as a label.

having done this, I can filter based on the countOfApples being above or below a certain value. I cannot see a means of aggregating based on this value. All the aggregation options seem to apply to the number of log entries matching the filter over the alignment period

Attempt 2:

Filter for those log-entries where countOfApples is specified and create a distribution metric with the Field Name set to jsonPayload.CountOfApples

This seems to get closer because I can now see the apple count in the metrics explorer but I cannot find the correct combination of Aligner/Reducers to just give me the total number of apples over the period? Selecting Aligner:delta & Reducer:sum results in an error message:

This aggregation does not produce a valid data type for a Line plot type. Click here to switch the aligner to sum and the reducer to 99th percentile

Is it possible to just monitor the total sum of all these values over each alignment period?

Bertold

You should filter to match "=" "countOfApples", try to avoid "Group by" When used without filtering or grouping, aggregation is applied across all the time series in a metric, reducing them to a single time series consisting of the mean, sum, or other measure as calculated across all the time series. Also for the alignment interval per the table here [1] you should use 1 minute, this is just for multiple time series and is a prerequisite to agregation. I'm going to try to reproduce this and let you know. [1]: cloud.google.com/monitoring/charts/metrics-selector#alignment – Georgeannageorgeanne

@Georgeannageorgeanne Thanks for responding but I'm not sure I understand what it is you are suggesting. I have no problem filtering the relevant logs; I'd like to be able to sum the total values of countOfApples over all filtered logs in the interval. So if we have 3 different log messages inside the window with a countOfApples of 1, 2 and 5 respectively, I'd want to see a metric with a value of 8. – Bertold

It seems that the labels/fields are a property of each datapoint, isn't possible to do a sum over those since you can't count metric counts. I'm not sure if using Grafana [1] could help you. [1]: docs.grafana.org/features/datasources/stackdriver – Georgeannageorgeanne

@Bertold can you hint on #56821681 – Exactitude

Hey.. can we calculate the mean of "countOfApples" and plot it in metrics? – Exactitude

Answer 1

As of 2019/05/03, it is not possible to create a counter metric based on the values stored in the logs. Putting the values into a label simply exposes them as strings, which lets you filter but not perform aggregations based on those values. According to the documentation, a counter metric counts log entries, not the values in those log entries. As you've noticed, there aren't enough operations available on distribution metrics to do what you want.

For now, your best bet is to write your own custom metric based on those log values. You can do this by exporting your logs to Cloud Pub/Sub and writing some code to process the logs from Pub/Sub and send custom metrics. Alternatively, you could try to configure the Stackdriver monitoring agent to extract the values using the tail plugin, and send them as custom metrics.

If you just need to graph and explore the values (rather than, e.g., use them for alerting), you could try Cloud Datalab.

Bryant

I'm running into this exact same issue. Has this feature been added yet in the last couple of years? Seems pretty logical to have this. – Coping

Answer 2

If anyone still looking how to solve it, it seems that now it's possible to do sum aggregation on distribution metric using sum_from function. Example:

fetch k8s_container
| metric 'logging.googleapis.com/user/tracking-data-len'
| group_by [], sum(sum_from(value))

Leroi