How to get google-cloud-ops-agent logging to recognize special "severity" field

Question

I am running ZooKeeper on a google compute instance and trying to setup google-cloud-ops-agent to properly parse the logs. I am most of the way there, but am having trouble getting it to recognize the severity field.

The docs state that severity is a special field that will get extracted from the jsonPayload, but I am not seeing that happening.

• special fields documentation
• logEntry documentation

My /etc/google-cloud-ops-agent/config.yaml

logging:
  receivers:
    zookeeper:
      type: files
      include_paths:
      - /zookeeper/logs/zookeeper.log
  processors:
    zookeeper:
      type: parse_regex
      field: message
      regex: '^(?<time>.{23}) \[(?<zknode>[^\]]+)] - (?<severity>\S+)\s+ \[(?<class>[^\]]+)] - (?<msg>.*)$'
      time_key: time
      time_format: "%Y-%m-%d %H:%M:%S,%L"
  service:
    pipelines:
      zookeeper:
        receivers: [zookeeper]
        processors: [zookeeper]

evidence the parse_regex is working properly (but notice severity still exists in jsonPayload)

Law

Any clues on the Zookeeper logs? What ops agent version are you also using? – Impatient

No, I never got it to work. I can't find the specific version I was using, I was following the instructions at cloud.google.com/monitoring/agent/ops-agent/installation; it was not the 'legacy' or 'preview' versions (at time of this writing). Unfortunately this project got defunded, so I had to move on. I appreciate the suggestions – Law

I've ran into the same problem. I've got a support ticket open and I'll update with what I find. – Poltroon

A public ticket has been created for this issue, See case for more details. issuetracker.google.com/issues/202309453 – Impatient

Answer 1

I fixed it by describing in processors:

    change_severity:
  type: modify_fields
  fields:
    severity:
      copy_from: jsonPayload.severity

and then to service -> pipelines:

      web_access:
    receivers:
      - web_access
    processors:
      - change_severity

Toffic

This is not the recommended way to resolve the issue. This relies on an implementation detail of the logging sub-agent that is subject to change in the future. The destination field should be just severity, not jsonPayload."logging.googleapis.com/severity". – Valladolid

Answer 2

When parsing an unstructured log, the Ops Agent places all fields under jsonPayload unless the field has one of the special names from that table, as you've pointed out from the docs. However, in that table severity is the destination field, not the source. The correct source field is spelled logging.googleapis.com/severity.

Unfortunately the regex engine does not allow special characters in destination field names. If it did, you could simply write your regex as e.g. regex: '^(?<time>.{23}) \[(?<zknode>[^\]]+)] - (?<logging.googleapis.com/severity>\S+)\s+ \[(?<class>[^\]]+)] - (?<msg>.*)$' and the problem would be fixed.

Because you can't write that regex, you need an additional processor to lift the jsonPayload.severity field to the top-level severity field:

logging:
  ...
  processors:
    move_severity:
      type: modify_fields
      fields:
        severity:
          move_from: jsonPayload.severity
  service:
    pipelines:
      ..
        processors:
          ..
          - move_severity

Additionally, severity only recognizes specific inputs, e.g. a warning must be spelled as WARNING and not WARN or W. If you're collecting logs that spell the severity level differently, you can add a corresponding map_values section to the move_severity processor.

Valladolid