How to send and filter structured logs from dataflow job

Question

The goal is to store audit logging from different apps/jobs and be able to aggregate them by some ids. We chose to have BigQuery for that purpose and so we need to get a structured information from the logs to the BigQuery.

We successfully use apps deployed in kubernetes engine that log output logs to stdout as json string and it is parsed and the structure can be seen in stackdriver as jsonPayload. We took it from this tutorial.

However when I use the same log appender from within dataflow job, it is not treated as a structured message and is viewed as a string in jsonPayload.message field.

I need the structure for two things:

• to use it in filter in custom exporter to big query
• to have the structure in BigQuery as described here

What is the easiest way to achieve this?

Mulvey

Having the same issue, did you figure this out? – Clamatorial

Same for use. Any update on this ? – Titustityus

Answer 1

I know it's quite late but surprisingly there is still no easy answer to this use case as of my research now (2022-03-22).

You have 2 ways to solve your issues:

• [Preferred] You yield your logs as TaggedOutput and you ingest them "by hand" i BigQuery using the writeTableRows function. (i.e. you don't use the recommended SLF4J to log at all but yield your logs as if it was a separate yet real piece of data to process separately).
• [Didn't try and would not recommend but just FYI] You call Stackdriver API directly OR you still yield your logs as above but you "log them at the end" by batch using Stackdriver API directly with the Java client (don't call it synchronously for each log and try to leverage batch to avoid significantly decreasing your performances). => This has throttling limitations

Laski