Logging to the Google Cloud in Google Container/Compute Engine with Go

Question

I have a GKE application with 20 nodes running Go. I would like to consolidate all the logs to view in the Google Developers Console log viewer, but I am having 2 problems. I can't get severity filtering, and each newline in my log message starts a new log entry in the viewer (problematic with newlines in the log).

I have the google-fluent-d setup so all stdout gets logged in the cloud, and I have made use of log.Lshortfile, call depth and log.Logger.Output to get filename and line number from the "log" library.

I've looked at this library: "google.golang.org/cloud/logging" but I am having trouble getting it to work.

Is there a library somewhere or an example of the best way to write logs in GKE and GCE?

Tristatristam

Answer 1

There are a couple things you can do depending on how ambitious you're feeling.

The easiest option would likely be to switch your logging format to be JSON. The google-fluentd agent will automatically parse the JSON for you, exporting each message's structured data to the cloud logging API. It will automatically grab severity information if a severity field is in the JSON, and using JSON can keep it from breaking on (escaped) newlines in your messages.

The tougher (but more flexible) option would be to get the client library that you found working. I'm not sure whether it's the same as the one you linked to, but I believe this is the most recent one. If you can give more context about the problems you had, I can help or wrangle someone from the cloud logging team to help.

Qumran

Thank you! Is there any documentation for what can be added to the JSON log? Eg what is the key for the actual message, or flags. – Tristatristam

It doesn't appear to be well-documented at all, sorry. The special fields are only really "severity" for log severity and "message" for the actual log message. The code itself is the place I got this from: github.com/GoogleCloudPlatform/fluent-plugin-google-cloud/blob/… – Qumran

This seems to have stopped working since upgrading nodes to >= 1.6. The whole JSON object now appears un-parsed under textPayload :-( – Ommatophore

The issue I mentioned in my previous comment is this one, which looks like it will be fixed soon :) github.com/kubernetes/kubernetes/issues/48108 – Ommatophore

Answer 2

I used custom logging class and instead of log every message I saved them in a variable, and send it after the server response - so I finally have in google cloud only one logging for every request with a very long textPayload.

The custom logging class looks something like this:

class CustomLogging {
   
    messages = "";

    log(message) {
        this.messages += '\n' + message;
    }

    sendLogs(message) {
        console.log(this.messages);
    }

}

Every request log look like this:

Skindeep