Docker logs driver gcplogs: How to format logs so that they're shown correctly in Stackdriver?

Question

tldr:
- I'm writing logs from a python application that's running outside of Google Cloud
- I'm using gcplogs to import logs to Stackdriver
- Logs imported must be in the wrong format because they're not displayed as they should in Stackdriver logging

I'm having a hard time figuring out how to configure the gcplogs Docker logs driver so that logs appear as they should in Stackdriver logging. Here's what the logs currently look like:

It looks like stackdriver isn't parsing these logs correctly. However, if the docker image is ran in GKE, the logs look correct:

Here Stackdriver has recognized that these logs were debug messages.

Since these two applications are using the same logger, I think the message that is logged from the application is in the right format, and that it must be the gcplogs configuration that's wrong or missing something. (See this repository for what the python code looks like)

Here's the content of /etc/docker/daemon.json on the machine that's running the Docker image:

{
  "log-driver": "gcplogs",
  "log-opts": {
    "gcp-project": "removed",
    "env": "host"
  }
}

The output of docker info | grep 'Logging Driver' is Logging Driver: gcplogs.

The output of docker version is:

Client:
 Version:       18.03.0-ce
 API version:   1.37
 Go version:    go1.9.2
 Git commit:    0520e24
 Built: Wed Mar 21 23:05:52 2018
 OS/Arch:       linux/amd64
 Experimental:  false
 Orchestrator:  swarm

Server:
 Engine:
  Version:      18.03.0-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.9.4
  Git commit:   0520e24
  Built:        Wed Mar 21 23:14:54 2018
  OS/Arch:      linux/amd64
  Experimental: false

Any tips on how to configure this so that the logs end up looking as they should in Stackdriver would be much appreciated.

Immunology

gcplogs agent is older and less malleable, hence may I confirm if you have tried with the Stackdriver agent to export the logs to Stackdriver? medium.com/tendozen/… – Maximo

Also see the following documentation on "Collecting Docker Log Files with Fluentd and sending to GCP." github.com/GoogleCloudPlatform/k8s-stackdriver/tree/master/… – Maximo

Get 404 when trying to access repo github.com/cognitedata/python-logs-export-docker-gcplogs-test. Is it private? – Antigorite

@Antigorite I'm sorry about that. I created the repository within the GitHub organization of my previous employer. I no longer have access to it. – Immunology