Can't see application log in Google Cloud Logs

Question

How can I view log messages on Google Cloud?: https://console.cloud.google.com/logs

This is what I see in the terminal when I run dev_appserver.py (locally running):

INFO     2016-05-16 14:00:45,118 module.py:787] default: "GET /static/images/contact.png HTTP/1.1" 304 -
INFO     2016-05-16 14:00:45,128 module.py:787] default: "GET /static/images/email.png HTTP/1.1" 304 -
INFO     2016-05-16 14:00:45,136 module.py:787] default: "GET /static/images/phone.png HTTP/1.1" 304 -
INFO     2016-05-16 14:00:45,487 basehandler.py:19] entering basehandler.py
INFO     2016-05-16 14:00:45,516 module.py:787] default: "GET /static/images/logo-349x209.png HTTP/1.1" 304 -
INFO     2016-05-16 14:00:45,562 requesthandlers.py:26] entering requesthandlers.py
INFO     2016-05-16 14:00:45,563 app.py:28] entering app.py
INFO     2016-05-16 14:00:45,563 app.py:198] Using development database

Both application log messages and request logging is displayed.
However when I view the log of the same code deployed I can only see the requests being logged:

The code I'm using to generate application log messages is something like:

import logging
logger = logging.getLogger("someLogger")

logger.info("entering app.py")

But I've also tried using logging.info(...) directly with the same results.

I've tried finding an answer to this in various resources but I've come up empty-handed, most refer to how to set log level when developing locally.
I'm guessing that I need to enable some setting in order to view application logs on Google Cloud Logs.

Resources that I've looked at:

• https://cloud.google.com/logging/docs/view/logs_viewer
• https://cloud.google.com/appengine/docs/python/logs/
• How to change the logging level of dev_appserver
• How do I write to the console in Google App Engine?
• Google App Engine - Can not find my logging messages
• https://docs.python.org/3/howto/logging.html

Instruction

Answer 1

App engine groups the logs by request. You need to expand the log using the triangle/pointer on the left of the request in the 'new' GAE log viewer.

Personally I prefer using the old GAE log viewer, but I am unsure how much longer it will be around:

https://appengine.google.com/logs?app_id=s~xxx

(This viewer shows request + logs and allows log expansion)

Knavery

I tried your suggestion, unfortunately this doesn't help as the application log messages I'm looking for are simply missing (somehow being filtered out) from the log viewer. Expanding the log entries that I can access do not help me find the missing application log messages that I'm looking for. – Instruction

I am experiencing a similar issue, sometimes they are included but the majority of the time they are missing from the payload – Hector

Answer 2

An easy way to integrate Google Cloud Platform logging into your Python code is to create a subclass from logging.StreamHandler. This way logging levels will also match those of Google Cloud Logging, enabling you to filter based on severity. This solution also works within Cloud Run containers.

Also you can just add this handler to any existing logger configuration, without needing to change current logging code.

import json
import logging
import os
import sys
from logging import StreamHandler

from flask import request


class GoogleCloudHandler(StreamHandler):
    def __init__(self):
        StreamHandler.__init__(self)

    def emit(self, record):
        msg = self.format(record)
        # Get project_id from Cloud Run environment
        project = os.environ.get('GOOGLE_CLOUD_PROJECT')

        # Build structured log messages as an object.
        global_log_fields = {}
        trace_header = request.headers.get('X-Cloud-Trace-Context')

        if trace_header and project:
            trace = trace_header.split('/')
            global_log_fields['logging.googleapis.com/trace'] = (
                f"projects/{project}/traces/{trace[0]}")

        # Complete a structured log entry.
        entry = dict(severity=record.levelname, message=msg)
        print(json.dumps(entry))
        sys.stdout.flush()

A way to configure and use the handler could be:

def get_logger():
    logger = logging.getLogger(__name__)

    if not logger.handlers:
        gcp_handler = GoogleCloudHandler()
        gcp_handler.setLevel(logging.DEBUG)

        gcp_formatter = logging.Formatter(
            '%(levelname)s %(asctime)s [%(filename)s:%(funcName)s:%(lineno)d] %(message)s')
        gcp_handler.setFormatter(gcp_formatter)
        logger.addHandler(gcp_handler)
    return logger

Transport