How do I change the default format of log messages in python app engine?

Question

I would like to log the module and classname by default in log messages from my request handlers.

The usual way to do this seems to be to set a custom format string by calling logging.basicConfig, but this can only be called once and has already been called by the time my code runs.

Another method is to create a new log Handler which can be passed a new log Formatter, but this doesn't seem right as I want to use the existing log handler that App Engine has installed.

What is the right way to have extra information added to all log messages in python App Engine, but otherwise use the existing log format and sink?

Crossness

Answer 1

I cooked this up by reading the logging module's __init__.py. I don't know if this is proper, but it seems to work:

import logging

logging.basicConfig(level=logging.DEBUG,
                    format='%(asctime)s %(name)-12s %(levelname)-8s %(message)s',
                    datefmt='%m-%d %H:%M',
                    )

logging.info('Danger Will Robinson!')
# 03-31 20:00 root         INFO     Danger Will Robinson!
root = logging.getLogger()
hdlr = root.handlers[0]
fmt = logging.Formatter('%(name)-12s: %(levelname)-8s %(message)s')
hdlr.setFormatter(fmt)
logging.info('Danger Will Robinson!')
# root        : INFO     Danger Will Robinson!

Holocaust

I noticed, that root.handlers[0] doesn't work, if basicConfig() was NOT called before. Why is that? – Cythiacyto

logging.basicConfig is a convenience function which sets up a root handler. If you don't define a handler then root.handlers is an empty list. So trying to access the first element of an empty list, e.g. root.handlers[0], raises an IndexError. Whenever you use the logging module you must either call logging.basicConfig or read a config from a file (e.g. logging.config.fileConfig) or set up the handler(s) yourself. – Holocaust

Answer 2

I found this to be working for Python 3.6, it will set the logging level / format for all subsequent logging calls, even if logging is called by previous imports.

logging_level = logging.INFO
logging_fmt = "%(levelname)s:%(name)s:%(message)s"   # the default
try:
    root_logger = logging.getLogger()
    root_logger.setLevel(logging_level)
    root_handler = root_logger.handlers[0]
    root_handler.setFormatter(logging.Formatter(logging_fmt))
except IndexError:
    logging.basicConfig(level=logging_level, format=logging_fmt)

Craft

Is this the most compact / elegant it can get with current versions of python 3? – Makedamakefast

@matanster unfortunately I am not expert enough to answer that, perhaps others can comment on this – Craft

@matanster as of Python 3.2, logger objects have hasHandler() method. This can help to get rid of the try - except. – Mauramauralia