Can structured logging be done with Pythons standard library?

Question

I recently read about structured logging (here). The idea seems to be to log not by appending simple strings as a line to a logfile, but instead JSON objects. This makes it possible to analyze the logfile by automatic tools.

Can Pythons logging library do structured logging? If not, is there a "mainstream" solution for it (e.g. like numpy/scipy is the mainstream solution for scientific calculations)? I found structlog, but I'm not sure how widespread it is.

Reifel

I actually don't remember why I didn't like structlog. I think the main point was that it is a bit different from the standard logging library and thus might rather lock-in than the other solutions below. – Reifel

Update: Ran into a problem with structlog - a 3rd party project was configuring handlers already. with structlog, I could not remove them. With the pythonjsonlogger it was not problem. – Reifel

Answer 1

If you install python-json-logger (288 stars, 70 forks) and have a logging configuration (YAML) like the following, you will get a structured logging file.

version: 1
formatters:
    detailed:
        class: logging.Formatter
        format: '[%(asctime)s]:[%(levelname)s]: %(message)s'
    json:
        class: pythonjsonlogger.jsonlogger.JsonFormatter
        format: '%(asctime)s %(levelname)s %(message)s'
handlers:
    console:
        class: logging.StreamHandler
        level: INFO
        formatter: detailed
    file:
        class: logging.FileHandler
        filename: logfile.log
        level: DEBUG
        formatter: json
root:
    level: DEBUG
    handlers:
        - console
        - file

Exceptions

You might also want to make exceptions / tracebacks use the structured format.

See Can I make Python output exceptions in one line / via logging?

Reifel

Why do we need to write tone of codes or edit existing code if this solution is simple and easy to use. – Middleoftheroader

Answer 2

Have you looked at python docs site section describing Implementing structured logging that explain how python built-in logger can be utilized for structured logging?

Below is a simple example as listed on above site .

import json
import logging

class StructuredMessage(object):
    def __init__(self, message, **kwargs):
        self.message = message
        self.kwargs = kwargs

    def __str__(self):
        return '%s >>> %s' % (self.message, json.dumps(self.kwargs))

m = StructuredMessage   # optional, to improve readability

logging.basicConfig(level=logging.INFO, format='%(message)s')
logging.info(m('message 1', foo='bar', bar='baz', num=123, fnum=123.456))

Which results in following log.

message 1 >>> {"fnum": 123.456, "num": 123, "bar": "baz", "foo": "bar"}

Hope this helps.

Nickell

Using _ seems very unsafe to me as it is the standard variable if you need a variable name, but don't intend to do it (e.g. for unpacking foo, _, bar = ('interesting', 'boring', 'important') – Reifel

@MartinThomas, I agree, However in my defense, I simply presented sample example at python docs. I've now modified answer and replaced _ with variable m. Hope this helps. – Nickell

py3 equivalent link – Chaldea

Answer 3

As of py3.2, it's possible to do this with the standard library, no external dependencies required:

from datetime import datetime
import json
import logging
import traceback


APP_NAME = 'hello world json logging'
APP_VERSION = 'git rev-parse HEAD'
LOG_LEVEL = logging._nameToLevel['INFO']


class JsonEncoderStrFallback(json.JSONEncoder):
  def default(self, obj):
    try:
      return super().default(obj)
    except TypeError as exc:
      if 'not JSON serializable' in str(exc):
        return str(obj)
      raise


class JsonEncoderDatetime(JsonEncoderStrFallback):
  def default(self, obj):
    if isinstance(obj, datetime):
      return obj.strftime('%Y-%m-%dT%H:%M:%S%z')
    else:
      return super().default(obj)


logging.basicConfig(
  format='%(json_formatted)s',
  level=LOG_LEVEL,
  handlers=[
    # if you wish to also log to a file:
    # logging.FileHandler(log_file_path, 'a'),
    logging.StreamHandler(sys.stdout),
  ],
)


_record_factory_bak = logging.getLogRecordFactory()
def record_factory(*args, **kwargs) -> logging.LogRecord:
  record = _record_factory_bak(*args, **kwargs)
  
  record.json_formatted = json.dumps(
    {
      'level': record.levelname,
      'unixtime': record.created,
      'thread': record.thread,
      'location': '{}:{}:{}'.format(
        record.pathname or record.filename,
        record.funcName,
        record.lineno,
      ),
      'exception': record.exc_info,
      'traceback': (
        traceback.format_exception(*record.exc_info)
        if record.exc_info
        else None
      ),
      'app': {
        'name': APP_NAME,
        'releaseId': APP_VERSION,
        'message': record.getMessage(),
      },
    },
    cls=JsonEncoderDatetime,
  )

  # clear exc data since it is included in the json format
  # without clearing this, logging.exception will print the
  # traceback across multiple lines, which is not json formatted
  record.exc_info = None
  record.exc_text = None

  return record
logging.setLogRecordFactory(record_factory)

Calling logging.info('HELLO %s', 'WORLD') ...

... results in {"level": "INFO", "unixtime": 1623532882.421775, "thread": 4660305408, "location": "<ipython-input-3-abe3276ceab4>:<module>:1", "exception": null, "traceback": null, "app": {"name": "hello world json logging", "releaseId": "git rev-parse HEAD", "message": "HELLO WORLD"}}

Hallock