is there any way to modify the current logging to json format with few other fields added python logging

Question

I have many application code written in python django and every application is using standard python logger module and just a simple message strings are getting logged.

Now is there any way where I can add my own customer middleware or django app where when ever the logging is called in their methods it should reach my function and I will add more value to the logs, make a proper json format and then write to the log file. So that in current application developer don't need to do lots of changes except that he need to add my django app or a middle ware

Parish

Answer 1

You can use json_log_formatter. You can install it by:

pip install JSON-log-formatter==0.2.0

In django, you need to put update your logs settings like this:

LOGGING = {
    ...
    'formatters': {
        "json": {
            '()': 'json_log_formatter.JSONFormatter',
        }
    },
    ...
}

And use it in your code like this:

import logging

logger_name = 'some_name'
logger = logging.getLogger(logger_name)

logger.info('Sign up', extra={'referral_code': '52d6ce'})

Here the extra parameter sent through here will be rendered in the log like this(from documentation):

{
    "message": "Sign up",
    "time": "2015-09-01T06:06:26.524448",
    "referral_code": "52d6ce"
}

Override Json Log Formatter

You can override json_log_formatter.JSONFormatter class to add extra info like IP address if needed. Like this:

import json_log_formatter


class CustomJsonFormatter(json_log_formatter.JSONFormatter):

    def json_record(self, message, extra, record):
        request = extra.pop('request', None)
        if request:
            extra['IP_ADDRESS'] = request.META.get('HTTP_X_FORWARDED_FOR')  # or other ways to get ip
        return super(CustomJsonFormatter, self).json_record(message, extra, record)

# usage
logger.info('Sign up', extra={'referral_code': '52d6ce', 'request': request })  # django request object
# Django Settings
     'formatters': {
        "json": {
            '()': 'path.to.CustomJsonFormatter',
        }

Rhetic

This is really good info, here is it possible to override the extra? because I need to write my own utility to gather the IP address of the machine where this code is running and few other parameters related to infra part and then log those as well so that when its running behind load balancer it will be easy to fine which machine generated this log. But user should not add this in his code I need to provide a general framework kind of thing – Parish

I have added a way for overriding the JSONFormatter, hope it helps – Rhetic

Is there any way where we can integrate this framework github.com/thangbn/json-logging-python into the django ? I don't want correlation ID etc.. Just the standard logger I will use . I am not sure how to included this into the LOGGER in the settings.py – Parish

Sorry, i havn't used this library before – Rhetic

This is logging all logs except unhandled exceptions of code. How can we log such unhandled exceptions? @Rhetic – Bouncer