Elegant setup of Python logging in Django
Asked Answered
N

4

111

I have yet to find a way of setting up Python logging with Django that I'm happy with. My requirements are fairly simple:

  • Different log handlers for different events - that is, I want to be able to log to different files
  • Easy access to loggers in my modules. The module should be able to find its logger with little effort.
  • Should be easily applicable to command-line modules. Parts of the system are stand-alone command line or daemon processes. Logging should be easily usable with these modules.

My current setup is to use a logging.conf file and setup logging in each module I log from. It doesn't feel right.

Do you have a logging setup that you like? Please detail it: how do you setup the configuration (do you use logging.conf or set it up in code), where/when do you initiate the loggers, and how do you get access to them in your modules, etc.

Noddy answered 21/10, 2009 at 5:7 Comment(2)
You might find the following screencast useful - ericholscher.com/blog/2008/aug/29/…. Also, better support for logging in Django has been proposed by Simon Willison (see simonwillison.net/2009/Sep/28/ponies).Dacoit
@Dominic Rodger - You can already do flexible logging of apps in Django, Simon's proposal mainly for facilitating logging in Django internals. There's work afoot in Python to add dictionary-based configuration to Python logging, from which Django may benefit.Remount
R
58

The best way I've found so far is to initialize logging setup in settings.py - nowhere else. You can either use a configuration file or do it programmatically step-by-step - it just depends on your requirements. The key thing is that I usually add the handlers I want to the root logger, using levels and sometimes logging.Filters to get the events I want to the appropriate files, console, syslogs etc. You can of course add handlers to any other loggers too, but there isn't commonly a need for this in my experience.

In each module, I define a logger using

logger = logging.getLogger(__name__)

and use that for logging events in the module (and, if I want to differentiate further) use a logger which is a child of the logger created above.

If my app is going to be potentially used in a site which doesn't configure logging in settings.py, I define a NullHandler somewhere as follows:

#someutils.py

class NullHandler(logging.Handler):
    def emit(self, record):
        pass

null_handler = NullHandler()

and ensure that an instance of it is added to all loggers created in the modules in my apps which use logging. (Note: NullHandler is already in the logging package for Python 3.1, and will be in Python 2.7.) So:

logger = logging.getLogger(__name__)
logger.addHandler(someutils.null_handler)

This is done to ensure that your modules play nicely in a site which doesn't configure logging in settings.py, and that you don't get any annoying "No handlers could be found for logger X.Y.Z" messages (which are warnings about potentially misconfigured logging).

Doing it this way meets your stated requirements:

  • You can set up different log handlers for different events, as you currently do.
  • Easy access to loggers in your modules - use getLogger(__name__).
  • Easily applicable to command-line modules - they also import settings.py.

Update: Note that as of version 1.3, Django now incorporates support for logging.

Remount answered 21/10, 2009 at 6:3 Comment(4)
Won't this require that every module have a handler defined in the config (you cannot use a handler for foo to handle foo.bar)? See the conversation we had years ago at groups.google.com/group/comp.lang.python/browse_thread/thread/…Earle
@andrew cooke: You can use a handler for foo to handle events logged to foo.bar. Re. that thread - both fileConfig and dictConfig now have options to prevent disabling old loggers. See this issue: bugs.python.org/issue3136, which came in a couple of months after your issue bugs.python.org/issue2697 - anyway, it's been sorted out since June 2008.Remount
wouldn't it be better to do logger = someutils.getLogger(__name__) where someutils.getLogger returns the logger from logging.getLogger with a null_handler already added?Economical
@Economical You don't need every logger having a NullHandler added - usually just the top level logger for your package hierarchy. So that would be overkill, IMO.Remount
B
130

I know this is a solved answer already, but as per django >= 1.3 there's a new logging setting.

Moving from old to new is not automatic, so I thought i'll write it down here.

And of course checkout the django doc for some more.

This is the basic conf, created by default with django-admin createproject v1.3 - mileage might change with latest django versions:

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'handlers': {
        'mail_admins': {
            'level': 'ERROR',
            'class': 'django.utils.log.AdminEmailHandler',
        }
    },
    'loggers': {
        'django.request': {
            'handlers': ['mail_admins'],
            'level': 'ERROR',
            'propagate': True,
        }
    }
}

This structure is based upon the standard Python logging dictConfig, that dictates the following blocks:

  • formatters - the corresponding value will be a dict in which each key is a formatter id and each value is a dict describing how to configure the corresponding Formatter instance.

  • filters - the corresponding value will be a dict in which each key is a filter id and each value is a dict describing how to configure the corresponding Filter instance.

  • handlers - the corresponding value will be a dict in which each key is a handler id and each value is a dict describing how to configure the corresponding Handler instance. Each handler has the following keys:

  • class (mandatory). This is the fully qualified name of the handler class.

  • level (optional). The level of the handler.

  • formatter (optional). The id of the formatter for this handler.

  • filters (optional). A list of ids of the filters for this handler.

I usually do at least this:

  • add a .log file
  • configure my apps to write to this log

Which translates into:

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'verbose': {
            'format': '%(levelname)s %(asctime)s %(module)s %(process)d %(thread)d %(message)s'
        },
        'simple': {
            'format': '%(levelname)s %(message)s'
        },
    },
    'filters': {
        'require_debug_false': {
            '()': 'django.utils.log.RequireDebugFalse'
        }
    },
    'handlers': {
        'null': {
            'level':'DEBUG',
            'class':'django.utils.log.NullHandler',
        },
        'console':{
            'level': 'DEBUG',
            'class': 'logging.StreamHandler',
            'formatter': 'simple'
        },
        # I always add this handler to facilitate separating loggings
        'log_file':{
            'level': 'DEBUG',
            'class': 'logging.handlers.RotatingFileHandler',
            'filename': os.path.join(VAR_ROOT, 'logs/django.log'),
            'maxBytes': '16777216', # 16megabytes
            'formatter': 'verbose'
        },
        'mail_admins': {
            'level': 'ERROR',
            'filters': ['require_debug_false'],
            'class': 'django.utils.log.AdminEmailHandler',
            'include_html': True,
        }
    },
    'loggers': {
        'django.request': {
            'handlers': ['mail_admins'],
            'level': 'ERROR',
            'propagate': True,
        },
        'apps': { # I keep all my of apps under 'apps' folder, but you can also add them one by one, and this depends on how your virtualenv/paths are set
            'handlers': ['log_file'],
            'level': 'INFO',
            'propagate': True,
        },
    },
    # you can also shortcut 'loggers' and just configure logging for EVERYTHING at once
    'root': {
        'handlers': ['console', 'mail_admins'],
        'level': 'INFO'
    },
}

edit

See request exceptions are now always logged and Ticket #16288:

I updated the above sample conf to explicitly include the correct filter for mail_admins so that, by default, emails are not sent when debug is True.

You should add a filter:

'filters': {
    'require_debug_false': {
        '()': 'django.utils.log.RequireDebugFalse'
    }
},

and apply it to the mail_admins handler:

    'mail_admins': {
        'level': 'ERROR',
        'filters': ['require_debug_false'],
        'class': 'django.utils.log.AdminEmailHandler',
        'include_html': True,
    }

Otherwise the django.core.handers.base.handle_uncaught_exception doesn't pass errors to the 'django.request' logger if settings.DEBUG is True.

If you don't do this in Django 1.5 you'll get a

DeprecationWarning: You have no filters defined on the 'mail_admins' logging handler: adding implicit debug-false-only filter

but things will still work correctly BOTH in django 1.4 and django 1.5.

** end edit **

That conf is strongly inspired by the sample conf in the django doc, but adding the log file part.

I often also do the following:

LOG_LEVEL = 'DEBUG' if DEBUG else 'INFO'

...
    'level': LOG_LEVEL
...

Then in my python code I always add a NullHandler in case no logging conf is defined whatsoever. This avoid warnings for no Handler specified. Especially useful for libs that are not necessarily called only in Django (ref)

import logging
# Get an instance of a logger
logger = logging.getLogger(__name__)
class NullHandler(logging.Handler): #exists in python 3.1
    def emit(self, record):
        pass
nullhandler = logger.addHandler(NullHandler())

# here you can also add some local logger should you want: to stdout with streamhandler, or to a local file...

[...]

logger.warning('etc.etc.')
Burglar answered 27/4, 2011 at 15:50 Comment(7)
Parand, it definitely is (IMHO!) worth stepping up to django 1.3 , though there are a few points to take care of for a smooth transition - open a new SO question if you get in trouble ;-)Burglar
by the way: I still use this kind of settings and the file log, but I moved to sentry for the production!Burglar
@clime well I tried to explain it in the answer itself: in case no logging conf is defined whatsoever. This avoid warnings for no Handler specified. Especially useful for libs that are not necessarily called only in Django (ref)Burglar
I don't see how you use this definition: 'null': { 'level':'DEBUG', 'class':'django.utils.log.NullHandler', }Lundy
@clime: that comes straight from python docs: docs.python.org/2/howto/… and 'null' is just a chosen name for a kind of handler that you can add eg. 'django': { 'handlers':['null'], 'propagate': True, 'level':'INFO', }, but of course your mileage and configuration might vary. Take that as a generic example to customize (yes I might have added this handler to make the example more complete)Burglar
My point is that with that logger you would not need that: nullhandler = logger.addHandler(NullHandler()) but probly I am wrong here. I don't really understand to this null stuff.Lundy
Django 3+: logging.NullHandler instead of django.utils.log.NullHandler and also maxBytes must be integer not strCazzie
R
58

The best way I've found so far is to initialize logging setup in settings.py - nowhere else. You can either use a configuration file or do it programmatically step-by-step - it just depends on your requirements. The key thing is that I usually add the handlers I want to the root logger, using levels and sometimes logging.Filters to get the events I want to the appropriate files, console, syslogs etc. You can of course add handlers to any other loggers too, but there isn't commonly a need for this in my experience.

In each module, I define a logger using

logger = logging.getLogger(__name__)

and use that for logging events in the module (and, if I want to differentiate further) use a logger which is a child of the logger created above.

If my app is going to be potentially used in a site which doesn't configure logging in settings.py, I define a NullHandler somewhere as follows:

#someutils.py

class NullHandler(logging.Handler):
    def emit(self, record):
        pass

null_handler = NullHandler()

and ensure that an instance of it is added to all loggers created in the modules in my apps which use logging. (Note: NullHandler is already in the logging package for Python 3.1, and will be in Python 2.7.) So:

logger = logging.getLogger(__name__)
logger.addHandler(someutils.null_handler)

This is done to ensure that your modules play nicely in a site which doesn't configure logging in settings.py, and that you don't get any annoying "No handlers could be found for logger X.Y.Z" messages (which are warnings about potentially misconfigured logging).

Doing it this way meets your stated requirements:

  • You can set up different log handlers for different events, as you currently do.
  • Easy access to loggers in your modules - use getLogger(__name__).
  • Easily applicable to command-line modules - they also import settings.py.

Update: Note that as of version 1.3, Django now incorporates support for logging.

Remount answered 21/10, 2009 at 6:3 Comment(4)
Won't this require that every module have a handler defined in the config (you cannot use a handler for foo to handle foo.bar)? See the conversation we had years ago at groups.google.com/group/comp.lang.python/browse_thread/thread/…Earle
@andrew cooke: You can use a handler for foo to handle events logged to foo.bar. Re. that thread - both fileConfig and dictConfig now have options to prevent disabling old loggers. See this issue: bugs.python.org/issue3136, which came in a couple of months after your issue bugs.python.org/issue2697 - anyway, it's been sorted out since June 2008.Remount
wouldn't it be better to do logger = someutils.getLogger(__name__) where someutils.getLogger returns the logger from logging.getLogger with a null_handler already added?Economical
@Economical You don't need every logger having a NullHandler added - usually just the top level logger for your package hierarchy. So that would be overkill, IMO.Remount
B
9

We initialize logging in the top-level urls.py by using a logging.ini file.

The location of the logging.ini is provided in settings.py, but that's all.

Each module then does

logger = logging.getLogger(__name__)

To distinguish testing, development and production instances, we have different logging.ini files. For the most part, we have a "console log" that goes to stderr with Errors only. We have an "application log" that uses a regular rolling log file that goes to a logs directory.

Barn answered 21/10, 2009 at 10:12 Comment(5)
I ended up using this, except initializing in settings.py instead of urls.pyNoddy
How do you use settings from settings.py in your logging.ini file? For example, I need the BASE_DIR setting, so I can tell it where to store my log files.Disarrange
@slypete: We don't use settings in the logging.ini. Since the logging is largely independent, we don't use any of the Django settings. Yes, there's a possibility of repeating something. No, it doesn't make much practical difference.Barn
In that case, I would a separate logging.ini file in every installation of my app.Disarrange
@slypete: You have a settings.py for each installation. You also have a logging.ini for each installation. Plus, you probably have an Apache conf file for each installation, also. Plus a wsgi interface file. I'm not sure what your point is.Barn
C
6

I am currently using a logging system, which I created myself. It uses CSV format for logging.

django-csvlog

This project still doesn't have full documentation, but I am working on it.

Carden answered 21/10, 2009 at 5:33 Comment(2)
Link mentioned in the answer is brokenAilbert
Link not working, please updateFootslog

© 2022 - 2024 — McMap. All rights reserved.