Use fileConfig to configure custom handlers in Python

Question

I'm using a config file to configure my logger in a Python application. This is the file:

[loggers]
keys=root

[logger_root]
level=INFO
handlers=console

[handlers]
keys=console,file_rotating

[handler_console]
class=StreamHandler
level=WARNING
formatter=console
args=(sys.stderr,)

[handler_file_rotating]
class=TimeRotatingFileHandler
level=DEBUG
formatter=file
args=('../logs/twicker.log', 'd', 1, 5)

[formatters]
keys=console,file

[formatter_console]
format=%(levelname)s - %(message)s

[formatter_file]
format=%(asctime)s - %(levelname)s - %(module)s - %(message)s

My problem is with TimeRotatingFileHandler. Everytime I run the app I get the next error:

ImportError: No module named 'TimeRotatingFileHandler'

What I'm doing wrong? I tried also changing class line to class=handlers.TimeRotatingFileHandler but in that case I get the next error:

ImportError: No module named 'handlers'

Lucey

It would be appreciated if you could choose the correct answer... – Lilylivered

Answer 1

I ran into the same problem when using dictConfig The solution for me was to fully qualify the module path like this:

[handler_file_rotating]
class=logging.handlers.TimeRotatingFileHandler
level=DEBUG
formatter=file
args=('../logs/twicker.log', 'd', 1, 5)

You might want to give that a try

Lilylivered

Answer 2

The class= is evaluated in the namespace of the logging module, and by default this does not have a binding to handlers. So you could do

import logging, logging.handlers
logging.handlers = logging.handlers

before calling fileConfig(), and then class=handlers.TimedRotatingHandler should work.

Mimamsa

Answer 3

To extend on this a little if you decide to make a custom handler all you need to do is define that handler near the top of your code and then define it as an object in logging.handlers.

Example:

class MyCustomRotatingClass(logging.handlers.RotatingFileHandler):
   """custom handler that queues messages
      to be uploaded in batches to the portal
      in a background thread
   """
   def __init__(self, basedir, mode='a', maxBytes=0, backupCount=0, encoding=None, delay=0):
       logging.handlers.RotatingFileHandler.__init__(self, 'TestSavvyExecute.log','a',250000,40,'utf-8',0)
       self.maxBytes = maxBytes

   def emit(self, record):
       try:
           if logging.handlers.RotatingFileHandler.shouldRollover(self, record):
               logging.handlers.RotatingFileHandler.doRollover(self)

           #ASCII characters use 1 byte each
           if len(record.msg) > self.maxBytes:
               oldMsg = record.msg

               record.msg = record.msg[0:self.maxBytes]
               logging.FileHandler.emit(self, record)

               record.msg = oldMsg[self.maxBytes + 1:]
               self.emit(record)
           else:
              logging.FileHandler.emit(self, record)
       except (KeyboardInterrupt, SystemExit):
           raise
       except:
           logging.handlers.RotatingFileHandler.handleError(self, record)          

logging.handlers.MyCustomRotatingClass = MyCustomRotatingClass
logging.config.fileConfig('translator.log.config')

Now you can easily reference it in your config file.

[handler_rollinglog]
class=handlers.MyCustomRotatingClass

Vickivickie

I tried to use your example above this BufferingSMTPHandler but it doesn't seem to send email when I switch it to fileConfig. Any ideas? gist.github.com/anonymous/1379446 – Salvo