How to log everything into a file using RotatingFileHandler by using logging.conf file?

Question

I am trying to use RotatingHandler for our logging purpose in Python. I have kept backup files as 500 which means it will create maximum of 500 files I guess and the size that I have set is 2000 Bytes (not sure what is the recommended size limit is).

If I run my below code, it doesn't log everything into a file. I want to log everything into a file -

#!/usr/bin/python

import logging
import logging.handlers

LOG_FILENAME = 'testing.log'

# Set up a specific logger with our desired output level
my_logger = logging.getLogger('agentlogger')

# Add the log message handler to the logger
handler = logging.handlers.RotatingFileHandler(LOG_FILENAME, maxBytes=2000, backupCount=100)

# create a logging format
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)

my_logger.addHandler(handler)

my_logger.debug('debug message')
my_logger.info('info message')
my_logger.warn('warn message')
my_logger.error('error message')
my_logger.critical('critical message')

# Log some messages
for i in range(10):
    my_logger.error('i = %d' % i)

This is what gets printed out in my testing.log file -

2013-11-22 12:59:34,782 - agentlogger - WARNING - warn message
2013-11-22 12:59:34,782 - agentlogger - ERROR - error message
2013-11-22 12:59:34,782 - agentlogger - CRITICAL - critical message
2013-11-22 12:59:34,782 - agentlogger - ERROR - i = 0
2013-11-22 12:59:34,782 - agentlogger - ERROR - i = 1
2013-11-22 12:59:34,783 - agentlogger - ERROR - i = 2
2013-11-22 12:59:34,783 - agentlogger - ERROR - i = 3
2013-11-22 12:59:34,783 - agentlogger - ERROR - i = 4
2013-11-22 12:59:34,783 - agentlogger - ERROR - i = 5
2013-11-22 12:59:34,783 - agentlogger - ERROR - i = 6
2013-11-22 12:59:34,784 - agentlogger - ERROR - i = 7
2013-11-22 12:59:34,784 - agentlogger - ERROR - i = 8
2013-11-22 12:59:34,784 - agentlogger - ERROR - i = 9

It doesn't print out INFO, DEBUG message into the file somehow.. Any thoughts why it is not working out?

And also, right now, I have defined everything in this python file for logging purpose. I want to define above things in the logging conf file and read it using the fileConfig() function. I am not sure how to use the RotatingFileHandler example in the logging.conf file?

UPDATE:-

Below is my updated Python code that I have modified to use with log.conf file -

#!/usr/bin/python

import logging
import logging.handlers

my_logger = logging.getLogger(' ')
my_logger.config.fileConfig('log.conf')

my_logger.debug('debug message')
my_logger.info('info message')
my_logger.warn('warn message')
my_logger.error('error message')
my_logger.critical('critical message')

# Log some messages
for i in range(10):
    my_logger.error('i = %d' % i)

And below is my log.conf file -

[loggers]
keys=root

[handlers]
keys=logfile

[formatters]
keys=logfileformatter

[logger_root]
level=DEBUG
handlers=logfile

[logger_zkagentlogger]
level=DEBUG
handlers=logfile
qualname=zkagentlogger
propagate=0

[formatter_logfileformatter]
format=%(asctime)s %(name)-12s: %(levelname)s %(message)s

[handler_logfile]
class=handlers.RotatingFileHandler
level=NOTSET
args=('testing.log',2000,100)
formatter=logfileformatter

But whenever I compile it, this is the error I got on my console -

$ python logtest3.py
Traceback (most recent call last):
  File "logtest3.py", line 6, in <module>
    my_logger.config.fileConfig('log.conf')
AttributeError: 'Logger' object has no attribute 'config'

Any idea what wrong I am doing here?

Kinna

Answer 1

It doesn't print out INFO, DEBUG message into the file somehow.. Any thoughts why it is not working out?

you don't seem to set a loglevel, so the default (warning) is used

from http://docs.python.org/2/library/logging.html :

Note that the root logger is created with level WARNING.

as for your second question, something like this should do the trick (I haven't tested it, just adapted from my config which is using the TimedRotatingFileHandler):

[loggers]
keys=root

[handlers]
keys=logfile

[formatters]
keys=logfileformatter

[logger_root]
level=DEBUG
handlers=logfile

[formatter_logfileformatter]
format=%(asctime)s %(name)-12s: %(levelname)s %(message)s

[handler_logfile]
class=handlers.RotatingFileHandler
level=NOTSET
args=('testing.log','a',2000,100)
formatter=logfileformatter

Fung

Thanks.. So what I am suppsoed to set to log everything into a file? – Kinna

I've added a few exerpts from a working config I use in a project, this should get you started. A full example with more loggers can be found github.com/gryphius/fuglu/blob/master/fuglu/conf/… – Fung

Thanks Gryphius for the help.. One last doubt, then how I am going to get the logger in the python code? Any thoughts? This is first time I am working with Python so having some problem. – Kinna

you get the loggers the same way you did in your code (logging.getLogger). see this example – Fung

Sorry for the delay.. I tried your suggestion by looking at your code but somehow my code gives compilation error whenever I am trying to run it... I have updated my question with more detail, not sure what wrong I am doing here.. – Kinna

don't call fileconfig on your logger... call it on the logging.config module – Fung

let us continue this discussion in chat – Kinna

:something like this my_logger = logging.getLogger(' ') my_logger.config('log.conf'). I tried this as well but same error. I am definitely missing some minor things... – Kinna

I think there's a slight error in the args line: it needs to include the file mode: args=('testing.log','a',2000,100) – Adverse

Answer 2

I know, it is very late ,but I just got same error, and while searching that are I got your problem. I am able to resolve my problem, and I thought it might be helpful for some other user also :

you have created a logger object and trying to access my_logger.config.fileConfig('log.conf') which is wrong you should use logger.config.fileConfig('log.conf') as I mention below and need to import logging.config as well :

#!/usr/bin/python

import logging
import logging.handlers
import logging.config

logging.config.fileConfig('log.config',disable_existing_loggers=0)
my_logger = logging.getLogger('you logger name as you mention in your conf file')

my_logger.debug('debug message')
my_logger.info('info message')
my_logger.warn('warn message')
my_logger.error('error message')
my_logger.critical('critical message')

after doing these changes, AttributeError: 'Logger' object has no attribute 'config' error must be gone.

Defraud