Logging, remove, inspect, and modify handlers configured by fileConfig()

Question

How can I remove, inspect, and modify handlers configured for my loggers using the fileConfig() function?

For removing, there is Logger.removeHandler(hdlr) method, but how do I get the handler in first place if it was configured from file?

Anticipatory

Answer 1

logger.handlers contains a list with all handlers of a logger.

Arterial

This is not documented. Is it safe to use it? – Dockyard

@Manish: Looking at the code, I would say it is safe to use, as long as you don't do multithreading. The handler list is a shared resource protected by a Lock and the list only contains references to the handlers. Don't use it, if you do multithreading. – Platter

This only contains standard handlers, what if you wrote your own? Also, calling it a list is misleading, I was looking for a list of instances and there was none. It's just a collection of classes. – Fibre

It's worth to note that logger.handlers doesn't include handlers derived from parent logger. All loggers, except RootLogger, has its parent logger which have its own handlers list. – Hamill

Caveat: logger.handlers might get modified by packages like pytest (at least if logger is the root logger, i.e. logger = logging.getLogger()). So if you want testable code (which behaves in tests the same way it does in production), I'd rather avoid accessing logger.handlers which is essentially global state. It really is beyond me why the Python developers decided to implement logging through global state instead of doing proper dependency injection. – Cowlick

Answer 2

This code will print all the loggers and for each logger its handlers

for k,v in  logging.Logger.manager.loggerDict.items()  :
        print('+ [%s] {%s} ' % (str.ljust( k, 20)  , str(v.__class__)[8:-2]) )
        if not isinstance(v, logging.PlaceHolder):
            for h in v.handlers:
                print('     +++',str(h.__class__)[8:-2] )

This will print out the loggers and handlers in your system including their states and levels.

This will help you debug your logging issues

output:
+ [root                ] {logging.RootLogger} {DEBUG}
-------------------------
   -name=root
   -handlers=[<logging.FileHandler object at 0x7fc599585390>, <logging.StreamHandler object at 0x7fc599585550>]
   -filters=[]
   -propagate=True
   -level=10
   -disabled=False
   -parent=None
     +++logging.FileHandler {NOTSET}
   -stream=<_io.TextIOWrapper name='/dev/logs/myapp.log' mode='w' encoding='UTF-8'>
   -mode=w
   -filters=[]
   -encoding=None
   -baseFilename=/home/dev/logs/myapp.log
   -level=0
   -lock=<unlocked _thread.RLock object owner=0 count=0 at 0x7fc5a85a4240>
   -delay=False
   -_name=None
   -formatter=<logging.Formatter object at 0x7fc599585358>
     +++logging.StreamHandler {DEBUG}
   -lock=<unlocked _thread.RLock object owner=0 count=0 at 0x7fc5a85a4210>
   -filters=[]
   -stream=<ipykernel.iostream.OutStream object at 0x7fc5aa6abb00>
   -level=10
   -_name=None
   -formatter=<logging.Formatter object at 0x7fc5995853c8>
+ [PathFinder          ] {logging.Logger} {NOTSET}
-------------------------
   -name=PathFinder
   -handlers=[]
   -filters=[]
   -manager=<logging.Manager object at 0x7fc5b09757f0>
   -propagate=True
   -level=0
   -disabled=False
   -parent=<logging.RootLogger object at 0x7fc5b09757b8>

Come

This is amazing and exactly the kind of thing I could not find anywhere else. – Dissatisfy

thanks @GregMueller i am working on a pip installed helper module that will offer other insights into the logging stack – Come

@jens yeah. Haven't committed in a while – Come

For those that want to see Mickey's debugger on github: github.com/mickeyperlstein/logging_debugger/blob/master/… – Addend

Answer 3

The answer by Mickey-Perlstein was just what I was looking for, but the listing appears to come from a much more complete version than the code provided.

This code is kind of even cruder, but crucially for my use and understanding, includes the root logger.

import logging

def listloggers():
    rootlogger = logging.getLogger()
    print(rootlogger)
    for h in rootlogger.handlers:
        print('     %s' % h)

    for nm, lgr in logging.Logger.manager.loggerDict.items():
        print('+ [%-20s] %s ' % (nm, lgr))
        if not isinstance(lgr, logging.PlaceHolder):
            for h in lgr.handlers:
                print('     %s' % h)

Output:

<RootLogger root (DEBUG)>
     <TimedRotatingFileHandler /path/to/myapp.log (DEBUG)>
     <StreamHandler <stdout> (DEBUG)>
+ [concurrent.futures  ] <Logger concurrent.futures (DEBUG)>
+ [concurrent          ] <logging.PlaceHolder object at 0x7f72f624eba8>
+ [asyncio             ] <Logger asyncio (DEBUG)>
+ [myapp               ] <Logger myapp (DEBUG)>
+ [flask.app           ] <Logger flask.app (DEBUG)>
+ [flask               ] <Logger flask (DEBUG)>
+ [werkzeug            ] <Logger werkzeug (ERROR)>

Philoprogenitive

I like your concise detailing, great "fork" – Come

This also listed the FileHandler which I defined in a config file on the RootLogger (Mickey's code did not.) – Magdalenmagdalena

Answer 4

Another approach might be to use a JSON or YAML config file which gets loaded into a dictionary which you can then view/manipulate before it's passed to the logger.config.

import yaml
import logging.config

with open (LOG_CONFIG, 'rt') as f:
   config=yaml.safe_load(f)
   config['handlers']['error_file_handler']['filename']='foo'
logging.config.dictConfig(config)

Vinny

Answer 5

Re removing handlers - if it helps anyone, I wanted to remove all handlers for all the logs I'd created (I was learning about Python logging) and came up with this:

def log_close():
    # Get all loggers
    loggers = [logging.getLogger(name) if 'your_app_name' in name else None for name in logging.root.manager.loggerDict]

    # For each valid logger remove all handlers
    for log in loggers:
        if log != None:
            while bool(len(log.handlers)):
                for handler in log.handlers:
                    log.removeHandler(handler)
# Set up log
log = logging.getLogger('your_app_name')
log.setLevel('DEBUG')
log_format = '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
date_format = '%Y-%m-%d %H:%M:%S'
formatter = logging.Formatter(log_format, date_format)
log_filename = 'your_app_name.log'
log_write_mode = 'w+'
file_handler = logging.FileHandler(log_filename, log_write_mode)
file_handler.setFormatter(formatter)
log.addHandler(file_handler)
console_handler = logging.StreamHandler()
console_handler.setFormatter(formatter)
log.addHandler(console_handler)

Azelea