Removing handlers from python's logging loggers

Question

I am playing with Python's logging system. I have noticed a strange behavior while removing handlers from a Logger object in a loop. Namely, my for loop removes all but one handler. Additional call to .removeHandler removes the last handler smoothly. No error messages are issued during the calls.

This is the test code:

import logging
import sys
logging.basicConfig()
dbg = logging.getLogger('dbg')
dbg.setLevel(logging.DEBUG)

testLogger = logging.getLogger('mylogger')
sh = logging.StreamHandler(sys.stdout)
fh = logging.FileHandler('mylogfile.log')
dbg.debug('before adding handlers: %d handlers'%len(testLogger.handlers))
testLogger.addHandler(fh)
testLogger.addHandler(sh)

dbg.debug('before removing. %d handlers: %s'%(len(testLogger.handlers), 
                                              str(testLogger.handlers)))
for h in testLogger.handlers:
    dbg.debug('removing handler %s'%str(h))
    testLogger.removeHandler(h)
    dbg.debug('%d more to go'%len(testLogger.handlers))

#HERE I EXPECT THAT NO HANDLER WILL REMAIN    
dbg.debug('after removing: %d handlers: %s'%(len(testLogger.handlers), 
                                              str(testLogger.handlers)))
if len(testLogger.handlers) > 0:
    #Why is this happening?
    testLogger.removeHandler(testLogger.handlers[0])
dbg.debug('after manually removing the last handler: %d handlers'%len(testLogger.handlers))    

I expect that at the end of the loop no handlers will remain in the testLogger object, however the last call to .removeHandler apparently fails, as can be seen from the output below. Nevertheless additional call to this function removes the handler as expected. Here is the output:

DEBUG:dbg:before adding handlers: 0 handlers
DEBUG:dbg:before removing. 2 handlers: [<logging.FileHandler instance at 0x021263F0>, <logging.StreamHandler instance at 0x021262B0>]
DEBUG:dbg:removing handler <logging.FileHandler instance at 0x021263F0>
DEBUG:dbg:1 more to go
DEBUG:dbg:after removing: 1 handlers: [<logging.StreamHandler instance at 0x021262B0>]
DEBUG:dbg:after manually removing the last handler: 0 handlers

More interestingly, if I replace the original loop with the following one, the loop works as expected and no handlers remain in the testLogger object at the end of the loop. Here is the modified loop:

while len(testLogger.handlers) > 0:
    h = testLogger.handlers[0]
    dbg.debug('removing handler %s'%str(h))
    testLogger.removeHandler(h)
    dbg.debug('%d more to go'%len(testLogger.handlers))

What explains this behaviour? Is this a bug or am I missing something?

Tradesman

for h in list(testLogger.handlers) – Moratorium

Answer 1

This isn't logger-specific behaviour. Never mutate (insert/remove elements) the list you're currently iterating on. If you need, make a copy. In this case testLogger.handlers.clear() should do the trick.

Gaylor

Solution is not so obvious and nice since we need to take into account thread safety which is obviously compromised here (take a look at addHandler implementation - locks are used there to modify handlers list). – Expiry

A safer alternative would be to modify the existing list: testLogger.handlers.clear() – Iliac

@Iliac I suppose testLogger.handlers.clear() calls the clear method of list type? Why is it safter than testLogger.handlers = []? – Caralie

Redefining a list rather than modifying the original list is a problem whenever you use an alias somewhere that points to the original list. Suppose: a = b = [1, 2, 3] a = [] Notice that b still points to [1, 2, 3]. In fairness, it is unlikely that you would ever use such an alias in this case, but it is good practice to maintain consistency so you don't get any unexpected surprises from cutting corners. – Iliac

Answer 2

instead of mutating undocumented .handler:

Option 1

logging.getLogger().removeHandler(logging.getLogger().handlers[0])

this way you remove exactly the preexisting handler object via offical api. Or to remove all handlers:

logger = logging.getLogger()
while logger.hasHandlers():
    logger.removeHandler(logger.handlers[0])

Option 2

logging.config.dictConfig(config={'level': logging.DEBUG, 'handlers': []})

Not only removes but prevents its creation. List root will have [] handlers

Chick

Option 1 does not work for non-root loggers: the method hasHandlers() also checks the handlers of the ancestor loggers, so if an ancestor logger has an handler then logger.hasHandlers() will always evaluate to True and therefore logger.handlers[0] will end up raising an IndexError. – Grados

Going off option 2, you can also just set your instantiated logger object's handlers attribute to an empty array, such as logger = logging.getLogger(__name__) and logger.handlers = [] – Phenix

Answer 3

If you don't want to delete them all (thanks for the tip @CatPlusPlus):

testLogger.handlers = [
    h for h in testLogger.handlers if not isinstance(h, logging.StreamHandler)]

Carthusian

Sorry,I found this didn't work.for isinstance can't used to judge handler's type – Cartogram

@Cartogram Why can't isinstance work for that purpose? I don't seem to have a problem using the suggestion in this answer myself. – Dunghill

I'm using python3 in Ubuntu 16.04. I use if (type(h) == logging.StreamHandler)] instead. – Cartogram

@Cartogram you should never do that because type doesn't check parent types (inheritance), isinstance does: https://mcmap.net/q/45193/-what-are-the-differences-between-type-and-isinstance-duplicate – Carthusian

For personnal purposes i wanted to check that a handker worked without spamming slack channel afterwards. handlers = logger.handlers; logger.error('Removing slack handler'); logger.handlers = [handler for handler in logger.handlers if handler != slack_handler] – Empressement

Answer 4

I would say if the intent for removing the logging handlers is to prevent sending logs to additional handlers (this is the case most of the times) then an alternative approach I follow is to set propagate to False for your current logger.

logger = logging.getLogger(__name__)
logger.propagate = False

That way your log messages will only go to the handlers you explicitly add and will not propagate to parent loggers.

Finespun

Answer 5

I've just found out that you can also do that within a logging .ini file, with the following block:

[logger_stpipe]
handlers=
propagate=1
qualname=stpipe

It basically deactivates all handlers for a given logger. But it's somewhat limited because you have to know the Logger's name in advance.

Thessalonian