How to separate log handlers in Python

Question

I have a situation where I want to create two separate logger objects in Python, each with their own independent handler. By "separate," I mean that I want to be able to pass a log statement to each object independently, without contaminating the other log.

main.py

import logging
from my_other_logger import init_other_logger

logger = logging.getLogger(__name__)
logging.basicConfig(level=logging.INFO, handlers=[logging.StreamHandler(sys.stdout)])

other_logger = init_other_logger(__name__)

logger.info('Hello World') # Don't want to see this in the other logger
other_logger.info('Goodbye World') # Don't want to see this in the first logger

my_other_logger.py

import logging
import os, sys

def init_other_logger(namespace):
    logger = logging.getLogger(namespace)
    logger.setLevel(logging.DEBUG)
    fh = logging.FileHandler(LOG_FILE_PATH)
    logger.addHandler(fh)
    formatter = logging.Formatter('%(levelname)s:%(name)s:%(message)s')
    fh.setFormatter(formatter)
    #logger.propagate = False
    return logger

The only configuration I've been able to identify as useful here is the logger.propagate property. Running the code above as-is pipes all log statements to both the log stream and the log file. When I have logger.propagate = False nothing is piped to the log stream, and both log objects again pipe their output to the log file.

How can I create one log object that sends logs to only one handler, and another log object that sends logs to another handler?

Kaiak

You might be looking for a LoggerAdapter to add context info – Sinaloa

Answer 1

Firstly, let's see what's going on before we can head over to the solution.

logger = logging.getLogger(__name__) : when you're doing this you're getting or creating a logger with the name 'main'. Since this is the first call, it will create that logger.

other_logger = init_other_logger(__name__) : when you're doing this, again, you're getting or creating a logger with the name 'main'. Since this is the second call, it will fetch the logger created above. So you're not really instantiating a new logger, but you're getting a reference to the same logger created above. You can check this by doing a print after you call init_other_logger of the form: print(logger is other_logger).

What happens next is you add a FileHandler and a Formatter to the 'main' logger (inside the init_other_logger function), and you invoke 2 log calls via the method info(). But you're doing it with the same logger.

So this:

logger.info('Hello World')
other_logger.info('Goodbye World')

is essentially the same thing as this:

logger.info('Hello World')
logger.info('Goodbye World')

Now it's not so surprising anymore that both loggers output to both the file and stream.

---

Solution

So the obvious thing to do is to call your init_other_logger with another name.

I would recommend against the solution the other answer proposes because that's NOT how things should be done when you need an independent logger. The documentation has it nicely put that you should NEVER instantiate a logger directly, but always via the function getLogger of the logging module.

As we discovered above when you do a call of logging.getLogger(logger_name) it's either getting or creating a logger with logger_name. So this works perfectly fine when you want a unique logger as well. However remember this function is idemptotent meaning it will only create a logger with a given name the first time you call it and it will return that logger if you call it with the same name no matter how many times you'll call it after.

So, for example:

• a first call of the form logging.getLogger('the_rock') - creates your unique logger

• a second call of the form logging.getLogger('the_rock') - fetches the above logger

You can see that this is particularly useful if you, for instance:

• Have a logger configured with Formatters and Filters somewhere in your project, for instance in project_root/main_package/__init__.py.
• Want to use that logger somewhere in a secondary package which sits in project_root/secondary_package/__init__.py.

In secondary_package/__init__.py you could do a simple call of the form: logger = logging.getLogger('main_package') and you'll use that logger with all its bells and whistles.

---

Attention!

Even if you, at this point, will use your init_other_logger function to create a unique logger it would still output to both the file and the console. Replace this line other_logger = init_other_logger(__name__) with other_logger = init_other_logger('the_rock') to create a unique logger and run the code again. You will still see the output written to both the console and the file.

Why ?

Because it will use both the FileHandler and the StreamHandler.

Why ?

Because the way the logging machinery works. Your logger will emit its message via its handlers, then it will propagate all the way up to the root logger where it will use the StreamHandler which you attached via the basicConfig call. So the propagate property you discovered is actually what you want in your case, because you're creating a custom logger, which you'd want to emit messages only via its manually attached handlers and not emit any further. Uncomment the logger.propagate = False after creating the unique logger and you'll see that everything works as expected.

Bestrew

Answer 2

Both of your handlers are installed on the same logger. This is why they aren't seperate.

logger is other_logger because logging.getLogger(__name__) is logging.getLogger(__name__)

Either create a logger directly for the second log logging.Logger(name) (I know the documentation says never to do this but if you want an entirely independent logger this is how to do it), or use a different name for the second log when calling logging.getLogger().

Gelding