Python: logging module - globally
Asked Answered
T

5

81

I was wondering how to implement a global logger that could be used everywhere with your own settings:

I currently have a custom logger class:

class customLogger(logging.Logger):
   ...

The class is in a separate file with some formatters and other stuff. The logger works perfectly on its own.

I import this module in my main python file and create an object like this:

self.log = logModule.customLogger(arguments)

But obviously, I cannot access this object from other parts of my code. Am i using a wrong approach? Is there a better way to do this?

Talanian answered 1/10, 2011 at 17:48 Comment(1)
I am not sure I understand what's the "logging" limitation that you are trying to overcomeBuckjump
O
157

Use logging.getLogger(name) to create a named global logger.

main.py

import log
logger = log.setup_custom_logger('root')
logger.debug('main message')

import submodule

log.py

import logging

def setup_custom_logger(name):
    formatter = logging.Formatter(fmt='%(asctime)s - %(levelname)s - %(module)s - %(message)s')

    handler = logging.StreamHandler()
    handler.setFormatter(formatter)

    logger = logging.getLogger(name)
    logger.setLevel(logging.DEBUG)
    logger.addHandler(handler)
    return logger

submodule.py

import logging

logger = logging.getLogger('root')
logger.debug('submodule message')

Output

2011-10-01 20:08:40,049 - DEBUG - main - main message
2011-10-01 20:08:40,050 - DEBUG - submodule - submodule message
Oliguria answered 1/10, 2011 at 18:10 Comment(4)
The logger defined in every submodule is not a global parameter , Is there a way to make it one? I want that when I define logger = logging.getLogger('root') in submodule.py it will be the same logger defined in logger = logging.getLogger('root') in submodule2.pyFrigate
What do you do if you configure your "global" logger after you have imported the module? (e.g. when forking a process with the multiprocessing module)Papen
Since you have to re-define the logger in your submodule.py, then the global definition is not such. Likewise one could have just re-initialised the logger in every sub-module reading from a config file (actually requiring even fewer key strokes).Delacruz
When calling setup_custom_logger(), Would suggest to use __name__ instead of hard-coded stringProcne
S
98

Since I haven't found a satisfactory answer, I would like to elaborate on the answer to the question a little bit in order to give some insight into the workings and intents of the logging library, that comes with Python's standard library.

In contrast to the approach of the OP (original poster) the library clearly separates the interface to the logger and configuration of the logger itself.

The configuration of handlers is the prerogative of the application developer who uses your library.

That means you should not create a custom logger class and configure the logger inside that class by adding any configuration or whatsoever.

The logging library introduces four components: loggers, handlers, filters, and formatters.

  • Loggers expose the interface that application code directly uses.
  • Handlers send the log records (created by loggers) to the appropriate destination.
  • Filters provide a finer grained facility for determining which log records to output.
  • Formatters specify the layout of log records in the final output.

A common project structure looks like this:

Project/
|-- .../
|   |-- ...
|
|-- project/
|   |-- package/
|   |   |-- __init__.py
|   |   |-- module.py
|   |   
|   |-- __init__.py
|   |-- project.py
|
|-- ...
|-- ...

Inside your code (like in module.py) you refer to the logger instance of your module to log the events at their specific levels.

A good convention to use when naming loggers is to use a module-level logger, in each module which uses logging, named as follows:

logger = logging.getLogger(__name__)

The special variable __name__ refers to your module's name and looks something like project.package.module depending on your application's code structure.

module.py (and any other class) could essentially look like this:

import logging
...
log = logging.getLogger(__name__)

class ModuleClass:
    def do_something(self):
        log.debug('do_something() has been called!')

The logger in each module will propagate any event to the parent logger which in return passes the information to its attached handler! Analogously to the python package/module structure, the parent logger is determined by the namespace using "dotted module names". That's why it makes sense to initialize the logger with the special __name__ variable (in the example above name matches the string "project.package.module").

There are two options to configure the logger globally:

  • Instantiate a logger in project.py with the name __package__ which equals "project" in this example and is therefore the parent logger of the loggers of all submodules. It is only necessary to add an appropriate handler and formatter to this logger.

  • Set up a logger with a handler and formatter in the executing script (like main.py) with the name of the topmost package.

When developing a library which uses logging, you should take care to document how the library uses logging - for example, the names of loggers used.

The executing script, like main.py for example, might finally look something like this:

import logging
from project import App

def setup_logger():
    # create logger
    logger = logging.getLogger('project')
    logger.setLevel(logging.DEBUG)

    # create console handler and set level to debug
    ch = logging.StreamHandler()
    ch.setLevel(logger.level)

    # create formatter
    formatter = logging.Formatter('%(asctime)s [%(levelname)s] %(name)s: %(message)s')

    # add formatter to ch
    ch.setFormatter(formatter)

    # add ch to logger
    logger.addHandler(ch)

if __name__ == '__main__' and __package__ is None:
     setup_logger()
     app = App()
     app.do_some_funny_stuff()

The method call log.setLevel(...) specifies the lowest-severity log message a logger will handle but not necessarily output! It simply means the message is passed to the handler as long as the message's severity level is higher than (or equal to) the one that is set. But the handler is responsible for handling the log message (by printing or storing it for example).

Hence the logging library offers a structured and modular approach which just needs to be exploited according to one's needs.

Logging documentation

Full executable example on replit.com

Slacks answered 4/5, 2017 at 23:58 Comment(6)
Thanks for the detailed answer. For python 3, here it is: docs.python.org/3/howto/logging.html#logging-advanced-tutorialAdditament
Awesome. This should be the accepted answer.Pill
I feel like that just does not answer the question. It was asked how to use a custom logger class, and logging does support custom logger classes that can be registered to the logging manager, that is then handed out by logging.get_logger(name).Kumar
This code does not run (level is undefined, example probably copy-pasted) nor works as intended after adjusting: this won't actually result in logging statements from module.py being shown, since setup_logger creates a new logger and configures that one. That logger isn't a parent of module.py's logger. Instead the root logger (which is a parent) should be configured, tldr; in setup_logger replacing all calls to logger. with logging.getLogger(). is a starting point.Lucilucia
@Lucilucia Yes, the level was undefined. I fixed that. Apart from that everything works as expected. I have just recreated it in a REPL to double confirm. See replit.com/@dega/py-logging Everything else you said is wrong. Read the documentation: "Loggers that are further down in the hierarchical list are children of loggers higher up in the list."Slacks
@Slacks sorry my mistake, I used the code without adjusting the hardcoded 'project'string hence the other loggers had no relationship with the main one since the used a different name; may I suggest you change logging.getLogger('project') to logging.getLogger(project.__name__), as that will catch mistakes like mine, can be linted, etc.?Lucilucia
D
14

The python logging module is already good enough as global logger, you might simply looking for this:

main.py

import logging
logging.basicConfig(level = logging.DEBUG,format = '[%(asctime)s] %(levelname)s [%(name)s:%(lineno)s] %(message)s')

Put the codes above into your executing script, then you can use this logger with the same configs anywhere in your projects:

module.py

import logging
logger = logging.getLogger(__name__)
logger.info('hello world!')

For more complicated configs you may use a config file logging.conf with logging

logging.config.fileConfig("logging.conf")
Dippy answered 10/6, 2021 at 7:12 Comment(2)
The answer that I was logging... for, thanks Funny LI #dadjokesRecital
This is an excellent solution except that it propagates the settings to whatever third party packages that are used by my project. Is there a simple way to limit this propagatio to just my package?Journalese
M
13

Create an instance of customLogger in your log module and use it as a singleton - just use the imported instance, rather than the class.

Mader answered 1/10, 2011 at 18:0 Comment(0)
K
3

You can just pass it a string with a common sub-string before the first period. The parts of the string separated by the period (".") can be used for different classes / modules / files / etc. Like so (specifically the logger = logging.getLogger(loggerName) part):

def getLogger(name, logdir=LOGDIR_DEFAULT, level=logging.DEBUG, logformat=FORMAT):
    base = os.path.basename(__file__)
    loggerName = "%s.%s" % (base, name)
    logFileName = os.path.join(logdir, "%s.log" % loggerName)
    logger = logging.getLogger(loggerName)
    logger.setLevel(level)
    i = 0
    while os.path.exists(logFileName) and not os.access(logFileName, os.R_OK | os.W_OK):
        i += 1
        logFileName = "%s.%s.log" % (logFileName.replace(".log", ""), str(i).zfill((len(str(i)) + 1)))
    try:
        #fh = logging.FileHandler(logFileName)
        fh = RotatingFileHandler(filename=logFileName, mode="a", maxBytes=1310720, backupCount=50)
    except IOError, exc:
        errOut = "Unable to create/open log file \"%s\"." % logFileName
        if exc.errno is 13: # Permission denied exception
            errOut = "ERROR ** Permission Denied ** - %s" % errOut
        elif exc.errno is 2: # No such directory
            errOut = "ERROR ** No such directory \"%s\"** - %s" % (os.path.split(logFileName)[0], errOut)
        elif exc.errno is 24: # Too many open files
            errOut = "ERROR ** Too many open files ** - Check open file descriptors in /proc/<PID>/fd/ (PID: %s)" % os.getpid()
        else:
            errOut = "Unhandled Exception ** %s ** - %s" % (str(exc), errOut)
        raise LogException(errOut)
    else:
        formatter = logging.Formatter(logformat)
        fh.setLevel(level)
        fh.setFormatter(formatter)
        logger.addHandler(fh)
    return logger

class MainThread:
    def __init__(self, cfgdefaults, configdir, pidfile, logdir, test=False):
        self.logdir = logdir
        logLevel = logging.DEBUG
        logPrefix = "MainThread_TEST" if self.test else "MainThread"
        try:
            self.logger = getLogger(logPrefix, self.logdir, logLevel, FORMAT)
        except LogException, exc:
            sys.stderr.write("%s\n" % exc)
            sys.stderr.flush()
            os._exit(0)
        else:
            self.logger.debug("-------------------- MainThread created.  Starting __init__() --------------------")

    def run(self):
        self.logger.debug("Initializing ReportThreads..")
        for (group, cfg) in self.config.items():
            self.logger.debug(" ------------------------------ GROUP '%s' CONFIG ------------------------------     " % group)
            for k2, v2 in cfg.items():
                self.logger.debug("%s <==> %s: %s" % (group, k2, v2))
            try:
                rt = ReportThread(self, group, cfg, self.logdir, self.test)
            except LogException, exc:
                sys.stderr.write("%s\n" % exc)
                sys.stderr.flush()
                self.logger.exception("Exception when creating ReportThread (%s)" % group)
                logging.shutdown()
                os._exit(1)
            else:
                self.threads.append(rt)
        self.logger.debug("Threads initialized.. \"%s\"" % ", ".join([t.name for t in self.threads]))
        for t in self.threads:
            t.Start()
        if not self.test:
            self.loop()


class ReportThread:
    def __init__(self, mainThread, name, config, logdir, test):
        self.mainThread = mainThread
        self.name = name
        logLevel = logging.DEBUG
        self.logger = getLogger("MainThread%s.ReportThread_%s" % ("_TEST" if self.test else "", self.name), logdir, logLevel, FORMAT)
        self.logger.info("init database...")
        self.initDB()
        # etc....

if __name__ == "__main__":
    # .....
    MainThread(cfgdefaults=options.cfgdefaults, configdir=options.configdir, pidfile=options.pidfile, logdir=options.logdir, test=options.test)
Karakoram answered 1/10, 2011 at 18:0 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.