How many times was logging.error() called?

Question

Maybe it's just doesn't exist, as I cannot find it. But using python's logging package, is there a way to query a Logger to find out how many times a particular function was called? For example, how many errors/warnings were reported?

Courtroom

Answer 1

The logging module doesn't appear to support this. In the long run you'd probably be better off creating a new module, and adding this feature via sub-classing the items in the existing logging module to add the features you need, but you could also achieve this behavior pretty easily with a decorator:

class CallCounted:
    """Decorator to determine number of calls for a method"""

    def __init__(self,method):
        self.method=method
        self.counter=0

    def __call__(self,*args,**kwargs):
        self.counter+=1
        return self.method(*args,**kwargs)


import logging
logging.error = CallCounted(logging.error)
logging.error('one')
logging.error('two')
print(logging.error.counter)

Output:

ERROR:root:one
ERROR:root:two
2

Treadway

This cannot see the error occurred in sub modules. – Siana

The issue with this approach is that the filepath and line number it logs in the log file are the path and line number of the decorator. – Jockstrap

Answer 2

You can also add a new Handler to the logger which counts all calls:

class MsgCounterHandler(logging.Handler):
    level2count = None

    def __init__(self, *args, **kwargs):
        super(MsgCounterHandler, self).__init__(*args, **kwargs)
        self.level2count = {}

    def emit(self, record):
        l = record.levelname
        if (l not in self.level2count):
            self.level2count[l] = 0
        self.level2count[l] += 1

You can then use the dict afterwards to output the number of calls.

Urtication

This was ideal to me. The decorator answer shadowed the true function name and line number where the logger was called from. With a custom handler, it worked perfectly. – Abb

Arrrrg, it took me a long time to try to figure out what the heck this "level2" business was all about. Some kind of "level 2" logging? Is 2 one of the level constants (such as logging.ERROR), and this code is supposed to count that only? Nope... Here the "2" is a homophone for "To". The variable level2count is intended to be a dict of counts, indexed by level. Hence it maps level to count. FWIW, the logging source code has analogous list _levelToName with "To" spelled out. So... this answer has a decent solution, poor choice of variable name. – Rhineland

Also, in the example code, level2count is declared twice: Once as a class variable (not later used), and then as an instance variable. – Rhineland

Answer 3

There'a a warnings module that -- to an extent -- does some of that.

You might want to add this counting feature to a customized Handler. The problem is that there are a million handlers and you might want to add it to more than one kind.

You might want to add it to a Filter, since that's independent of the Handlers in use.

Bumblebee

Answer 4

Based on Rolf's answer and how to write a dictionary to a file, here another solution which stores the counts in a json file. In case the json file exists and continue_counts=True, it restores the counts on initialisation.

import json
import logging
import logging.handlers
import os


class MsgCounterHandler(logging.Handler):
    """
    A handler class which counts the logging records by level and periodically writes the counts to a json file.
    """

    level2count_dict = None

    def __init__(self, filename, continue_counts=True, *args, **kwargs):
        """
        Initialize the handler.

        PARAMETER
        ---------
        continue_counts: bool, optional
            defines if the counts should be loaded and restored if the json file exists already.
        """
        super(MsgCounterHandler, self).__init__(*args, **kwargs)

        filename = os.fspath(filename)
        self.baseFilename = os.path.abspath(filename)

        self.continue_counts = continue_counts

        # if another instance of this class is created, get the actual counts
        if self.level2count_dict is None:
            self.level2count_dict = self.load_counts_from_file()

    def emit(self, record):
        """
        Counts a record.
        In case, create add the level to the dict.
        If the time has come, update the json file.
        """
        level = record.levelname
        if level not in self.level2count_dict:
            self.level2count_dict[level] = 0
        self.level2count_dict[level] += 1

        self.flush()

    def flush(self):
        """
        Flushes the dictionary.
        """
        self.acquire()
        try:
            json.dump(self.level2count_dict, open(self.baseFilename, 'w'))
        finally:
            self.release()

    def load_counts_from_file(self):
        """
        Load the dictionary from a json file or create an empty dictionary
        """
        if os.path.exists(self.baseFilename) and self.continue_counts:
            try:
                level2count_dict = json.load(open(self.baseFilename))
            except Exception as a:
                logging.warning(f'Failed to load counts with: {a}')
                level2count_dict = {}
        else:
            level2count_dict = {}
        return level2count_dict

Finite