Python logging - overview of currently installed loggers/handlers

Question

I am trying to set up a Fluentd log handler from a Django project using a logger from code:

def get_fluentd_logger(name):
    import logging
    from fluent import handler

    logging.basicConfig(level=logging.DEBUG)
    logger = logging.getLogger(name)
    logger.addHandler(handler.FluentHandler(name, host='localhost', port=24224))
    return logger

handler.FluentHandler comes from package fluent-logger and I am running fluent locally.

fluent.conf:

<source>
  type forward
  port 24224
</source>

<match **>
  type copy
  <store>
        type stdout
  </store>

  <store>
          type forward
          send_timeout 60s
          recover_wait 10s
          heartbeat_interval 1s
          phi_threshold 8
          hard_timeout 60s

          <server>
            name monitoring
            host 1.2.3.4
            port 24224
            weight 100
          </server>
  </store>
</match>

When I run this from a non-django python project it works fine, but when called from django it just does not do anything.

The question is: is there a way to see the currently installed loggers and their handlers so I can debug this situation?

[EDIT]

When done from the django settings like this:

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'simple': {
            'format': '%(levelname)s: %(message)s'
        },
    },
    'handlers': {
       'fluentdebug':{
            'level':'DEBUG',
            'class':'fluent.handler.FluentHandler',
            'formatter': 'simple',
            'tag':'foo.bar.baz',
            'host':'localhost',
            'port':24224,
        },
    },
    'loggers': {
        'foo.bar.baz': {
            'handlers': ['fluentdebug'],
            'level': 'DEBUG',
            'propagate': True,
        },
    }
}

It does work. I however would like to be able to do this from code because foo.bar.baz can take many values and I dont want to pollute this file with 20 loggers and handlers that do exactly the same.

Maybe the real question is: Why cant I add loggers from code to logging after Django has performed it's setup?

Domiciliate

offtopic: It is funny to see how activity around questions takes off after America is finally awake. Two hours to go from now :) – Domiciliate

Answer 1

I am going to answer myself because if somebody runs in the same problem it is easier to find. The problem seems to be with the FluentHandler. Using Brandon's Logging tree as mentioned by @Thomas we could see that the loggers were correctly added.

However further debugging showed that if you not explicitly set the loglevel it is set to NotSet. Somehow in "ordinary" python this is ok and the record is emitted. In Django however NotSet level logs are thrown away. I don't know if this is standard Django behaviour, but it caused a lot of headaces. Below is the code that worked:

def get_fluentd_logger(name):
    import logging
    from fluent import handler

    logging.basicConfig(level=logging.DEBUG)
    logger = logging.getLogger(name)
    logger.level = logging.INFO
    handler = handler.FluentHandler(name, host='localhost', port=24224)
    handler.level = logging.INFO
    logger.addHandler(handler)
    return logger

Domiciliate

Not mine, it's Brandon Rhodes' – Bergstein

Answer 2

This isn't something I've dealt with before, but Logging Tree sounds like what you might be looking for. You ought to read Brandon's post about it, but here's an example of output:

<--""
   Level WARNING
   Handler Stream <open file '<stderr>', mode 'w' at ...>
   |
   o<--[cherrypy]
       |
       o<--"cherrypy.access"
       |   Level INFO
       |   Handler Stream <open file '<stdout>', mode 'w' ...>
       |
       o<--"cherrypy.error"
           Level INFO
           Handler Stream <open file '<stderr>', mode 'w' ...>

Bergstein

This looks interesting. Ill give it a go. – Domiciliate

Python is a dynamic language, you can read the can and dig in the innards of the logging module yourself. Especially since this looks like a debugging issue and there's not particular concern with working on all versions of python. – Taal