How to configure logging to syslog in Python?

Question

I can't get my head around Python's logging module. My needs are very simple: I just want to log everything to syslog. After reading documentation I came up with this simple test script:

import logging
import logging.handlers

my_logger = logging.getLogger('MyLogger')
my_logger.setLevel(logging.DEBUG)

handler = logging.handlers.SysLogHandler()

my_logger.addHandler(handler)

my_logger.debug('this is debug')
my_logger.critical('this is critical')

But this script does not produce any log records in syslog. What's wrong?

Mechanical

Where are you checking your syslog messages? SysLogHandler() emits those messages to udp socket in port 514 in localhost. – Spiel

You're absolutely right. And I've seen that 'localhost-514' in documentation but haven't thought that /dev/log is should be used by default.. Sigh.. – Mechanical

Answer 1

Change the line to this:

handler = SysLogHandler(address='/dev/log')

This works for me

import logging
import logging.handlers

my_logger = logging.getLogger('MyLogger')
my_logger.setLevel(logging.DEBUG)

handler = logging.handlers.SysLogHandler(address = '/dev/log')

my_logger.addHandler(handler)

my_logger.debug('this is debug')
my_logger.critical('this is critical')

Mcmillen

Note that, as the doc says, '/var/run/syslog' is the Right Thing on OS X. – Peritonitis

how can we identify these logs in syslog ? like can we give any application name OR any tag like syslogtag=django ? – Patsypatt

and remember config the /etc/syslog.d/conf file, and restart syslog/rsyslog service – Lugworm

@Patsypatt I have tested that with (but not without) a formatter like logging.Formatter(fmt='myscriptname[%(process)d]: %(levelname)s: %(message)s', ...), an rsyslog condition like $programname == 'myscriptname' works. – Faunia

@Faunia thanks mate, nice of you to look into the issue even after so long. I will get back to you if I face some issue. People like you keep the community running – Patsypatt

For OS X users, I'd recommend 'sudo ln -s /var/run/syslog /dev/log' to allow for cross-platform compatibility as well. – Weigh

Note: On macOS 10.14, and possibly earlier, the default syslog config appears to ignore messages below warning level---e.g. my_logger.info('foo') would not show up in the log, whereas my_logger.warning('bar') would. – Lasonyalasorella

I needed syslog to forward just messages going to facility LOCAL7, which looks like address = ('/dev/log'), facility=logging.handlers.SysLogHandler.LOG_LOCAL7. – Hemimorphite

Answer 2

I found the syslog module to make it quite easy to get the basic logging behavior you describe:

import syslog
syslog.syslog("This is a test message")
syslog.syslog(syslog.LOG_INFO, "Test message at INFO priority")

There are other things you could do, too, but even just the first two lines of that will get you what you've asked for as I understand it.

Cockatrice

I keep the logging module since it allows to change logger setting without affecting all the statements. Also allows to change behavior in case you want to have different types of logging at the time – Himeji

Answer 3

You should always use the local host for logging, whether to /dev/log or localhost through the TCP stack. This allows the fully RFC compliant and featureful system logging daemon to handle syslog. This eliminates the need for the remote daemon to be functional and provides the enhanced capabilities of syslog daemon's such as rsyslog and syslog-ng for instance. The same philosophy goes for SMTP. Just hand it to the local SMTP software. In this case use 'program mode' not the daemon, but it's the same idea. Let the more capable software handle it. Retrying, queuing, local spooling, using TCP instead of UDP for syslog and so forth become possible. You can also [re-]configure those daemons separately from your code as it should be.

Save your coding for your application, let other software do it's job in concert.

Cranmer

you raise a fair point. could you indicate common addresses and ports used by various logging daemons? is there a standard discovery mechanism to determine whether or not the daemon is bound to a tcp socket? – Floria

Am I the only one that doesn't get how this answer is related to the question? – Dnieper

@Dnieper Yeah, you're not crazy. I came here wondering how to get my program's logs to the local syslog for appropriate processing. If it were a stand-alone micro-essay, or the introductory paragraph to the OP's question, it would be sensical and correct; but the guy hasn't actually told us *how* to "let the more capable software handle" my Python application's critical log events. I have no idea why it's being upvoted. – Tragedy

Answer 4

Piecing things together from here and other places, this is what I came up with that works on ubuntu 12.04 and centOS6

Create a file in /etc/rsyslog.d/ that ends in .conf and add the following text

local6.*        /var/log/my-logfile

Restart rsyslog, reloading did NOT seem to work for the new log files. Maybe it only reloads existing conf files?

sudo restart rsyslog

Then you can use this test program to make sure it actually works.

import logging, sys
from logging import config

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'verbose': {
            'format': '%(levelname)s %(module)s P%(process)d T%(thread)d %(message)s'
            },
        },
    'handlers': {
        'stdout': {
            'class': 'logging.StreamHandler',
            'stream': sys.stdout,
            'formatter': 'verbose',
            },
        'sys-logger6': {
            'class': 'logging.handlers.SysLogHandler',
            'address': '/dev/log',
            'facility': "local6",
            'formatter': 'verbose',
            },
        },
    'loggers': {
        'my-logger': {
            'handlers': ['sys-logger6','stdout'],
            'level': logging.DEBUG,
            'propagate': True,
            },
        }
    }

config.dictConfig(LOGGING)


logger = logging.getLogger("my-logger")

logger.debug("Debug")
logger.info("Info")
logger.warn("Warn")
logger.error("Error")
logger.critical("Critical")

Smail

To restart rsyslog on centOS7, sudo service rsyslog restart – Obliquely

Answer 5

I add a little extra comment just in case it helps anyone because I found this exchange useful but needed this little extra bit of info to get it all working.

To log to a specific facility using SysLogHandler you need to specify the facility value. Say for example that you have defined:

local3.* /var/log/mylog

in syslog, then you'll want to use:

handler = logging.handlers.SysLogHandler(address = ('localhost',514), facility=19)

and you also need to have syslog listening on UDP to use localhost instead of /dev/log.

Rickirickie

there is no 'need' to have syslog listen on UDP. Your example will perfectly work with address='/dev/log' also. – Mechanical

yes, sure, but with address = ('localhost',514), the day you have a log server, you replace localhost by the server's address and you've got remote logging ;-) – Rickirickie

Where does the facility=19 come from? why isn't it facility="local3" – Smail

@Mark0978 19 is the numeric representation of local3 as defined by RFC3146 (and subsequently RFC5424) – Sfumato

I wondered about this too, and find that the facility codes are in the source for Python's SysLogHandler – Zebe

Answer 6

Is your syslog.conf set up to handle facility=user?

You can set the facility used by the python logger with the facility argument, something like this:

handler = logging.handlers.SysLogHandler(facility=SysLogHandler.LOG_DAEMON)

Spectral

You need to specify what is the LOG_DAEMON you provide as a value for the facility parameter. – Galway

That would be SysLogHandler.LOG_DAEMON. – Hygrostat

Answer 7

import syslog
syslog.openlog(ident="LOG_IDENTIFIER",logoption=syslog.LOG_PID, facility=syslog.LOG_LOCAL0)
syslog.syslog('Log processing initiated...')

the above script will log to LOCAL0 facility with our custom "LOG_IDENTIFIER"... you can use LOCAL[0-7] for local purpose.

Surbase

your comment has nothing to do with original request – Mechanical

@Mechanical I would agree that this is relevant. I'm going to guess that the syslog package is a bit more efficient than the pure Python implementation? (if less flexible) – Byblow

Answer 8

From https://github.com/luismartingil/per.scripts/tree/master/python_syslog

#!/usr/bin/python
# -*- coding: utf-8 -*-

'''
Implements a new handler for the logging module which uses the pure syslog python module.

@author:  Luis Martin Gil
@year: 2013
'''
import logging
import syslog

class SysLogLibHandler(logging.Handler):
    """A logging handler that emits messages to syslog.syslog."""
    FACILITY = [syslog.LOG_LOCAL0,
                syslog.LOG_LOCAL1,
                syslog.LOG_LOCAL2,
                syslog.LOG_LOCAL3,
                syslog.LOG_LOCAL4,
                syslog.LOG_LOCAL5,
                syslog.LOG_LOCAL6,
                syslog.LOG_LOCAL7]
    def __init__(self, n):
        """ Pre. (0 <= n <= 7) """
        try:
            syslog.openlog(logoption=syslog.LOG_PID, facility=self.FACILITY[n])
        except Exception , err:
            try:
                syslog.openlog(syslog.LOG_PID, self.FACILITY[n])
            except Exception, err:
                try:
                    syslog.openlog('my_ident', syslog.LOG_PID, self.FACILITY[n])
                except:
                    raise
        # We got it
        logging.Handler.__init__(self)

    def emit(self, record):
        syslog.syslog(self.format(record))

if __name__ == '__main__':
    """ Lets play with the log class. """
    # Some variables we need
    _id = 'myproj_v2.0'
    logStr = 'debug'
    logFacilityLocalN = 1

    # Defines a logging level and logging format based on a given string key.
    LOG_ATTR = {'debug': (logging.DEBUG,
                          _id + ' %(levelname)-9s %(name)-15s %(threadName)-14s +%(lineno)-4d %(message)s'),
                'info': (logging.INFO,
                         _id + ' %(levelname)-9s %(message)s'),
                'warning': (logging.WARNING,
                            _id + ' %(levelname)-9s %(message)s'),
                'error': (logging.ERROR,
                          _id + ' %(levelname)-9s %(message)s'),
                'critical': (logging.CRITICAL,
                             _id + ' %(levelname)-9s %(message)s')}
    loglevel, logformat = LOG_ATTR[logStr]

    # Configuring the logger
    logger = logging.getLogger()
    logger.setLevel(loglevel)

    # Clearing previous logs
    logger.handlers = []

    # Setting formaters and adding handlers.
    formatter = logging.Formatter(logformat)
    handlers = []
    handlers.append(SysLogLibHandler(logFacilityLocalN))
    for h in handlers:
        h.setFormatter(formatter)
        logger.addHandler(h)

    # Yep!
    logging.debug('test debug')
    logging.info('test info')
    logging.warning('test warning')
    logging.error('test error')
    logging.critical('test critical')

Wilt

This is very interesting, but it doesn't work on python 2.6.6 (RHEL 6.4): Traceback (most recent call last): File "syslog_bridge.py", line 68, in <module> handlers.append(SysLogLibHandler(logFacilityLocalN)) File "syslog_bridge.py", line 29, in init syslog.openlog(syslog.LOG_PID, self.FACILITY[n]) TypeError: ident string [, logoption [, facility]] – Armindaarming

Edited based on: github.com/luismartingil/scripts/commit/… – Wilt

Answer 9

Here's the yaml dictConfig way recommended for 3.2 & later.

In log cfg.yml:

version: 1
disable_existing_loggers: true

formatters:
    default:
        format: "[%(process)d] %(name)s(%(funcName)s:%(lineno)s) - %(levelname)s: %(message)s"

handlers:
    syslog:
        class: logging.handlers.SysLogHandler
        level: DEBUG
        formatter: default
        address: /dev/log
        facility: local0

    rotating_file:
        class: logging.handlers.RotatingFileHandler
        level: DEBUG
        formatter: default
        filename: rotating.log
        maxBytes: 10485760 # 10MB
        backupCount: 20
        encoding: utf8

root:
    level: DEBUG
    handlers: [syslog, rotating_file]
    propogate: yes

loggers:
    main:
        level: DEBUG
        handlers: [syslog, rotating_file]
        propogate: yes

Load the config using:

log_config = yaml.safe_load(open('cfg.yml'))
logging.config.dictConfig(log_config)

Configured both syslog & a direct file. Note that the /dev/log is OS specific.

Robynroc

Answer 10

I fix it on my notebook. The rsyslog service did not listen on socket service.

I config this line bellow in /etc/rsyslog.conf file and solved the problem:

$SystemLogSocketName /dev/log

Epochal

Answer 11

I use JSON logging and wanted use SysLogHandler with UDP port 514. Eventually got JSON handler config working. In the handlers section, I have:

{
    "syslog": {
        "class": "logging.handlers.SysLogHandler",
        "address": ["127.0.0.1", 514],
        "facility": "local6",
        "formatter": "syslog_fmt"
}

Didn't find this anywhere else.

[Edit] To be clearer about what is going on here: this is only for python code, and using python's built in logging module. The module allows format and destination(s) of the log messages to be configured. One way to configure the log format and destination(s) is by using a JSON file that is used to configure the logging.

The above example allowed me to send log messages to a syslog daemon.

A complete example of such a file is included below.

{
    "version": 1,
    "disable_existing_loggers": "False",
    "formatters": {
        "verbose": {
            "format": "%(asctime)s:%(levelname)s:%(process)d:%(filename)s:%(funcName)s:L%(lineno)d:%(message)s"
        },
        "syslog": {
            "format": "%(levelname)s:%(process)d:%(filename)s:%(funcName)s:L%(lineno)d:%(message)s"
        }
    },
    "handlers": {
        "console": {
            "class":"logging.StreamHandler",
            "formatter": "standard"
        },
        "syslog": {
        "class": "logging.handlers.SysLogHandler",
        "address": ["127.0.0.1", 514],
        "facility": "local6",
        "formatter": "syslog_fmt"
        }
    },
    "loggers": {
        "": {
            "handlers": ["console","syslog"],
            "level": "DEBUG",
            "propagate": "True"
        }
    }
}

The example above sends python log messages to both syslog and the console. The format of messages for destinations is different (syslog already prefixes each message with a timestamp). For the syslog destination, the log uses facility LOCAL6.

Futures

What does the whole config look like, and how do I read that from python? – Mcdevitt

What does JSON logging mean exactly? your syslog config is in json format, right? you dont log in json format? @Futures – Mcdevitt

JSON logging is configuring python logging using a JSON file format.The JSON is read with logging.config.dictConfig(<JSON string>). – Futures

Answer 12

You can also add a file handler or rotating file handler to send your logs to a local file: http://docs.python.org/2/library/logging.handlers.html

Villa

Not very useful if you want to run more than one process. – Biblical

Using a file handler does not answer the question. – Yakka