How to redirect stdout and stderr to logger in Python

Question

I have a logger that has a RotatingFileHandler. I want to redirect all Stdout and Stderr to the logger. How to do so?

Serin

Do you have external modules/libraries that write to FDs 1 and 2 directly? – Gerger

@IgnacioVazquez-Abrams I don't really understand what you meant but I'll try to explain. I'm using several python processes, and from all of them I want to redirect all stdout and stderr message to my logger. – Serin

Possible duplicate of How do I duplicate sys.stdout to a log file in python? – Ubiety

Answer 1

Not enough rep to comment, but I wanted to add the version of this that worked for me in case others are in a similar situation.

class LoggerWriter:
    def __init__(self, level):
        # self.level is really like using log.debug(message)
        # at least in my case
        self.level = level

    def write(self, message):
        # if statement reduces the amount of newlines that are
        # printed to the logger
        if message != '\n':
            self.level(message)

    def flush(self):
        # create a flush method so things can be flushed when
        # the system wants to. Not sure if simply 'printing'
        # sys.stderr is the correct way to do it, but it seemed
        # to work properly for me.
        self.level(sys.stderr)

and this would look something like:

log = logging.getLogger('foobar')
sys.stdout = LoggerWriter(log.debug)
sys.stderr = LoggerWriter(log.warning)

Elinaelinor

I get a weird output because of the flush method: warning archan_pylint:18: <archan_pylint.LoggerWriter object at 0x7fde3cfa2208>. It seems the stderr object is printed rather than a newline or else, so I just removed the flush method and it seems to work now. – Rudman

@Cameron Please look at my answer below for a small improvement in output readability. – Texas

It works both with python2 and 3, in case you log to a file (e.g. logging.basicConfig(filename='example.log', level=logging.DEBUG). But if you want e.g. logging.basicConfig(stream=sys.stdout, level=logging.DEBUG) then it does not work (on python3 it also causes stack overflow). (I guess because it captures std out), so not so useful for logging e.g. from a Kubernetes pod to std out. Note that the code found by shellcat_zero does work also with stream=sys.stdout. – Hamil

def flush(self): pass avoids printing <archan_pylint.LoggerWriter object at 0x7fde3cfa2208> to the log – Loathly

Answer 2

UPDATE for Python 3:

• Including a dummy flush function which prevents an error where the function is expected (Python 2 was fine with just linebuf='').
• Note that your output (and log level) appears different if it is logged from an interpreter session vs being run from a file. Running from a file produces the expected behavior (and output featured below).
• We still eliminate extra newlines which other solutions do not.

class StreamToLogger(object):
    """
    Fake file-like stream object that redirects writes to a logger instance.
    """
    def __init__(self, logger, level):
       self.logger = logger
       self.level = level
       self.linebuf = ''

    def write(self, buf):
       for line in buf.rstrip().splitlines():
          self.logger.log(self.level, line.rstrip())

    def flush(self):
        pass

Then test with something like:

import StreamToLogger
import sys
import logging

logging.basicConfig(
        level=logging.DEBUG,
        format='%(asctime)s:%(levelname)s:%(name)s:%(message)s',
        filename='out.log',
        filemode='a'
        )
log = logging.getLogger('foobar')
sys.stdout = StreamToLogger(log,logging.INFO)
sys.stderr = StreamToLogger(log,logging.ERROR)
print('Test to standard out')
raise Exception('Test to standard error')

See below for old Python 2.x answer and the example output:

All of the prior answers seem to have problems adding extra newlines where they aren't needed. The solution that works best for me is from http://www.electricmonk.nl/log/2011/08/14/redirect-stdout-and-stderr-to-a-logger-in-python/, where he demonstrates how send both stdout and stderr to the logger:

import logging
import sys
 
class StreamToLogger(object):
   """
   Fake file-like stream object that redirects writes to a logger instance.
   """
   def __init__(self, logger, log_level=logging.INFO):
      self.logger = logger
      self.log_level = log_level
      self.linebuf = ''
 
   def write(self, buf):
      for line in buf.rstrip().splitlines():
         self.logger.log(self.log_level, line.rstrip())
 
logging.basicConfig(
   level=logging.DEBUG,
   format='%(asctime)s:%(levelname)s:%(name)s:%(message)s',
   filename="out.log",
   filemode='a'
)
 
stdout_logger = logging.getLogger('STDOUT')
sl = StreamToLogger(stdout_logger, logging.INFO)
sys.stdout = sl
 
stderr_logger = logging.getLogger('STDERR')
sl = StreamToLogger(stderr_logger, logging.ERROR)
sys.stderr = sl
 
print "Test to standard out"
raise Exception('Test to standard error')

The output looks like:

2011-08-14 14:46:20,573:INFO:STDOUT:Test to standard out
2011-08-14 14:46:20,573:ERROR:STDERR:Traceback (most recent call last):
2011-08-14 14:46:20,574:ERROR:STDERR:  File "redirect.py", line 33, in 
2011-08-14 14:46:20,574:ERROR:STDERR:raise Exception('Test to standard error')
2011-08-14 14:46:20,574:ERROR:STDERR:Exception
2011-08-14 14:46:20,574:ERROR:STDERR::
2011-08-14 14:46:20,574:ERROR:STDERR:Test to standard error

Note that self.linebuf = '' is where the flush is being handled, rather than implementing a flush function.

Meter

This code is licensed GPL. I'm not sure if it can even be posted on SO, which requires compatibility with CC by-sa. – Kalasky

Any idea why I'm getting this error message? "Exception ignored in: <__main__.StreamToLogger object at 0x7f72a6fbe940> AttributeError 'StreamToLogger' object has no attribute 'flush' " – Rush

Remove the last two lines of the code snippet, the error message goes away.... – Rendon

It's 'safer' to extend TextIOBase. Somewhere in my library is calling sys.stdout.isatty() and the StreamToLogger failed because of no attribute 'isatty'. It works after I define class StreamToLogger(TextIOBase). – Calcutta

This does not cover my case. In my code I'm doing: check_call(command, shell=True, stdout=sys.stdout, stderr=sys.stderr ) . This leads to an error cause deep in the code python does this: c2pwrite = stdout.fileno() – Cleavland

If you want to restore the default stdout setting see for example here – Gavingavini

Answer 3

You can use redirect_stdout context manager:

import logging
from contextlib import redirect_stdout

logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)
logging.write = lambda msg: logging.info(msg) if msg != '\n' else None

with redirect_stdout(logging):
    print('Test')

or like this

import logging
from contextlib import redirect_stdout


logger = logging.getLogger('Meow')
logger.setLevel(logging.INFO)
formatter = logging.Formatter(
    fmt='[{name}] {asctime} {levelname}: {message}',
    datefmt='%m/%d/%Y %H:%M:%S',
    style='{'
)
ch = logging.StreamHandler()
ch.setLevel(logging.INFO)
ch.setFormatter(formatter)
logger.addHandler(ch)

logger.write = lambda msg: logger.info(msg) if msg != '\n' else None

with redirect_stdout(logger):
    print('Test')

Oleg

docs.python.org/3/library/contextlib.html : "Context manager for temporarily redirecting sys.stdout to another file or file-like object." "Note that the global side effect on sys.stdout means that this context manager is not suitable for use in library code and most threaded applications. It also has no effect on the output of subprocesses. However, it is still a useful approach for many utility scripts." So it seems quite inconvenient (if possible at all) to cover a whole application (e.g. I have a microservice which runs a grpc server, which starts threads when serving requests). – Hamil

Don't the other solutions here have the same global side effect on sys.stdout/sys.stderr as well? @Hamil – Zhang

Answer 4

If it's an all-Python system (i.e. no C libraries writing to fds directly, as Ignacio Vazquez-Abrams asked about) then you might be able to use an approach as suggested here:

class LoggerWriter:
    def __init__(self, logger, level):
        self.logger = logger
        self.level = level

    def write(self, message):
        if message != '\n':
            self.logger.log(self.level, message)

and then set sys.stdout and sys.stderr to LoggerWriter instances.

Twilley

thank you, that did the job, but for some reason stderr send it's message each word separately, do you know why? – Serin

@Serin presumably because write is called word-by-word. You can adapt my example code to suit your needs more closely. – Twilley

What if sys.stderr.flush() is called after redirecting stderr? – Cabrilla

@Cabrilla then you have to define a flush() method, too. – Twilley

I cant make library code not use sys.stderr .flush() etc. What is the best way to handle all its attributes? – Cabrilla

What if C libraries are involved? Then what? How to get the C library to output to the same LoggerWriter? – Irwin

Answer 5

Output Redirection Done Right!

The Problem

logger.log and the other functions (.info/.error/etc.) output each call as a separate line, i.e. implicitly add (formatting and) a newline to it.

sys.stderr.write on the other hand just writes its literal input to stream, including partial lines. For example: The output "ZeroDivisionError: division by zero" is actually 4(!) separate calls to sys.stderr.write:

sys.stderr.write('ZeroDivisionError')
sys.stderr.write(': ')
sys.stderr.write('division by zero')
sys.stderr.write('\n')

The 4 most upvoted approaches (1, 2, 3, 4) thus result in extra newlines -- simply put "1/0" into your program and you will get the following:

2021-02-17 13:10:40,814 - ERROR - ZeroDivisionError
2021-02-17 13:10:40,814 - ERROR - : 
2021-02-17 13:10:40,814 - ERROR - division by zero

The Solution

Store the intermediate writes in a buffer. The reason I am using a list as buffer rather than a string is to avoid the Shlemiel the painter’s algorithm. TLDR: It is O(n) instead of potentially O(n^2)

class LoggerWriter:
    def __init__(self, logfct):
        self.logfct = logfct
        self.buf = []

    def write(self, msg):
        if msg.endswith('\n'):
            self.buf.append(msg.removesuffix('\n'))
            self.logfct(''.join(self.buf))
            self.buf = []
        else:
            self.buf.append(msg)

    def flush(self):
        pass

# To access the original stdout/stderr, use sys.__stdout__/sys.__stderr__
sys.stdout = LoggerWriter(logger.info)
sys.stderr = LoggerWriter(logger.error)

2021-02-17 13:15:22,956 - ERROR - ZeroDivisionError: division by zero

For versions below Python 3.9, you could replace replace msg.removesuffix('\n') with either msg.rstrip('\n') or msg[:-1].

Zhang

Nice but you are assuming that if there is a '\n' in msg it will always be at the end of msg, and that a single msg will never have more than one '\n'. This is probably true in most current Python implementations, but I'm not sure if it is defined as a language standard, so I prefer the "check each time" approach. It is not as bad as it seems because each time Shlemiel gets a new paint bucket (a '\n') he brings it to the current painting point, starting anew from zero. – Texas

@ToniHomedesiSaun A message with '\n's in it is ok and will just be printed as a multiline log, but as you said most internal error messages are chunked calls to sys.stderr and will appear as separate logs. But I guess you could also msg.split('\n') if you are not ok with multiline logs. – Zhang

I tried your solution with Python 3.9.16 but nothing get's logged in my file handler. Could you please take a look? Thank you! pastebin.com/fKweeqYQ – Culinarian

@JimB You are passing the wrong object. As the name logfct implies, you have to pass the function .info of a logger = logging.getLogger() object. Instead, you are passing logging.INFO, which is just a constant representing a log level that is just the number 20. – Zhang

Answer 6

As an evolution to Cameron Gagnon's response, I've improved the LoggerWriterclass to:

class LoggerWriter(object):
    def __init__(self, writer):
        self._writer = writer
        self._msg = ''

    def write(self, message):
        self._msg = self._msg + message
        while '\n' in self._msg:
            pos = self._msg.find('\n')
            self._writer(self._msg[:pos])
            self._msg = self._msg[pos+1:]

    def flush(self):
        if self._msg != '':
            self._writer(self._msg)
            self._msg = ''

now uncontrolled exceptions look nicer:

2018-07-31 13:20:37,482 - ERROR - Traceback (most recent call last):
2018-07-31 13:20:37,483 - ERROR -   File "mf32.py", line 317, in <module>
2018-07-31 13:20:37,485 - ERROR -     main()
2018-07-31 13:20:37,486 - ERROR -   File "mf32.py", line 289, in main
2018-07-31 13:20:37,488 - ERROR -     int('')
2018-07-31 13:20:37,489 - ERROR - ValueError: invalid literal for int() with base 10: ''

Texas

You are right, the top answers produce spurious newlines on e.g. Exceptions. My answer follows a very similar approach. – Zhang

Answer 7

Quick but Fragile One-Liner

sys.stdout.write = logger.info

sys.stderr.write = logger.error

What this does is simply assign the logger functions to the stdout/stderr .write call which means any write call will instead invoke the logger functions.

The downside of this approach is that both calls to .write and the logger functions typically add a newline so you will end up with extra lines in your log file, which may or may not be a problem depending on your use case.

Another pitfall is that if your logger writes to stderr itself we get infinite recursion (a stack overflow error). So only output to a file.

Zhang

Answer 8

With flush added to Vinay Sajip's answer:

class LoggerWriter:
    def __init__(self, logger, level): 
        self.logger = logger
        self.level = level 

    def write(self, message):
        if message != '\n':
            self.logger.log(self.level, message)

    def flush(self): 
        pass

Bobbysoxer

Note that using an empty flush() method as done here is ok, since the logging handler handles flushing internally: https://mcmap.net/q/37838/-does-python-logging-flush-every-log – Carbonari

Answer 9

Solving problem where StreamHandler causes infinite Recurison

My logger was causing an infinite recursion, because the Streamhandler was trying to write to stdout, which itself is a logger -> leading to infinite recursion.

Solution

Reinstate the original sys.__stdout__ for the StreamHandler ONLY, so that you can still see the logs showing in the terminal.

class DefaultStreamHandler(logging.StreamHandler):
    def __init__(self, stream=sys.__stdout__):
        # Use the original sys.__stdout__ to write to stdout
        # for this handler, as sys.stdout will write out to logger.
        super().__init__(stream)


class LoggerWriter(io.IOBase):
    """Class to replace the stderr/stdout calls to a logger"""

    def __init__(self, logger_name: str, log_level: int):
        """:param logger_name: Name to give the logger (e.g. 'stderr')
        :param log_level: The log level, e.g. logging.DEBUG / logging.INFO that
                          the MESSAGES should be logged at.
        """
        self.std_logger = logging.getLogger(logger_name)
        # Get the "root" logger from by its name (i.e. from a config dict or at the bottom of this file)
        #  We will use this to create a copy of all its settings, except the name
        app_logger = logging.getLogger("myAppsLogger")
        [self.std_logger.addHandler(handler) for handler in app_logger.handlers]
        self.std_logger.setLevel(app_logger.level)  # the minimum lvl msgs will show at
        self.level = log_level  # the level msgs will be logged at
        self.buffer = []

    def write(self, msg: str):
        """Stdout/stderr logs one line at a time, rather than 1 message at a time.
        Use this function to aggregate multi-line messages into 1 log call."""
        msg = msg.decode() if issubclass(type(msg), bytes) else msg

        if not msg.endswith("\n"):
            return self.buffer.append(msg)

        self.buffer.append(msg.rstrip("\n"))
        message = "".join(self.buffer)
        self.std_logger.log(self.level, message)
        self.buffer = []


def replace_stderr_and_stdout_with_logger():
    """Replaces calls to sys.stderr -> logger.info & sys.stdout -> logger.error"""
    # To access the original stdout/stderr, use sys.__stdout__/sys.__stderr__
    sys.stdout = LoggerWriter("stdout", logging.INFO)
    sys.stderr = LoggerWriter("stderr", logging.ERROR)


if __name__ == __main__():
    # Load the logger & handlers
    logger = logging.getLogger("myAppsLogger")
    logger.setLevel(logging.DEBUG)
    # HANDLER = logging.StreamHandler()
    HANDLER = DefaultStreamHandler()  # <--- replace the normal streamhandler with this
    logger.addHandler(HANDLER)
    logFormatter = logging.Formatter("[%(asctime)s] - %(name)s - %(levelname)s - %(message)s")
    HANDLER.setFormatter(logFormatter)

    # Run this AFTER you load the logger
    replace_stderr_and_stdout_with_logger()

And then finally call the replace_stderr_and_stdout_with_logger() after you've initialised your logger (the last bit of the code)

Eriha

I was facing the same problem. sys.__stdout__ really solved the problem. Now, we can dump the data to both console and the file – Aloin

Answer 10

If you want to logging info and error messages into separates stream (info into stdout, errors into stderr) you can use this trick:

class ErrorStreamHandler(log.StreamHandler):
"""Print input log-message into stderr, print only error/warning messages"""
def __init__(self, stream=sys.stderr):
    log.Handler.__init__(self, log.WARNING)
    self.stream = stream

def emit(self, record):
    try:
        if record.levelno in (log.INFO, log.DEBUG, log.NOTSET):
            return
        msg = self.format(record)
        stream = self.stream
        # issue 35046: merged two stream.writes into one.
        stream.write(msg + self.terminator)
        self.flush()
    except RecursionError:  # See issue 36272
        raise
    except Exception:
        self.handleError(record)


class OutStreamHandler(log.StreamHandler):
"""Print input log-message into stdout, print only info/debug messages"""
def __init__(self, loglevel, stream=sys.stdout):
    log.Handler.__init__(self, loglevel)
    self.stream = stream

def emit(self, record):
    try:
        if record.levelno not in (log.INFO, log.DEBUG, log.NOTSET):
            return
        msg = self.format(record)
        stream = self.stream
        # issue 35046: merged two stream.writes into one.
        stream.write(msg + self.terminator)
        self.flush()
    except RecursionError:  # See issue 36272
        raise
    except Exception:
        self.handleError(record)

Usage:

log.basicConfig(level=settings.get_loglevel(),
                    format="[%(asctime)s] %(levelname)s: %(message)s",
                    datefmt='%Y/%m/%d %H:%M:%S', handlers=[ErrorStreamHandler(), OutStreamHandler(settings.get_loglevel())])

Hintze

Answer 11

check this solution, it worked for me.

In the StreamToLogger class below, you pass both a logger instance and a log level to each instance of the class. The log level can be different for stdout and stderr.

This method offers more flexibility because you can decide the log level dynamically when creating each instance. This is useful if you want different levels for stdout and stderr. and The write method directly logs the message using the specified log level.

class StreamToLogger:

    def __init__(self, logger, log_level=logging.INFO):
        self.logger = logger
        self.log_level = log_level

    def write(self, message):
        if message.rstrip() != "":
            self.logger.log(self.log_level, message.rstrip())

    def flush(self):
        pass


stdout_log_level_name = os.getenv('STDOUT_PYTHON_LOG_LEVEL', 'INFO').upper()
stdout_logging_level = getattr(logging, stdout_log_level_name, logging.INFO)
sys.stdout = StreamToLogger(custom_logger, stdout_logging_level)


stderr_log_level_name = os.getenv('STDERR_PYTHON_LOG_LEVEL', 'ERROR').upper()
stderr_logging_level = getattr(logging, stderr_log_level_name, logging.ERROR)
sys.stderr = StreamToLogger(custom_logger, stderr_logging_level)

custom_logger is my logger, replace it with your logger instance.

hope this helps, Happy coding .

Thoroughfare

Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center. – Anaphrodisiac

Answer 12

I come up with this one, Fixes random new-lines appearing in the log file by not logging anything until a new-line character is passed:

The Code

import logging
import sys

logging.basicConfig(
    level=logging.DEBUG,
    format='[%(asctime)s] [%(name)s/%(levelname)s]: %(message)s',
    filename="out.log",
    filemode='a'
)


class OutLogger(object):
    def __init__(self, logger: logging.Logger):
        self.logger = logger
        self.info_ = ""
        self.error_ = ""

    def infolog(self, message: str):
        for line in message.splitlines(keepends=True):
            if line.endswith("\n"):
                self.logger.info(self.info_ + line[:-1])
                self.info_ = ""
            else:
                self.info_ += line

    def errorlog(self, message: str):
        for line in message.splitlines(keepends=True):
            if line.endswith("\n"):
                self.logger.error(self.error_ + line[:-1])
                self.error_ = ""
            else:
                self.error_ += line


if __name__ == "__main__":
    logger = OutLogger(logging.getLogger("Admin Tools"))

    sys.stdout.write = logger.infolog
    sys.stderr.write = logger.errorlog

    print("Test to standard out")
    raise Exception('Test to standard error')

out.log File

[2023-12-24 16:57:08,585] [Admin Tools/INFO]: Test to standard out
[2023-12-24 16:57:08,586] [Admin Tools/ERROR]: Traceback (most recent call last):
[2023-12-24 16:57:08,586] [Admin Tools/ERROR]:   File "c:\Users\AAM\Desktop\ubuntu\admin-tools\logger.py", line 42, in <module>
[2023-12-24 16:57:08,586] [Admin Tools/ERROR]:     raise Exception('Test to standard error')
[2023-12-24 16:57:08,586] [Admin Tools/ERROR]: Exception: Test to standard error

Cliffhanger