Redirect Python 'print' output to Logger

Question

I have a Python script that makes use of 'Print' for printing to stdout. I've recently added logging via Python Logger and would like to make it so these print statements go to logger if logging is enabled. I do not want to modify or remove these print statements.

I can log by doing 'log.info("some info msg")'. I want to be able to do something like this:

if logging_enabled:
  sys.stdout=log.info
print("test")

If logging is enabled, "test" should be logged as if I did log.info("test"). If logging isn't enabled, "test" should just be printed to the screen.

Is this possible? I know I can direct stdout to a file in a similar manner (see: redirect prints to log file)

Amnesty

Answer 1

You have two options:

1. Open a logfile and replace sys.stdout with it, not a function:

   log = open("myprog.log", "a")
   sys.stdout = log
   
   >>> print("Hello")
   >>> # nothing is printed because it goes to the log file instead.
   

2. Replace print with your log function:

   # If you're using python 2.x, uncomment the next line
   #from __future__ import print_function
   print = log.info
   
   >>> print("Hello!")
   >>> # nothing is printed because log.info is called instead of print
   

Manton

In the first example, is it possible to have both, in file and on the screen stdout? – Boar

@HrvojeT: See How to duplicate sys.stdout to a log file? – Outcry

Arguments passed to log.info do not have the same meaning as those passed to print. For example, the file= parameter allows you to print to any file. This feature is lost if you replace the print function by a logger as a whole. Only prints to stdout/stderr should be intercepted. – Tangram

WARNING!! This setup will fail if print contains any parameter which is inappropriate in log.info, like: print("Hello", file = sys.stderr) – Augend

Answer 2

Of course, you can both print to the standard output and append to a log file, like this:

# Uncomment the line below for python 2.x
#from __future__ import print_function

import logging

logging.basicConfig(level=logging.INFO, format='%(message)s')
logger = logging.getLogger()
logger.addHandler(logging.FileHandler('test.log', 'a'))
print = logger.info

print('yo!')

Faraway

WARNING!! This setup will fail if print contains any parameter which is inappropriate in logger.info, like: print("yo!", file = sys.stderr) – Augend

Answer 3

One more method is to wrap the logger in an object that translates calls to write to the logger's log method.

Ferry Boender does just this, provided under the GPL license in a post on his website. The code below is based on this but solves two issues with the original:

1. The class doesn't implement the flush method which is called when the program exits.
2. The class doesn't buffer the writes on newline as io.TextIOWrapper objects are supposed to which results in newlines at odd points.

import logging
import sys


class StreamToLogger(object):
    """
    Fake file-like stream object that redirects writes to a logger instance.
    """
    def __init__(self, logger, log_level=logging.INFO):
        self.logger = logger
        self.log_level = log_level
        self.linebuf = ''

    def write(self, buf):
        temp_linebuf = self.linebuf + buf
        self.linebuf = ''
        for line in temp_linebuf.splitlines(True):
            # From the io.TextIOWrapper docs:
            #   On output, if newline is None, any '\n' characters written
            #   are translated to the system default line separator.
            # By default sys.stdout.write() expects '\n' newlines and then
            # translates them so this is still cross platform.
            if line[-1] == '\n':
                self.logger.log(self.log_level, line.rstrip())
            else:
                self.linebuf += line

    def flush(self):
        if self.linebuf != '':
            self.logger.log(self.log_level, self.linebuf.rstrip())
        self.linebuf = ''


logging.basicConfig(
    level=logging.DEBUG,
    format='%(asctime)s:%(levelname)s:%(name)s:%(message)s',
    filename="out.log",
    filemode='a'
)

stdout_logger = logging.getLogger('STDOUT')
sl = StreamToLogger(stdout_logger, logging.INFO)
sys.stdout = sl

stderr_logger = logging.getLogger('STDERR')
sl = StreamToLogger(stderr_logger, logging.ERROR)
sys.stderr = sl

This allows you to easily route all output to a logger of your choice. If needed, you can save sys.stdout and/or sys.stderr as mentioned by others in this thread before replacing it if you need to restore it later.

Hawkie

How do you use this? do you place this in the module which you expect to generatestdout for each of the modules that can potentially send output to the stdout? Or will any output sent to the stdout be sent to the file, as long I have configured it as above in the __main__ function? – Paratroops

Crashes at program exit : the StreamToLogger class must also implement flush(). – Tangram

Does anyone know the purpose of the self.linebuf = ''. I can't see any reference to it being used anywhere. – Motorboat

Answer 4

A much simpler option,

import logging, sys
logging.basicConfig(filename='path/to/logfile', level=logging.DEBUG)
logger = logging.getLogger()
sys.stderr.write = logger.error
sys.stdout.write = logger.info

Wight

This solution only works, if you don't use logging.StreamHandler() to also print the log to your screen. Because if you do, you send the message in an infinite loop: The stream handler tries to write it to sys.stdout.write, where it is redirected to the logger and then again to the stream handler. – Underwent

You can use sys.__stderr__ for StreamHandler, that one never changes. – Justinejustinian

It says: sys.stdout.write = details.info AttributeError: 'file' object attribute 'write' is read-only – Clougher

@Clougher I believe you are running in Python2.7. According to this, that is an error in 2.7. I was able to replicate your error in my python2.7 too. I personally would recommend you switch to Python3.x. – Wight

@aidan.plenert.macdonald, do you know if there is any workaround for python2.7? It's a script to work with legacy code. – Clougher

@Clougher Looks like you can set sys.stderr = CustomClass, and when I tested sys.stderr = 1, all errors were then suppressed. So I would recommend making an object that behaved like a file an takes your error output. – Wight

add a logging.StreamHandler() to this code generate hundreds lines error message, same as Decrayer's comment. – Cofer

This code works fine for me as I get the reference to the original sys.stdout before setting it. – Andaman

This gives me an error sys.stderr.write = logger.error AttributeError: 'file' object attribute 'write' is read-only – Thaumatrope

Answer 5

Once your defined your logger, use this to make print redirect to logger even with mutiple parameters of print.

print = lambda *tup : logger.info(str(" ".join([str(x) for x in tup]))) 

Peculate

Answer 6

You really should do that the other way: by adjusting your logging configuration to use print statements or something else, depending on the settings. Do not overwrite print behaviour, as some of the settings that may be introduced in the future (eg. by you or by someone else using your module) may actually output it to the stdout and you will have problems.

There is a handler that is supposed to redirect your log messages to proper stream (file, stdout or anything else file-like). It is called StreamHandler and it is bundled with logging module.

So basically in my opinion you should do, what you stated you don't want to do: replace print statements with actual logging.

Guimar

I will use logging for future scripts, but for the purpose of what I am doing it's not worth the time to update everything for logger. I will look in to StreamHandler though. – Amnesty

@Rauffle: As you wish. I strongly suggest using the second solution mentioned by C0deH4cker, otherwise you may have problems I mentioned in my answer. – Guimar

I have to use certain 3rd party libraries which send the output to stdout. I am using logging in my application, so my application itself doesnt have any print statements. You mentioned that StreamHandler can send log messages to a file or stdout. Does StreamHandler send output from stdout to a file? – Paratroops

Yeah, this is particularly interesting if you're dealing with a 3rd party library that does logging improperly via print(). I'd like print to basically become logging.info for every single one of their print calls. Then I'd get timestamps, thread ids, etc with the right filter. – Plosion

Answer 7

Below snipped works perfectly inside my PySpark code. If someone need in case for understanding -->

import os
import sys
import logging
import logging.handlers

log = logging.getLogger(__name_)

handler = logging.FileHandler("spam.log")
formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
handler.setFormatter(formatter)
log.addHandler(handler)
sys.stderr.write = log.error 
sys.stdout.write = log.info 

(will log every error in "spam.log" in the same directory, nothing will be on console/stdout)

(will log every info in "spam.log" in the same directory,nothing will be on console/stdout)

to print output error/info in both file as well as in console remove above two line.

Happy Coding Cheers!!!

Palembang