Suppress newline in Python logging module

Question

I'm trying to replace an ad-hoc logging system with Python's logging module. I'm using the logging system to output progress information for a long task on a single line so you can tail the log or watch it in a console. I've done this by having a flag on my logging function which suppresses the newline for that log message and build the line piece by piece.

All the logging is done from a single thread so there's no serialisation issues.

Is it possible to do this with Python's logging module? Is it a good idea?

Sac

If you want to log multiple things in the same line consider not using logging for it but use a good old file object and write to it directly (and to a different file than the logfile used for other, line-based, things). – Landsman

@Landsman I'm writing to a file and stdout directly right now. It feels like I'm rewriting logging though, so I'd rather use logging if it ends up being less effort. – Sac

Answer 1

Let's start with your last question: No, I do not believe it's a good idea. IMO, it hurts the readability of the logfile in the long run.

I suggest sticking with the logging module and using the '-f' option on your 'tail' command to watch the output from the console. You will probably end up using the FileHandler. Notice that the default argument for 'delay' is False meaning the output won't be buffered.

If you really needed to suppress newlines, I would recommend creating your own Handler.

Brooking

Answer 2

If you wanted to do this you can change the logging handler terminator. I'm using Python 3.4. This was introduced in Python 3.2 as stated by Ninjakannon.

handler = logging.StreamHandler()
handler.terminator = ""

When the StreamHandler writes it writes the terminator last.

Freebooter

This was introduced in Python 3.2. – Rainbolt

How do I integrate this with the logger? – Nardoo

You don't. The handlers are the object that writes the information to a file, socket, stream, or other. Most handlers inherit from the StreamHandler. The terminator variable is a class level variable, so you can change this for all handlers with logging.StreamHandler.terminator = "". This should change all handlers except ones where the terminator was explicitly set. You could also loop through the logger handlers and set the terminator for handler in logger.handlers:... – Freebooter

See logging-cookbook in the docs for examples of using handlers. Looks like this answer needs something like logging.getLogger('').addHandler(handler) to be complete. – Colorific

Using terminator works with FileHandler too. – Velez

Answer 3

The new line, \n, is inserted inside the StreamHandler class.

If you're really set on fixing this behaviour, then here's an example of how I solved this by monkey patching the emit(self, record) method inside the logging.StreamHandler class.

A monkey patch is a way to extend or modify the run-time code of dynamic languages without altering the original source code. This process has also been termed duck punching.

Here is the custom implementation of emit() that omits line breaks:

def customEmit(self, record):
    # Monkey patch Emit function to avoid new lines between records
    try:
        msg = self.format(record)
        if not hasattr(types, "UnicodeType"): #if no unicode support...
            self.stream.write(msg)
        else:
            try:
                if getattr(self.stream, 'encoding', None) is not None:
                    self.stream.write(msg.encode(self.stream.encoding))
                else:
                    self.stream.write(msg)
            except UnicodeError:
                self.stream.write(msg.encode("UTF-8"))
        self.flush()
    except (KeyboardInterrupt, SystemExit):
        raise
    except:
        self.handleError(record)

Then you would make a custom logging class (in this case, subclassing from TimedRotatingFileHandler).

class SniffLogHandler(TimedRotatingFileHandler):
    def __init__(self, filename, when, interval, backupCount=0,
                 encoding=None, delay=0, utc=0):

        # Monkey patch 'emit' method
        setattr(StreamHandler, StreamHandler.emit.__name__, customEmit)

        TimedRotatingFileHandler.__init__(self, filename, when, interval,
                                          backupCount, encoding, delay, utc)

Some people might argue that this type of solution is not Pythonic, or whatever. It might be so, so be careful.

Also, be aware that this will globally patch SteamHandler.emit(...), so if you are using multiple logging classes, then this patch will affect the other logging classes as well!

Check out these for further reading:

• What is monkey-patching?
• Is monkeypatching considered good programming practice?
• Monkeypatching For Humans

Hope that helps.

Beefeater

Answer 4

Python 3.5.9

class MFileHandler(logging.FileHandler):
  """Handler that controls the writing of the newline character"""

  special_code = '[!n]'

  def emit(self, record) -> None:

    if self.special_code in record.msg:
      record.msg = record.msg.replace( self.special_code, '' )
      self.terminator = ''
    else:
      self.terminator = '\n'

    return super().emit(record)

Then

fHandler = MFileHandler(...)

Example:

# without \n
log.info( 'waiting...[!n]' )
 ...
log.info( 'OK' )

# with \n
log.info( 'waiting...' )
 ...
log.info( 'OK' )

log.txt:

waiting...OK
waiting...
OK

Keratinize

Is this still the best way to do this? – Woodwork

@Woodwork For me, yes. But I would not recommend using this option if the logfile is filled from different threads. – Keratinize

Answer 5

I encountered a need to log a certain section in a single line as I iterated through a tuple, but wanted to retain the overall logger.

Collected the output first into a single string and later sent it out to logger once I was out of the section. Example of concept

for fld in object._fields: 
  strX = (' {} --> {} ').format(fld, formattingFunction(getattr(obj,fld)))
debugLine += strX
logger.debug('{}'.format(debugLine))

Mhd