Python- How to flush the log? (django)

Question

I'm working with Django-nonrel on Google App Engine, which forces me to use logging.debug() instead of print().

The "logging" module is provided by Django, but I'm having a rough time using it instead of print().

For example, if I need to verify the content held in the variable x, I will put
logging.debug('x is: %s' % x). But if the program crashes soon after (without flushing the stream), then it never gets printed.

So for debugging, I need debug() to be flushed before the program exits on error, and this is not happening.

Altamirano

not an answer to your question but why would you use print anyway?! – Apc

What exactly do you mean by "flushing" the log? – Glassine

Hopefully the example listed helps in understanding. – Altamirano

I'd try something like the AdminEmailHandler logging handler for specific cases such as your example. A database logging handler would be the better choice, not sure if there's a free Sentry alternative out there. – Frisco

Might not be the right one, but this works for me.Hope this helps someone. I tried using flush() method, but did not work. As I exit my application using sys.exit(..), I have created a wrapper method(exit()) around it...which waited for 1 second(time.sleep(1)) before exiting the Application. I am also catching any unexpected error and calling exit() method. This timeout seems to flushout any pending log messages – Terrify

Answer 1

I think this may work for you, assuming you're only using one(or default) handler:

>>> import logging
>>> logger = logging.getLogger()
>>> logging.debug('wat wat')
>>> logger.handlers[0].flush()

It's kind of frowned upon in the documentation, though.

Application code should not directly instantiate and use instances of Handler. Instead, the Handler class is a base class that defines the interface that all handlers should have and establishes some default behavior that child classes can use (or override). http://docs.python.org/2/howto/logging.html#handler-basic

And it could be a performance drain, but if you're really stuck, this may help with your debugging.

True

Uh, how awre people supposed to flush their logs if the only way is through a handler that you're not supposed to access? That sounds broken. – Pannell

No, that is not at all what the documentation says. The quote above means that you should only instantiate and use subclasses of Handler, instead of Handler directly. Your code only uses whatever handler classes are already instantiated by someone else, so there is no problem. – Courtesan

Very helpful indeed; I still find it extremely strange that I am not able to tell the stupid logger in its config that it always flush; but that my client code has to enforce that. – Contractile

It returns "out of range". I initialized the logger using a .ini file with two handlers. logging.hasHandlers() returns true. logger.handlers[0].flush() IndexError: list index out of range – Devisee

Answer 2

If the use case is that you have a python program that should flush its logs when exiting, use logging.shutdown().

From the python documentation:

logging.shutdown()

Informs the logging system to perform an orderly shutdown by flushing and closing all handlers. This should be called at application exit and no further use of the logging system should be made after this call. [...]

Hearsh

Answer 3

Django logging relies on the standard python logging module.

This module has a module-level method: logging.shutdown() which flushes all of the handlers and shuts down the logging system (i.e. logging can not longer be used after it is called)

Inspecting the code of this function shows that currently (python 2.7) the logging module holds a list of weak references to all handlers in a module-level variable called _handlerList so all of the handlers can be flushed by doing something like

[h_weak_ref().flush() for h_weak_ref in logging._handlerList]

because this solution uses the internals of the module @Mikes solution above is better, but it relies on having access to a logger, it can be generalized as follows:

 [h.flush() for h in my_logger.handlerList]

Huoh

Answer 4

In Python3.8:

logging.shutdown( )

Check: https://docs.python.org/3.8/library/logging.html#logging.shutdown

Retrospection

Answer 5

It could be that the logger is not logging (seemingly not flushing) because an exception is being raised in a thread. The thread will need to catch its own exceptions and log them itself.

You cannot rely on a main thread to catch and log thread exceptions.

For example, a timer that runs a function periodically will have this behavior and that function will need to catch and log its exceptions.

Eyecatching

Answer 6

I had the same problem where I wanted my log.info to be flushed to the std:out. Turns out I just had to add a stream handler to stdout and it does the job.

logger.addHandler(logging.StreamHandler(sys.stdout)).

Also, add the right level for logging but that's obvious.

Faliscan

Answer 7

a simple function that always working for register you debug messsages while programming. dont use it for production, since it will not rotate:

def make_log(message):
    import datetime
    with open('mylogfile.log','a') as f:
        f.write(f"{datetime.datetime.now()} {message}\n")

then use as

make_log('my message to register')

when to put on production, just comment the last 2 lines

def make_log(message):
    import datetime
    #with open('mylogfile.log','a') as f:
    #    f.write(f"{datetime.datetime.now()} {message}\n")

Comfy

implementing some trivial logger does not solve the problem. There is a good reason, why sophisticated loggers are available and with with implementation you will (sooner or later) encounter most of them the hard way. Some reasons: Performance, log levels, – James