Python is not working in unbuffered mode

Question

I'm stuggeling with an issue with python. I used Python 2.7.13 and Python 3.6.0 on Red Hat Enterprise Linux Server release 7.1 (Maipo). In order to monitor the proccesses output, I want to use tail -f to take a live look into the STDOUT and STDERR. The Keyword here is unbuffered output. Many suggestion on the internet say use python -u ... or the environment variable PYTHONUNBUFFERED like PYTHONUNBUFFERED=1 python ... or stdbuf -e0 -o0 python .... Nevertheless nothing is wokring for the following test script.

import sys
import time
while(True):
   print("Test String")
   time.sleep(1);

For all different commands I always have buffered output. Even though when I want to use STDERR. It's still buffered which really confuses me because STDERR should be unbuffered by default. Using sys.stdout.flush() or sys.stderr.flush() is also not doing the job. When using flush=True inside print() it's working as intended.

I'm looking for a solution which doesn't make it necessary to edit code because I can't edit all programs in order to get unbuffered and immediately flushed out output. How can I achieve this?

Looking forward to your answers!

Best wishes!

Carpophagous

What do you expect your test script does? – Floruit

You can override print(flush=True) function. – Muskmelon

So my original script has some output of deep learning and neural network steps I want to monitor. In my test script I narrowed it down to some dummy output to see if the output is unbuffered. @ElisByberi for that I've got to edit code. Some programs are so huge it would end in a mess to edit it all.. I'm looking for something python -u or PYTHONUNBUFFERED instead that they're working.. I don't even know why they're not working as advertised. – Carpophagous

Again. What do you want print("Test String") to do that it doesn't? no matter if buffered or not, the output will be the same. – Floruit

As described in my original post, I want to call a python script in the background which is creating stdout and stderr output in files. From time to time I want to monitor output created by this script via print with tail to see how the progress is. print("Test String") is just a dummy script with which I want to test that. Until now, stdout and stderr as well as print create buffered output when redirecting them into a file. Therefore it is hard to monitor my logs with tailf because I get new output every 4-6 hours. – Carpophagous

Answer 1

You can override print() function in Python 3. This way you will not need to change every print() function in your scripts.

import builtins


def print(*args):
    builtins.print(*args, sep=' ', end='\n', file=None, flush=True)


print(
    'hello', 'world',
    'I have overrode print() function!',
    1,  # integer
    [1, 2],  # list
    {1, 2},  # set
    (1, 2),  # tuple
    {1: 2}  # dict
)

will print:

hello world I have overrode print() function! 1 [1, 2] {1, 2} (1, 2) {1: 2}

Muskmelon

Thanks for your suggestion. Unfortunately the output is ('Test String',). I'd be happiest when python -uor PYTHONUNBUFFERED would work as advertised.. do you know if I missed something regarding those to options? – Carpophagous

@TheOrangeman To be honest, I did not test the code. I put it there as an example only. It works now as expected. Thank you for reporting it. – Muskmelon

@TheOrangeman Try this command: python -e PYTHONUNBUFFERED=0 yourscript.py – Muskmelon

Hey Elis, thanks for correction. As you said, the builtins function works now as advertised by you. When trying python -e both of my python Installations say that the -e parameter is an unknown option :( – Carpophagous

@TheOrangeman In python --help are listed all options and arguments (and corresponding environment variables). It is capital E. This would work: python -E PYTHONUNBUFFERED=0 yourscript.py – Muskmelon

Also not working. The Documentation tells me about this -E Ignore all PYTHON* environment variables, e.g. PYTHONPATH and PYTHONHOME, that might be set. Weirdly enough PYTHONUNBUFFERED=1 python [...] is working now... I triple checked the spelling of the environment variable. I can't explain this and feel really dumb that I took your time. Thanks for your efforts @Elias Byberi. – Carpophagous

@ElisByberi does this function override even prints inside packages? – Infante

@LeonardoRick You can override a function in a script only, then you can import that script in other scripts. E.g. You override function print() in script custom.py and wherever you import custom.py function print() will get overridden. – Muskmelon

@ElisByberi The problem is when the print is maded inside a dependecy library of my project. This way I can't override this method print so your solution do not work for me – Infante

@LeonardoRick It’s not perfectly clear to me what it is you’re trying to say. – Muskmelon

@ElisByberi I use a library (pyrogram) that emits a print itself. This print will not be unbeffered if I just override the function on my main.py – Infante

@LeonardoRick I emphasize that "You can override a function in a script only." E.g. If you override function print() in script main.py, it will not override that function anywhere else. – Muskmelon