Print statements not working when serve_forever() is called? [duplicate]

Question

I have the following small python script to run a local server for testing some html:

print('opened')

from http.server import HTTPServer, SimpleHTTPRequestHandler

server_address = ('', 8000)
httpd = HTTPServer(server_address, SimpleHTTPRequestHandler)

print("Listening at https://127.0.0.1:8000/ . . .")
httpd.serve_forever()

When I run this in the terminal, it blocks the print statements: nothing is printed. But the server works and I can go to localhost:8000 in the browser and access my html files. If, however, I comment out the last line, the call to serve_forever(), it works, printing both opened and Listening at https:127.0.0.1:8000/ . . .. Except of course it doesn't actually work, since now the server isn't being run.

I find this very confusing. The previous lines are executed before the last line. Why would the last line cause the previous lines to not work?

Python3 on Windows7 if anyone was going to ask, but I doubt that's relevant.

Lange

Can you make sure that there are no illegal characters on the last line? The only thing blocking the first line from executing would be a syntax error AFAIK. You do run this code in a separate terminal / process / thread from your client code I assume? – Swanhilda

There are no illegal characters. I'm not sure what you mean about 'separate from client code.' Does the code work for you? – Lange

Without any handler server_forever ? What do you expect ? Add an handler or don't use Base modules ..... – Palette

"What do I expect?" Well, the server works. It still serves files at localhost:8000 no problem. So really I have to ask what did you expect? I'm just not clear as to why it's blocking the print statements from executing. – Lange

Would anything be printed if you call sys.stdout.flush() before the last print? – Expugnable

Does it get printed if you close the server cleanly (by catching the keyboard interrupt exception)? – Headspring

Because I tried executing the same and I got both the messages along with the server running. Could you please also mention your platform and python3 version? – Headspring

@Expugnable that worked. I'm very confused. Even more so because the code works for Ajay without that. – Lange

@AjayBrahmakshatriya I'm not sure what you mean by "by catching the keyboard interrupt exception." How can I do that? And it says at the bottom of the question, python 3 on Windows 7 :) – Lange

@Aerovistae there are subversions in python. Like mine is 3.5.2. Now about catching keyboard interrupt exception, you can see this - #21121447 – Headspring

@Aerovistae well (1) the python script in the question is the actual script you are running right? (2) how is python installed? via the official site, via MSYS2, or some other means? (3) what is the exact python version? run python3 --version to find out (4) how do you run the script? (e.g. from cmd.exe and run python3 script.py? or double-click on the script? ...?) – Expugnable

And the reason your printing wasn't happening is that, your operating system supplied library was buffering the output. Since I was using Linux, it didn't do it for me. sys.stdout.flush forces it to display whatever is buffered up to that point. – Headspring

FYI, The same code as OP's does work well to me on Mac OS X. – Productive

I think you will like these posts #231251 and #25369286 It won't work well without flushing, either. – Productive

@Expugnable & Ajay I am using python 3.6.0, sorry Ajay I misread what you were asking about the sub-version. I will read through these comments and answers as soon as I get a chance this weekend. – Lange

Consider accepting an answer (effective&simple solution) and manually awarding the bounty. – Tacket

Thank you, with my 12k rep I didn't know how the basics of the site work! And my previous comment definitely doesn't say "I'll look at this when I get a chance this weekend." – Lange

I confirmed that my answer is working on a similar issue #43279986 reedited my answer to contain this info! – Pierre

HI, @Aerovistae, see also the manual at docs.python.org/3/library/functions.html?highlight=print#print . You can find that print is buffered, that is all your print statement are stored in memory and actually shown when certain events occur, such as program successful termination. This is an optimization technique, easy to switch off with -u option of interpreter or special sys variable. No idea why. I hope the goal was to motivate server developer rely on logging which is superior in many aspects, yet it might be just following C language behavior or limitations of early computers. – Darnall

@Darnall your solution works too! Just adding -u erases the problem. I find what you've said profoundly confusing, though-- how can I know what events cause the buffer to release? For instance the program print('hi') input('hanging on user input') will print "hi" before the program terminates while it waits for my input. Why is that? – Lange

@Aerovistae I do not think somebody build an exhaustive list, flush need even was added after some discussions bugs.python.org/issue11633. The behavior might depend on OS, version, etc. Typically input, explicit flush commands, termination, I guess buffer overflow, sometimes mere end of line. – Darnall

Answer 1

That maybe related with the "infamous" need to flush in order for your prints to work!

Related reading material:

• Theoretical knowledge on the subject
• How to flush output of Python print?, python `print` does not work in loop, as suggested by @Kerorin:

---

Because you are using Python 3 and since version 3.3 you don't have to follow the solutions given in the above great answers.
The print build-in type has an option flush which by default is False. Do:

print('opened', flush=True)

from http.server import HTTPServer, SimpleHTTPRequestHandler

server_address = ('', 8000)
httpd = HTTPServer(server_address, SimpleHTTPRequestHandler)

print('Listening at https://127.0.0.1:8000/ . . .', flush=True)
httpd.serve_forever()

---

PS: This is a confirmed solution on a similar issue

Pierre

Answer 2

There several solutions for this phenomena:

Disable output buffering

Reopen stdout file descriptor with write mode, and 0 as the buffer size (unbuffered). I suggest to write this line as the first line in your code, and this way, all your code will remain the same except the stdout buffer:

sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)

Run python with unbuffered binary stdout and stderr

Force the binary layer of the stdout and stderr streams (which is available as their buffer attribute) to be unbuffered. The text I/O layer will still be line-buffered if writing to the console, or block-buffered if redirected to a non-interactive file.

So just run your script like this:

python -u <your_pyScript>

Or by setting the environment variable PYTHONUNBUFFERED

Set flush keyword argument to true

Since Python 3.3, you can force the normal print() function to flush without the need to use sys.stdout.flush() just set the "flush" keyword argument to true:

print("...", flush=True)

Changing the default in one module to flush=True

You can change the default for the print function by using functools.partial on the global scope of a module:

import functools
print = functools.partial(print, flush=True)

We can see it works just as expected:

>>> print('foo')
foo

Gorgoneion

Your 2nd, 3rd, and 4th solutions all work perfectly of course, but the first gives me an error: File "C:\Users\Aerovistae\AppData\Local\Programs\Python\Python36-32\lib\os.py", line 1015, in fdopen return io.open(fd, *args, **kwargs) ValueError: can't have unbuffered text I/O Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='cp1252'> OSError: [Errno 9] Bad file descriptor – Lange

python -u works like a charm – Jacobine

Answer 3

import logging
logging.info('Listening at https://127.0.0.1:8000/ . . .')

Hi, please, consider using logging instead of printing, you do not want bother with all the shortcomings of the print statement. Print is for beginners, may be for interactive mode. All the professional server-side coders rely on logging.

Check In python, why use logging instead of print? for the full list of logging goodness.

Darnall

Doesn't actually answer the question, which is about understanding why this happens, but does provide a very useful related suggestion anyway! Upvoted. I hadn't encountered the logging module before. – Lange