How to read live output from subprocess python 2.7 and Apache

Question

I have an Apache web server and I made a python script to run a command. Command that I'm running is launching a ROS launch file, that is working indefinitely. I would like to read output from the subprocess live and display it in the page. With my code so far I could only manage to make output to be printed after I terminate the process. I've tried all kinds of solutions from the web but none of them seem to work

command = "roslaunch package test.launch"
proc = subprocess.Popen(
   command,
   stdout=subprocess.PIPE,
   stderr=subprocess.STDOUT,
   env=env,
   shell=True,
   bufsize=1,
)
print "Content-type:text/html\r\n\r\n"
for line in iter(proc.stdout.readline, ''):
   strLine = str(line).rstrip()
   print(">>> " + strLine)
   print("<br/>")

Seventeenth

have you tried to call sys.stdout.flush()? – Conflation

There could be other buffers between your code and the screen. Have you tried obvious things such as, drop subprocess and just print with a pause: print "Content-type:text/html\r\n\r\n" for i in range(100): print ("%02d" % i) * 10000; time.sleep(1) (perhaps you should use Transfer-Encoding: chunk) – Conflation

ok tnx I will try, but what do you mean by drop subprocess? How do I do that? – Seventeenth

"drop subprocess" means that for debugging purposes, remove all subprocess-related code -- if you can't make simple print statements work then there is no point to complicate your example using subprocess. See how to create a minimal complete code example – Conflation

I tried and it works but it prints all at once after 100 seconds, and not one by one... – Seventeenth

Answer 1

The problem is that the output of roslaunch is being buffered. subprocess is not the best tool for real-time output processing in such situation, but there is a perfect tool for just that task in Python: pexpect. The following snippet should do the trick:

import pexpect

command = "roslaunch package test.launch"
p = pexpect.spawn(command)
print "Content-type:text/html\r\n\r\n"
while not p.eof():
    strLine = p.readline()
    print(">>> " + strLine)
    print("<br/>")

Roughshod

Answer 2

Andrzej Pronobis' answer above suffices for UNIX-based systems but the package pexpect does not work as effectively as one would expect for Windows in certain particular scenarios. Here, spawn() doesn't work for Windows as expected. We still can use it with some alterations that can be seen here in the official docs.

The better way here might be to use wexpect (official docs here). It caters to Windows alone.

Kleper