why does python.subprocess hang after proc.communicate()?

Question

I've got an interactive program called my_own_exe. First, it prints out alive, then you input S\n and then it prints out alive again. Finally you input L\n. It does some processing and exits.

However, when I call it from the following python script, the program seemed to hang after printing out the first 'alive'.

Can anyone here tell me why this is happening?

// after reading the follow ups (thank you guys), i modified the code as following:

import subprocess
import time

base_command = "./AO_FelixStrategy_UnitTest --bats 31441 --chix 12467 --enxutp 31884 --turq 26372 --symbol SOGN --target_date " + '2009-Oct-16'
print base_command

proc2 = subprocess.Popen(base_command, shell=True , stdin=subprocess.PIPE,)

time.sleep(2);
print "aliv"
proc2.communicate('S\n')

print "alive"
time.sleep(6)

print "alive"
print proc2.communicate('L\n')
time.sleep(6)

the program now goes well with the first input 'S\n', but then stopped, and I the second 'L\n' is kinda ignored.

Can anyone give me an idea why it's like this?

Inandin

Answer 1

From the docs for communicate:

Interact with process: Send data to stdin. Read data from stdout and stderr, until end-of-file is reached. Wait for process to terminate.

So after communicate() runs, the process has been terminated.

If you want to write and read without waiting for the process to stop:

• Don't ever use shell=True - it needlessy invokes a shell to in turn call your program, so there will be another process between you and your program. That has lots of unpleasant side-effects. The default is shell=False so you should stick with that. Change your Popen line to:

  p = subprocess.Popen(["./AO_FelixStrategy_UnitTest",
                        "--bats", "31441", "--chix", "12467",
                        "--enxutp", "31884", "--turq", "26372",
                        "--symbol", "SOGN", "--target_date", '2009-Oct-16'],
                       stdin=subprocess.PIPE, 
                       stdout=subprocess.PIPE)
  

• Use p.stdin.write to write to the process. Use p.stdout.read to read from it.

• Calling p.stdout.read if there's nothing to read will block. Calling p.stdin.write if the write buffer is full will block. So you have to make sure you have something to read/write - you do that on unix OS by using select. On windows you unfortunately must resort to threads. At least that is what Popen.communicate does internally.
• If you didn't write AO_FelixStrategy_UnitTest then you have possible additional problems:
  • It could be reading from somewhere else, not standard input. Some programs read directly from the terminal, others use some OS API to read. That means data written to stdin won't go to the program. This is often true for password prompts.
  • Remember that you have to account for AO_FelixStrategy_UnitTest buffers. By default standard C PIPE communication is buffered so you may not see any output until after you've closed the input side (by doing p.stdin.close(). Unless AO_FelixStrategy_UnitTest flushes the output periodically.

Here's some example code, based on what you describe. It could work depending on how AO_FelixStrategy_UnitTest was developed:

p = subprocess.Popen(["./AO_FelixStrategy_UnitTest",
                      "--bats", "31441", "--chix", "12467",
                      "--enxutp", "31884", "--turq", "26372",
                      "--symbol", "SOGN", "--target_date", '2009-Oct-16'],
                     stdin=subprocess.PIPE, 
                     stdout=subprocess.PIPE)
output = p.communicate('S\nL\n')[0]
print output

Barocchio

Isn't the process restarted every time he calls proc2? Or is not calling it since it has print in front of two of them? And if that's the case, why is freezing after the first print and not the second? – Longo

@Anthony: No. The process is not reestarted. It is freezing after the first print because communicate is waiting for the process to end, but the process never ends because it is probably stuck on the second prompt (the one where 'L\n' should be entered). – Barocchio

thanks. anyway, this only partly solve the problem, i can use communicate only once, correct? if I need to INTERACTIVELY read and write and read and write, this "output = p.communicate('S\nL\n')[0]" could not work, correct? – Inandin

@ccfenix: exactly. By using .communicate(), You write those stuff as soon as you can and close stdin, and keep listening on stdout until the process ends. If you want to interactively talk with the process, you need to do what I said about using p.stdin.write and p.stdout.read instead of .communicate() in the question. I also stated some caveats, like the part about buffering that can happen on the external executable. – Barocchio

A more intuitive idiom is subprocess.Popen(base_command.split() ... – Oralee

@smci, intuitive, maybe, but awful: it breaks as soon as your data has spaces you want held together, so if someone tries to parameterize your code by substituting in a filename before the split, you just got the ability for an attacker who controls the filename to add arbitrary arguments. Remember the Zen of Python: "Explicit is better than implicit". – Organogenesis

@CharlesDuffy: yes if you have a better code implementation that parses and splits options and args in a meaningful-whitespace-aware way, then please post it. – Oralee

@Oralee The better implementation is what this answer is already doing. There's no reason to post any alternative. – Organogenesis

@CharlesDuffy: No it absolutely isn't. This solution is only showing one manually-split string intended for one specific unittest. Hard-coding the split is neither a generalized nor dynamic solution handling arbitrary input strings of args, via code. (and by the way, I've successfully written scripts with subprocess where we sometimes also need to be able to recognize pass-through args to a sub-sub-command, whose args and options we mightn't explicitly know.) So please post what (generalizable) code solution for arg-splitting you recommend. – Oralee

@smci, if the OP's logic needs to be generalized, how to do that depends on domain details. We can't do that from just one example. In some cases the natural way to represent the datasets might be a dictionary, like {"bats": 31441, "chix": 12467, "target-date": "2009-Oct-16"}, so one would be building a dictionary->list transform -- but what's appropriate depends on the natural domain of the data. – Organogenesis

@smci, but, important point here, if you transform dict->list, then you're retaining details about where argument list elements begin and end; dict->string->list (if it were indeed true that the natural form for the list of cases is a dict) adds new failure modes for no good reason. After all, your test case may contain "target_date": "invalid \"date here\" today"; you don't want it being split into one argument invalid, an argument "date, an argument here" and an argument today (as str.split() would do), nor into invalid, date here, today via shlex.split(). – Organogenesis

@smci, this isn't a matter of preference: It's objective fact that there are argument lists that cannot be created via a str.split() operation, so when you go that route you're restricting the domain of your tests to exclude anywhere a space needs to be literal rather than syntactic. (The closest thing to a way to work around that is to put NUL literals -- which aren't legal in argument lists -- in your string and split on those explicitly instead of the default split-on-whitespace behavior, but at that point you might as well just be specifying an explicit list in the first place) – Organogenesis

@Oralee (And remember, a good unit test suite checks corner-case behaviors, so when you design your test suite in such a way that it can't represent corner cases, you're ruling yourself out from being able to test for major classes of security bugs &c). – Organogenesis

@CharlesDuffy: we're no longer talking about str.split(), that's your strawman, we clearly stopped way talking about it back when I referenced arg-parsing and opt-parsing (and I very clearly said solutions would need to handle meaningful whitespace). We both know doing that can sometimes be domain-dependent, but UNIX convention does mitigate that, e.g. 'subcmd -optA -optB argB -optC argC1 argc2 --pass-thru -subsubcmdargX -subsubcmdargY`. Anyway this question title+ body are not just about unittests with manually hardcoded command-strings, so as I said it's not very generalizable... – Oralee

... which is why I suggested the respondent (or you) could actually post code that implements that. – Oralee

@Oralee and @CharlesDuffy the best way to split is to use shlex.split but that seems to belong to another question, instead of hijacking this answer for that? – Barocchio

Answer 2

communicate() reads data from stdout and stderr until end-of-file is reached. - It waits until your program quits.

Vet

So what is the right method for a back-and-forth interaction with a subprocess? – Longo

@Anthony: The correct way is to use p.stdin.write() and p.stdout.read(). However those can block, so you use select() on unix or threads on windows to avoid blocking. Check what .communicate() does by inspecting your subprocess.py source code. Check my answer for more details. – Barocchio

It wasn't my question, I was just trying to generate something the SO might find helpful for fixing his code. That and I was genuinely curious, as I was working on something vaguely similar in PHP last weekend and I like to know how the better-half lives. – Longo

Answer 3

Although the answers above say that the process.communicate() is safe after the process has been terminated (the process has quit), there is a caveat.

• If the process has spawned any child processes, they seem to be blocking process.communicate() as long as child processes are still alive after the calling kill() for the parent. It seems that process.wait() may return a return code for the process (the process has exited), but process.communicate() will still hang if called.

• This is because of how UNIX-based operating systems work (Linux/macOS)

• Events leading to this situation can subtly happen if multiprocessing.Manager Python class is used from the standard library. Manager will spawn a child process behind the scenes. This is not visible to the developer, so this may came as a surprise.

• Because the child process Manager has spawned does not have Process.daemon flag set, this spawned process will block the terminate() or kill()for the whole process group, causing process.communicate() not to work even after the process has seemingly terminated

• In this stage, the process is in a "zombie" state, not dead but not alive either

For the reference, I did not find a straightforward solution on how to safely use Manager if you want to use process.communicate().

Slaw

Just bumped into the very similar problem when using PHP and Symfony::Process. I've used it to spawn a Bash script which in turn spawns a subprocess (a long sleep and then some other stuff) in the background; and despite the main bash process exited, Symfony::Process still was waiting for all the sleep time. Turned out that as long as original process' stdout is used by some of its children, PHP will wait for it — Python likely too. So the solution is to redirect child's stdout and stderr. – Genitals

Answer 4

comunicate will only run once and then will close the pipe, therefore if you want to send several commands to it you need to send one after the other in the same string.

Here is an example that worked for me after some investigation, trying threads, subprocess32, stdin.write, stdout.read etc etc. This information is not in the official python reference information for communicate: https://docs.python.org/2/library/subprocess.html

The only place where I found this out was here: Python subprocess communicate kills my process

Here is the code, it is simple, no threads, no subprocess32, works on linux and windows. Yes you have to know how many times to send commands to the other process but in general you do know that. On top of this you can add threads, cwd, shell=True or anything else you may want but this is the simplest case:

def do_commands(self, cmd, parms):
    proc = subprocess.Popen(cmd,  stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.PIPE )

    # wait for the process to terminate
    out, err = process.communicate(cmd_parms)
    errcode = process.returncode

    return errcode, out, err

So for example if you want to send multiple carriage returns (\n) to the application being called and the a param in the middle (interactively mind you) you would call it something like this:

cmd_parms = "\n\n\n\n\nparm\n\n"

errcode, out, err = do_commands(command, cmd_parms)

Hope it helps.

Ovalle