Python: How to determine subprocess children have all finished running

Question

I am trying to detect when an installation program finishes executing from within a Python script. Specifically, the application is the Oracle 10gR2 Database. Currently I am using the subprocess module with Popen. Ideally, I would simply use the wait() method to wait for the installation to finish executing, however, the documented command actually spawns child processes to handle the actual installation. Here is some sample code of the failing code:

import subprocess
OUI_DATABASE_10GR2_SUBPROCESS = ['sudo',
                                 '-u',
                                 'oracle',
                                 os.path.join(DATABASE_10GR2_TMP_PATH,
                                              'database',
                                              'runInstaller'),
                                 '-ignoreSysPrereqs',
                                 '-silent',
                                 '-noconfig',
                                 '-responseFile '+ORACLE_DATABASE_10GR2_SILENT_RESPONSE]
oracle_subprocess = subprocess.Popen(OUI_DATABASE_10GR2_SUBPROCESS)
oracle_subprocess.wait()

There is a similar question here: Killing a subprocess including its children from python, but the selected answer does not address the children issue, instead it instructs the user to call directly the application to wait for. I am looking for a specific solution that will wait for all children of the subprocess. What if there are an unknown number of subprocesses? I will select the answer that addresses the issue of waiting for all children subprocesses to finish.

More clarity on failure: The child processes continue executing after the wait() command since that command only waits for the top level process (in this case it is 'sudo'). Here is a simple diagram of the known child processes in this problem: Python subprocess module -> Sudo -> runInstaller -> java -> (unknown)

Shillong

I just added more clarity at the bottom of the question. – Shillong

Can you prevent oracle from spawning subprocesses? – Cunning

No, I don't understand the processes of the oracle installation program. I am looking for a generic answer that can be applied to any process-child spawning problem. – Shillong

sudo should wait for its children to finish. Does runInstaller not wait for its children? That seems odd to me. If true, there isn't much you can do (at least on Unix-style OSes). The parent of the orphaned processes will become the init process (pid 1). – Importunate

I think sudo is waiting for its direct child to finish (a bash script called runInstaller), but that child is calling java. The runInstaller script does not wait for java to finish executing and finishes before all children have finished. – Shillong

Answer 1

Ok, here is a trick that will work only under Unix. It is similar to one of the answers to this question: Ensuring subprocesses are dead on exiting Python program. The idea is to create a new process group. You can then wait for all processes in the group to terminate.

pid = os.fork()
if pid == 0:
    os.setpgrp()
    oracle_subprocess = subprocess.Popen(OUI_DATABASE_10GR2_SUBPROCESS)
    oracle_subprocess.wait()
    os._exit(0)
else:
    os.waitpid(-pid)

I have not tested this. It creates an extra subprocess to be the leader of the process group, but avoiding that is (I think) quite a bit more complicated.

I found this web page to be helpful as well. http://code.activestate.com/recipes/278731-creating-a-daemon-the-python-way/

Importunate

I'll try it out and get back to you on the results. – Shillong

Still failing, it may have to do with the fact that a shell script is executing the Java program but I'm not sure why. – Shillong

Answer 2

You can just use os.waitpid with the the pid set to -1, this will wait for all the subprocess of the current process until they finish:

import os
import sys
import subprocess


proc = subprocess.Popen([sys.executable,
                         '-c',
                         'import subprocess;'
                         'subprocess.Popen("sleep 5", shell=True).wait()'])

pid, status = os.waitpid(-1, 0)

print pid, status

This is the result of pstree <pid> of different subprocess forked:

python───python───sh───sleep

Hope this can help :)

Carrew

That's a very interesting way of getting it to work, but it assumes that you can intercept each child process being spawned. In order for this to work in my problem, I would have to dismantle the actual Java program being run and insert the equivalent of this code in Java (which I can't do because I don't have the source code). I agree though that this would work if all I had was Python executables and a shell script at the very bottom, but unfortunately I am dealing with code that I am not able to modify. – Shillong

@pokstad: I'm not sure what do you mean by : but it assumes that you can intercept each child process being spawned, but in the same case i'm not sure if it will work for your case , i don't have a real case like yours in my hand to test this with :) , well hope this was helpful at least :) – Carrew

Yes it was helpful, thank you for your help. What I meant by 'intercepting' is that the reason your example works is because each process in your example has code to wait for it's child to finish. The executable I want to wait for does not have this code, otherwise it would work. So in order for mine to work like yours, I would need to insert some 'wait' code into it, which I am not able to do. – Shillong

@Shillong : yes that's true, never mind hope you got what you're searching for :) – Carrew

Answer 3

Check out the following link http://www.oracle-wiki.net/startdocsruninstaller which details a flag you can use for the runInstaller command.

This flag is definitely available for 11gR2, but I have not got a 10g database to try out this flag for the runInstaller packaged with that version.

Regards

Detergency

+1 Thanks Mark. I would accept this as an answer but the question was general for all subprocesses so I don't want everyone else to feel cheated. – Shillong

Answer 4

Everywhere I look seems to say it's not possible to solve this in the general case. I've whipped up a library called 'pidmon' that combines some answers for Windows and Linux and might do what you need.

I'm planning to clean this up and put it on github, possibly called 'pidmon' or something like that. I'll post a link if/when I get it up.

EDIT: It's available at http://github.com/dbarnett/python-pidmon.

I made a special waitpid function that accepts a graft_func argument so that you can loosely define what sort of processes you want to wait for when they're not direct children:

import pidmon
pidmon.waitpid(oracle_subprocess.pid, recursive=True,
        graft_func=(lambda p: p.name == '???' and p.parent.pid == ???))

or, as a shotgun approach, to just wait for any processes started since the call to waitpid to stop again, do:

import pidmon
pidmon.waitpid(oracle_subprocess.pid, graft_func=(lambda p: True))

Note that this is still barely tested on Windows and seems very slow on Windows (but did I mention it's on github where it's easy to fork?). This should at least get you started, and if it works at all for you, I have plenty of ideas on how to optimize it.

Medicine

And make no mistake, this is a hack, not an elegant "solution to the problem", but I think it might help some people. – Medicine

Also, I am aware of os.waitpid, I was just testing it with processes that weren't children of the python process, and AFAIK waitpid only works with children of the current process. I'll probably do something to try waitpid and fall back to this manual method. – Medicine

I like how you threw that up on Github right quick. I'll try it out and see if it works. I believe the main issue here is that processes spawned from shell scripts behave differently. Hope it works, will get back to you soon! – Shillong

@pokstad: Yeah, Github is the awesome-sauce that makes code "real", social, and readily-available on the internet. I try to take advantage of it whenever possible. – Medicine

@pokstad: I noticed when I was testing it on Linux that, for instance, if I ran gnome-terminal from inside gnome-terminal, it opens it indirectly through a "factory" so it messes up the process hierarchy. Your issue might be similar. I'm not sure there's anything to be done for it, but I was considering what sort of hacks I could include so that it would look "greedily" for descendant processes and catch funny spawning. BTW, there's no setup.py file with dependencies yet, but obviously win32com is a dep to use it on Windows. – Medicine

I tried the code and no dice. I'm trying to figure out how to get past that factory problem. I even modified the bash script by appending a wait command at the end of the script. One idea I'm trying to implement is creating something like a chroot jail where the process is restricted to a sub-shell. A lot of documentation I've been finding indicates that shell scripts create subshells that allow multiprocessing, but at the same time creates the issue of knowing when all the child processes finish. – Shillong

@pokstad: Okay, I've implemented another stab at something that might do the trick for you, and provided neat examples! This is all code that I've written specifically for your problem, so I really hope it fixes your problem. Sorry if it crashes right out of the box, it's been really cumbersome to test it on Windows, but I'm at least trying to. – Medicine