spawning process from python

Question

im spawning a script that runs for a long time from a web app like this:

os.spawnle(os.P_NOWAIT, "../bin/producenotify.py", "producenotify.py", "xx",os.environ)

the script is spawned successfully and it runs, but till it gets over i am not able to free the port that is used by the web app, or in other words i am not able to restart the web app. how do i spawn off a process and make it completely independent of the web app?

this is on linux os.

Bigeye

Why aren't you using subprocess.Popen to do this? – Dierdredieresis

S. Lott: will i be able to kill / restart the web app/ parent process after doing subprocess.Popen? thanks – Bigeye

Answer 1

As @mark clarified it's a Linux system, the script could easily make itself fully independent, i.e., a daemon, by following this recipe. (You could also do it in the parent after an os.fork and only then os.exec... the child process).

Edit: to clarify some details wrt @mark's comment on my answer: super-user privileges are not needed to "daemonize" a process as per the cookbook recipes, nor is there any need to change the current working directory (though the code in the recipe does do that and more, that's not the crucial part -- rather it's the proper logic sequence of fork, _exit and setsid calls). The various os.exec... variants that do not end in e use the parent process's environment, so that part is easy too -- see Python online docs.

To address suggestions made in others' comments and answers: I believe subprocess and multiprocessing per se don't daemonize the child process, which seems to be what @mark needs; the script could do it for itself, but since some code has to be doing forks and setsid, it seems neater to me to keep all of the spawning on that low-level plane rather than mix some high-level and some low-level code in the course of the operation.

Here's a vastly reduced and simplified version of the recipe at the above URL, tailored to be called in the parent to spawn a daemon child -- this way, the code can be used to execute non-Python executables just as well. As given, the code should meet the needs @mark explained, of course it can be tailored in many ways -- I strongly recommend reading the original recipe and its comments and discussions, as well as the books it recommends, for more information.

import os
import sys

def spawnDaemon(path_to_executable, *args)
    """Spawn a completely detached subprocess (i.e., a daemon).

    E.g. for mark:
    spawnDaemon("../bin/producenotify.py", "producenotify.py", "xx")
    """
    # fork the first time (to make a non-session-leader child process)
    try:
        pid = os.fork()
    except OSError, e:
        raise RuntimeError("1st fork failed: %s [%d]" % (e.strerror, e.errno))
    if pid != 0:
        # parent (calling) process is all done
        return

    # detach from controlling terminal (to make child a session-leader)
    os.setsid()
    try:
        pid = os.fork()
    except OSError, e:
        raise RuntimeError("2nd fork failed: %s [%d]" % (e.strerror, e.errno))
        raise Exception, "%s [%d]" % (e.strerror, e.errno)
    if pid != 0:
        # child process is all done
        os._exit(0)

    # grandchild process now non-session-leader, detached from parent
    # grandchild process must now close all open files
    try:
        maxfd = os.sysconf("SC_OPEN_MAX")
    except (AttributeError, ValueError):
        maxfd = 1024

    for fd in range(maxfd):
        try:
           os.close(fd)
        except OSError: # ERROR, fd wasn't open to begin with (ignored)
           pass

    # redirect stdin, stdout and stderr to /dev/null
    os.open(os.devnull, os.O_RDWR)  # standard input (0)
    os.dup2(0, 1)
    os.dup2(0, 2)

    # and finally let's execute the executable for the daemon!
    try:
      os.execv(path_to_executable, args)
    except Exception, e:
      # oops, we're cut off from the world, let's just give up
      os._exit(255)

Cameron

alex, it is a unix-like system. fedora linux. the web app does not have super user privileges. is that a problem? also the daemon should have the same working directory and environment variables as the webapp. how do i achieve these? thanks – Bigeye

alex, tx for clarification. i checked the recipe and it is little confusing. is that code in the parent web app? or does it go in the new script that is to be spawned off? thanks a lot again! – Bigeye

can I return the daemon pid to the function caller at the end of the function? – Ginger

Why do you have two consecutive raise statements? I have always thought the second would never be executed. – Maloney

In my application, the first child turns into a zombie with Alex' code. Before the "return", I inserted os.waitpid(pid, 0), which worked for me. But I'd appreciate if someone could confirm this. – Greeley

Answer 2

You can use the multiprocessing library to spawn processes. A basic example is shown here:

from multiprocessing import Process

def f(name):
    print 'hello', name

if __name__ == '__main__':
    p = Process(target=f, args=('bob',))
    p.start()
    p.join()

Shagreen

this has the problem that the main process can not be killed till the subprocess is done. thanks! – Bigeye

Note multiprocessing library was added to Python 2.6. Here is documentation docs.python.org/library/multiprocessing.html – Berri

@Bigeye If you want the main process to terminate immediately and not wait for the subprocesses, you just need to mark the subprocesses as daemons. – Dimorphism

@Dimorphism "Additionally, these are not Unix daemons or services, they are normal processes that will be terminated (and not joined) if non-daemonic processes have exited." (documentation) doesn't that say that daemonic process will be killed in this scenario? – Luik

Isn't that what I said? If you mark the subprocesses as daemons, the subprocesses will be terminated immediately if the parent process (i.e. non-daemonic) is terminated. – Dimorphism

What about calling os.setsid() from f()? – Coddle

I really love multiprocessing but it doesn't help with the question. – Greeley