Pythonic way to detach a process?

Question

I'm running an etcd process, which stays active until you kill it. (It doesn't provide a daemon mode option.) I want to detach it so I can keep running more python.

What I would do in the shell;

etcd & next_cmd

I'm using python's sh library, at the enthusiastic recommendation of the whole internet. I'd rather not dip into subprocess or Popen, but I haven't found solutions using those either.

What I want;

sh.etcd(detach=True)
sh.next_cmd()

or

sh.etcd("&")
sh.next_cmd()

Unfortunately detach is not a kwarg and sh treats "&" as a flag to etcd.

Am I missing anything here? What's the good way to do this?

Lustig

Answer 1

To implement sh's &, avoid cargo cult programming and use subprocess module directly:

import subprocess

etcd = subprocess.Popen('etcd') # continue immediately
next_cmd_returncode = subprocess.call('next_cmd') # wait for it
# ... run more python here ...
etcd.terminate() 
etcd.wait()

This ignores exception handling and your talk about "daemon mode" (if you want to implement a daemon in Python; use python-daemon. To run a process as a system service, use whatever your OS provides or a supervisor program such as supervisord).

Etam

Using subprocess.Popen('python file.py') on Windows returns FileNotFoundError: [Errno 2] No such file or directory: 'python file.py'. What could be the problem? – Threlkeld

It looks like a good separate question. Related: Calling a python script with input within a python script using subprocess – Etam

@Dr_Zaszuś Try adding shell=True as a kwarg to your Popen call since the command has multiple words. Or you can pass the words as a list of strings. docs.python.org/3.7/library/subprocess.html#subprocess.Popen – Burrton

Answer 2

Author of sh here. I believe you want to use the _bg special keyword parameter http://amoffat.github.io/sh/#background-processes

This will fork your command and return immediately. The process will continue to run even after your script exits.

Libby

Answer 3

subprocess is easy enough to do this too:

This approach works (python3). The key is using "start_new_session=True"

UPDATE: despite Popen docs saying this works, it does not. I found by forking the child and then doing os.setsid() it works as I want

client.py:

#!/usr/bin/env python3
import time
import subprocess
subprocess.Popen("python3 child.py", shell=True, start_new_session=True)
i = 0
while True:
    i += 1
    print("demon: %d" % i)
    time.sleep(1)

child.py:

#!/usr/bin/env python3
import time
import subprocess
import os

pid = os.fork()
if (pid == 0):
    os.setsid()

    i = 0
    while True:
        i += 1
        print("child: %d" % i)
        time.sleep(1)
        if i == 10:
            print("child exiting")
            break

output:

./client.py
demon: 1
child: 1
demon: 2
child: 2
^CTraceback (most recent call last):
  File "./client.py", line 9, in <module>
    time.sleep(1)
KeyboardInterrupt

$ child: 3
child: 4
child: 5
child: 6
child: 7
child: 8
child: 9
child: 10
child exiting

Blasius

looks like you copy-pasted this answer in all similar stackoverflow questions, but it's not correct - it does not detach the subprocess from it's parent, it simply creates a subprocess (on linux, python 3.8) – Grane

Bit confused as the Popen docs say "If start_new_session is true the setsid() system call will be made in the child process prior to the execution of the subprocess. (POSIX only)" And "The setsid() library function allows a process to disassociate itself from its parent:". So it "should" be working. Admittedly I tested on mac. Ok I need to check... – Blasius

You're correct... Ok I found a workaround by forking in the child again and doing os.setsid. That seems to work - can you confirm ? This does sound like a Popen bug as I've seen in many places "As of Python 3.2, you can use subprocess.Popen() and pass start_new_session=True to accomplish fully detach the child process from the parent." – Blasius

Answer 4

Note in the following two examples there is a call to time.sleep(...) to give etcd time to finish starting up before we send it a request. A real solution would probably involving probing the API endpoint to see if it was available and looping if not.

Option 1 (abusing the multiprocessing module):

import sh
import requests
import time

from multiprocessing import Process

etcd = Process(target=sh.etcd)

try:
    # start etcd
    etcd.start()
    time.sleep(3)

    # do other stuff
    r = requests.get('http://localhost:4001/v2/keys/')
    print r.text
finally:
    etcd.terminate()

This uses the multiprocessing module to handle the mechanics of spawning a background tasks. Using this model, you won't see the output from etcd.

Option 2 (tried and true):

import os
import signal
import time
import requests

pid = os.fork()
if pid == 0:
    # start etcd
    os.execvp('etcd', ['etcd'])

try:
    # do other stuff
    time.sleep(3)
    r = requests.get('http://localhost:4001/v2/keys/')
    print r.text
finally:
    os.kill(pid, signal.SIGTERM)

This uses the traditional fork and exec model, which works just as well in Python as it does in C. In this model, the output of etcd will show up on your console, which may or may not be what you want. You can control this by redirecting stdout and stderr in the child process.

Wenona

similar to this solution which uses multiprocessing and os.fork() – Giacinta

Answer 5

Posting this if for no other reason than finding it next time I google the same question:

 if os.fork() == 0:
    os.close(0)
    os.close(1)
    os.close(2)
    subprocess.Popen(('etcd'),close_fds=True)
    sys.exit(0)

Popen close_fds closes the file descriptors other than 0,1,2, so the code closes them explicitly.

Whatnot