How to start a background process with nohup using Fabric?

Question

Through Fabric, I am trying to start a celerycam process using the below nohup command. Unfortunately, nothing is happening. Manually using the same command, I could start the process but not through Fabric. Any advice on how can I solve this?

def start_celerycam():
    '''Start celerycam daemon'''
    with cd(env.project_dir):
        virtualenv('nohup bash -c "python manage.py celerycam --logfile=%scelerycam.log --pidfile=%scelerycam.pid &> %scelerycam.nohup &> %scelerycam.err" &' % (env.celery_log_dir,env.celery_log_dir,env.celery_log_dir,env.celery_log_dir))

        

Elixir

Answer 1

I'm using Erich Heine's suggestion to use 'dtach' and it's working pretty well for me:

def runbg(cmd, sockname="dtach"):
    return run('dtach -n `mktemp -u /tmp/%s.XXXX` %s' % (sockname, cmd))

This was found here.

Parietal

I tried a bunch of different ways of doing this. This was the one that finally worked. – Odonnell

I tried this, it works! But when I add nohup in 'cmd', still not working, so I just removed nohup and '&'. – Seventeenth

How can I see the logs of the detached execution. Can it be configured?Or will there be a specific location where I can look for? – Hamelin

Answer 2

As I have experimented, the solution is a combination of two factors:

• run process as a daemon: nohup ./command &> /dev/null &
• use pty=False for fabric run

So, your function should look like this:

def background_run(command):
    command = 'nohup %s &> /dev/null &' % command
    run(command, pty=False)

And you can launch it with:

execute(background_run, your_command)

Akerboom

mesg: ttyname failed: Inappropriate ioctl for devic – Saguaro

Answer 3

This is an instance of this issue. Background processes will be killed when the command ends. Unfortunately on CentOS 6 doesn't support pty-less sudo commands.

The final entry in the issue mentions using sudo('set -m; service servicename start'). This turns on Job Control and therefore background processes are put in their own process group. As a result they are not terminated when the command ends.

For even more information see this link.

Embodiment

+1: Thanks! The pty=false suggestion wasn't working for me on RHEL (because sudo required TTY), but prefixing set -m; worked great! Could we get more information on what the ramifications of set -m; are, though? What exactly is that doing / how is this solution working? Is there something I should be wary of? Is there some cleanup I should perform later on? – Scowl

For me, both run("nohup %s >& /dev/null < dev/null &" % cmd, pty=False) as well as run ("set -m; nohup %s &" % cmd) worked for me. – Keslie

Answer 4

you just need to run

run("(nohup yourcommand >& /dev/null < /dev/null &) && sleep 1")

Templia

Wow, how did you find this? A slightly simpler version would also be (nohup yourcommand &> /dev/null < /dev/null &) && /bin/true – Housemaid

Answer 5

DTACH is the way to go. It's a software you need to install like a lite version of screen. This is a better version of the "dtach"-method found above, it will install dtach if necessary. It's to be found here where you can also learn how to get the output of the process which is running in the background:

from fabric.api import run
from fabric.api import sudo
from fabric.contrib.files import exists


def run_bg(cmd, before=None, sockname="dtach", use_sudo=False):
    """Run a command in the background using dtach

    :param cmd: The command to run
    :param output_file: The file to send all of the output to.
    :param before: The command to run before the dtach. E.g. exporting
                   environment variable
    :param sockname: The socket name to use for the temp file
    :param use_sudo: Whether or not to use sudo
    """
    if not exists("/usr/bin/dtach"):
        sudo("apt-get install dtach")
    if before:
        cmd = "{}; dtach -n `mktemp -u /tmp/{}.XXXX` {}".format(
            before, sockname, cmd)
    else:
        cmd = "dtach -n `mktemp -u /tmp/{}.XXXX` {}".format(sockname, cmd)
    if use_sudo:
        return sudo(cmd)
    else:
        return run(cmd)

May this help you, like it helped me to run omxplayer via fabric on a remote rasberry pi!

Colonial

Did you actually every get it to pump the results to an output_file? Your code doesn't seem to actually use that parameter. I'm trying to understand how I can log events? – Tad

Hi, I'm looking for a way to get the output logs. Did you guys managed to find any solution to store the logs to an output file? – Hamelin

Answer 6

You can use :

run('nohup /home/ubuntu/spider/bin/python3 /home/ubuntu/spider/Desktop/baidu_index/baidu_index.py > /home/ubuntu/spider/Desktop/baidu_index/baidu_index.py.log 2>&1 &', pty=False)

Paulina

Answer 7

nohup did not work for me and I did not have tmux or dtach installed on all the boxes I wanted to use this on so I ended up using screen like so:

run("screen -d -m bash -c '{}'".format(command), pty=False)

This tells screen to start a bash shell in a detached terminal that runs your command

Funest

Answer 8

You could be running into this issue

Try adding 'pty=False' to the sudo command (I assume virtualenv is calling sudo or run somewhere?)

Bungalow

yeah still it hangs even with pty=False – Elixir

Answer 9

This worked for me:

sudo('python %s/manage.py celerycam --detach --pidfile=celerycam.pid' % siteDir)

Edit: I had to make sure the pid file was removed first so this was the full code:

# Create new celerycam
sudo('rm celerycam.pid', warn_only=True)
sudo('python %s/manage.py celerycam --detach --pidfile=celerycam.pid' % siteDir)

Mikesell

Answer 10

I was able to circumvent this issue by running nohup ... & over ssh in a separate local shell script. In fabfile.py:

@task
def startup():
    local('./do-stuff-in-background.sh {0}'.format(env.host))

and in do-stuff-in-background.sh:

#!/bin/sh

set -e
set -o nounset

HOST=$1

ssh $HOST -T << HERE
   nohup df -h 1>>~/df.log 2>>~/df.err &
HERE

Of course, you could also pass in the command and standard output / error log files as arguments to make this script more generally useful.

(In my case, I didn't have admin rights to install dtach, and neither screen -d -m nor pty=False / sleep 1 worked properly for me. YMMV, especially as I have no idea why this works...)

Jaine