Getting ssh to execute a command in the background on target machine

Question

This is a follow-on question to the How do you use ssh in a shell script? question. If I want to execute a command on the remote machine that runs in the background on that machine, how do I get the ssh command to return? When I try to just include the ampersand (&) at the end of the command it just hangs. The exact form of the command looks like this:

ssh user@target "cd /some/directory; program-to-execute &"

Any ideas? One thing to note is that logins to the target machine always produce a text banner and I have SSH keys set up so no password is required.

Singleaction

Answer 1

This should solve your problem:

nohup myprogram > foo.log 2> foo.err < /dev/null &

---

The syntax and unusual use of < /dev/null are explained especially well in this answer, quoted here for your convenience.

< /dev/null is used to instantly send EOF to the program, so that it doesn't wait for input (/dev/null, the null device, is a special file that discards all data written to it, but reports that the write operation succeeded, and provides no data to any process that reads from it, yielding EOF immediately).

So the command:

nohup myscript.sh >myscript.log 2>&1 </dev/null &
#\__/             \___________/ \__/ \________/ ^
# |                    |          |      |      |
# |                    |          |      |  run in background
# |                    |          |      |
# |                    |          |   don't expect input
# |                    |          |   
# |                    |        redirect stderr to stdout
# |                    |           
# |                    redirect stdout to myscript.log
# |
# keep the command running 
# no matter whether the connection is lost or you logout

will move to background the command, outputting both stdout and stderr to myscript.log without waiting for any input.

---

See also the wikipedia artcle on nohup, also quoted here for your convenience.

Nohuping backgrounded jobs is for example useful when logged in via SSH, since backgrounded jobs can cause the shell to hang on logout due to a race condition. This problem can also be overcome by redirecting all three I/O streams.

Devitrify

Those files are created in the current directory. So the limit is the amount of free space on the partition. Of course you can also redirect to /dev/null. – Asteria

Any ideas on backgrounding the process after it's finished asking for prompts? (like a gpg --decrypt that's finished asking for the password) – Gladsome

Trying to start a resque worker in the background using nohup, but it doesn;t work.. :( – Sarson

I used this solution to start jboss using plink – Herold

@Jax, is the & necessary? It only works for me without it; oddly it does detach from the long running job. Here's my command: ssh -t user@server "cd apps/myapp && nohup make my-make-target > myapp.log 2>&1 < /dev/null". – Disfranchise

Ok, I figured out it was the -t on my ssh that was causing this. – Disfranchise

Can you please explain what < /dev/null mean? Thanks. – Countable

Wikipedia text changed a bit (though still discusses it); the detailed reference is snailbook.com/faq/background-jobs.auto.html – Tani

It seems your answer works for most of the cases. However, for google-chrome, it fails. I mean ssh -n server 'nohup google-chome http://google.com 1>/dev/null 2>&1 &' – Bilbao

Are you sure nohup is needed? I'm getting good results by running the process in the background and redirecting its output to files like ssh "myprogram >foo.out 2>foo.err &". – Sonar

Also from the wikipedia article on nohup: "Also note that a closing SSH session does not always send a HUP signal to depending processes. Among others, this depends on whether a pseudo-terminal was allocated or not." So while strictly the nohup might not always be needed, you're better long term with it than without. – Devitrify

Oh, you're so great!! I'm fighting with it for above 1 hour, thanks! Finally, your answer fixed it. – Personalize

@QianChen < /dev/null is explained here https://mcmap.net/q/93624/-what-is-lt-dev-null-in-bash as avoiding of input wait – Cardigan

Are you sure that this is correct? As far as I know, /dev/null can't be used for input, only for output. For input, it would eventually be /dev/zero. But according to my tests you don't need the input redirection at all. – Pharyngoscope

I believe /dev/zero would return an infinite series of zeros as your input, which is definitely not what you want in this context. I haven't seen any documentation that says you're not allowed to use /dev/null for input, and in fact have seen several references to the fact that it always returns EOF, which is exactly what we want in this case. I believe if it's required or not will come down to the specific shell being triggered on the other end of your ssh connection, but I'd welcome a better explanation from someone more familiar. – Devitrify

Answer 2

This has been the cleanest way to do it for me:-

ssh -n -f user@host "sh -c 'cd /whereever; nohup ./whatever > /dev/null 2>&1 &'"

The only thing running after this is the actual command on the remote machine

Mcgann

-n is not needed as -f implies -n – Vetiver

ssh -f leaves the ssh process connected, just backgrounded. The /dev/null solutions allow ssh to disconnect quickly, which might be preferable. – Tani

This solution works also for sending commands via ssh to a Syonology NAS – Richman

I need the results of "ls -l" to show up on my remote, this command does not do it. – Executory

@Executory Then redirect to some named file rather than /dev/null. – Otter

On OpenWrt, nohup is not installed by default, but this seems to work: ssh user@host "sh -c 'command1; command2' >/dev/null 2>&1 </dev/null &" (without -f as Beni noted). – Fremitus

Answer 3

Redirect fd's

Output needs to be redirected with &>/dev/null which redirects both stderr and stdout to /dev/null and is a synonym of >/dev/null 2>/dev/null or >/dev/null 2>&1.

Parantheses

The best way is to use sh -c '( ( command ) & )' where command is anything.

ssh askapache 'sh -c "( ( nohup chown -R ask:ask /www/askapache.com &>/dev/null ) & )"'

Nohup Shell

You can also use nohup directly to launch the shell:

ssh askapache 'nohup sh -c "( ( chown -R ask:ask /www/askapache.com &>/dev/null ) & )"'

Nice Launch

Another trick is to use nice to launch the command/shell:

ssh askapache 'nice -n 19 sh -c "( ( nohup chown -R ask:ask /www/askapache.com &>/dev/null ) & )"'

Crudity

I know this is a very old answer of yours, but could you add some comments on why the parentheses way is the best way, what (if any) difference adding nohup makes, and why and when you would use nice? I think that would add a lot to this answer. – Storer

Maybe to partly answer this: With nohup, you don't need to append the & to the command to be run. – Selfpreservation

Answer 4

If you don't/can't keep the connection open you could use screen, if you have the rights to install it.

user@localhost $ screen -t remote-command
user@localhost $ ssh user@target # now inside of a screen session
user@remotehost $ cd /some/directory; program-to-execute &

To detach the screen session: ctrl-a d

To list screen sessions:

screen -ls

To reattach a session:

screen -d -r remote-command

Note that screen can also create multiple shells within each session. A similar effect can be achieved with tmux.

user@localhost $ tmux
user@localhost $ ssh user@target # now inside of a tmux session
user@remotehost $ cd /some/directory; program-to-execute &

To detach the tmux session: ctrl-b d

To list screen sessions:

tmux list-sessions

To reattach a session:

tmux attach <session number>

The default tmux control key, 'ctrl-b', is somewhat difficult to use but there are several example tmux configs that ship with tmux that you can try.

Ecru

how would one use screen for this? – Moorwort

Answer 5

I just wanted to show a working example that you can cut and paste:

ssh REMOTE "sh -c \"(nohup sleep 30; touch nohup-exit) > /dev/null &\""

Fowle

Very helpful. THanks. – Subito

Nice because only example showing outer command wrapped in double-quotes, and inner double-quotes escaped. This is almost necessary if invoking the command from PowerShell, where variables are interpolated only from within double-quotes and never from within single quotes. – Transpierce

Answer 6

You can do this without nohup:

ssh user@host 'myprogram >out.log 2>err.log &'

Sonar

This is a very under-rated answer, thank you! – Ostia

The only change I would suggest making to this answer is </dev/null so we can't block on stdin either. – Pemphigus

this will never work as intended (what the op asked for). the shell (and thus ssh) will wait for the backgrounded process to terminate. – Weise

Answer 7

Quickest and easiest way is to use the 'at' command:

ssh user@target "at now -f /home/foo.sh"

Otto

This would be a great general solution if at accepted command line arguments after the time and not only from a file. – Disfranchise

You can simulate a file with <<< like in: ssh user@target "at now -f <<< 'my_comnads'" – Eigenvalue

Involving a separate daemon (and using tooling that won't work unless that daemon is running) seems like a lot of unnecessary complexity for something that doesn't require an extra daemon at all. – Pemphigus

Answer 8

I think you'll have to combine a couple of these answers to get what you want. If you use nohup in conjunction with the semicolon, and wrap the whole thing in quotes, then you get:

ssh user@target "cd /some/directory; nohup myprogram > foo.out 2> foo.err < /dev/null"

which seems to work for me. With nohup, you don't need to append the & to the command to be run. Also, if you don't need to read any of the output of the command, you can use

ssh user@target "cd /some/directory; nohup myprogram > /dev/null 2>&1"

to redirect all output to /dev/null.

Targett

Answer 9

This worked for me may times:

ssh -x remoteServer "cd yourRemoteDir; ./yourRemoteScript.sh </dev/null >/dev/null 2>&1 & " 

Slake

Answer 10

You can do it like this...

sudo /home/script.sh -opt1 > /tmp/script.out &

Babul

Perfect, works with a PuTTY SSH session. I can exit and the script continues on the machine. Thank you. – Callicrates

Answer 11

It appeared quite convenient for me to have a remote tmux session using the tmux new -d <shell cmd> syntax like this:

ssh someone@elsewhere 'tmux new -d sleep 600'

This will launch new session on elsewhere host and ssh command on local machine will return to shell almost instantly. You can then ssh to the remote host and tmux attach to that session. Note that there's nothing about local tmux running, only remote!

Also, if you want your session to persist after the job is done, simply add a shell launcher after your command, but don't forget to enclose in quotes:

ssh someone@elsewhere 'tmux new -d "~/myscript.sh; bash"'

Charged

Answer 12

1. If you run remote command without allocating tty, redirect stdout/stderr works, nohup is not necessary.

   ssh user@host 'background command &>/dev/null &'

2. If you use -t to allocate tty to run interactive command along with background command, and background command is the last command, like this:

   ssh -t user@host 'bash -c "interactive command; nohup backgroud command &>/dev/null &"'

   It's possible that background command doesn't actually start. There's race here:

   1. bash exits after nohup starts. As a session leader, bash exit results in HUP signal sent to nohup process.
   2. nohup ignores HUP signal.

   If 1 completes before 2, the nohup process will exit and won't start the background command at all. We need to wait nohup start the background command. A simple workaroung is to just add a sleep:

   ssh -t user@host 'bash -c "interactive command; nohup backgroud command &>/dev/null & sleep 1"'

The question was asked and answered years ago, I don't know if openssh behavior changed since then. I was testing on: OpenSSH_8.6p1, OpenSSL 1.1.1g FIPS 21 Apr 2020

Cherenkov

Answer 13

Actually, whenever I need to run a command on a remote machine that's complicated, I like to put the command in a script on the destination machine, and just run that script using ssh.

For example:

# simple_script.sh (located on remote server)

#!/bin/bash

cat /var/log/messages | grep <some value> | awk -F " " '{print $8}'

And then I just run this command on the source machine:

ssh user@ip "/path/to/simple_script.sh"

Lyceum

Answer 14

I was trying to do the same thing, but with the added complexity that I was trying to do it from Java. So on one machine running java, I was trying to run a script on another machine, in the background (with nohup).

From the command line, here is what worked: (you may not need the "-i keyFile" if you don't need it to ssh to the host)

ssh -i keyFile user@host bash -c "\"nohup ./script arg1 arg2 > output.txt 2>&1 &\""

Note that to my command line, there is one argument after the "-c", which is all in quotes. But for it to work on the other end, it still needs the quotes, so I had to put escaped quotes within it.

From java, here is what worked:

ProcessBuilder b = new ProcessBuilder("ssh", "-i", "keyFile", "bash", "-c",
 "\"nohup ./script arg1 arg2 > output.txt 2>&1 &\"");
Process process = b.start();
// then read from process.getInputStream() and close it.

It took a bit of trial & error to get this working, but it seems to work well now.

Pregnancy

Why do you have literal quotes inside the argument following -c? I don't see how that could possibly work. It's like ssh -i keyFile bash -c '"nohup ./script arg1 arg2 > output.txt 2>&1 &"' in bash, which doesn't work either and for the same reason (incorrect literal quotes nested inside the necessary and correct syntactic ones). – Pemphigus

Answer 15

YOUR-COMMAND &> YOUR-LOG.log &    

This should run the command and assign a process id you can simply tail -f YOUR-LOG.log to see results written to it as they happen. you can log out anytime and the process will carry on

Marolda

&>your-log.log only works if the remote system's shell is bash (or otherwise has the extension at hand). If it could possibly be sh, >your-log.log 2>&1 is better. – Pemphigus

Answer 16

If you are using zsh then use program-to-execute &! is a zsh-specific shortcut to both background and disown the process, such that exiting the shell will leave it running.

Crackdown

Answer 17

A follow-on to @cmcginty's concise working example which also shows how to alternatively wrap the outer command in double quotes. This is how the template would look if invoked from within a PowerShell script (which can only interpolate variables from within double-quotes and ignores any variable expansion when wrapped in single quotes):

ssh user@server "sh -c `"($cmd) &>/dev/null </dev/null &`""

Inner double-quotes are escaped with back-tick instead of backslash. This allows $cmd to be composed by the PowerShell script, e.g. for deployment scripts and automation and the like. $cmd can even contain a multi-line heredoc if composed with unix LF.

Transpierce

Answer 18

First follow this procedure:

Log in on A as user a and generate a pair of authentication keys. Do not enter a passphrase:

a@A:~> ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/a/.ssh/id_rsa): 
Created directory '/home/a/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/a/.ssh/id_rsa.
Your public key has been saved in /home/a/.ssh/id_rsa.pub.
The key fingerprint is:
3e:4f:05:79:3a:9f:96:7c:3b:ad:e9:58:37:bc:37:e4 a@A

Now use ssh to create a directory ~/.ssh as user b on B. (The directory may already exist, which is fine):

a@A:~> ssh b@B mkdir -p .ssh
b@B's password: 

Finally append a's new public key to b@B:.ssh/authorized_keys and enter b's password one last time:

a@A:~> cat .ssh/id_rsa.pub | ssh b@B 'cat >> .ssh/authorized_keys'
b@B's password: 

From now on you can log into B as b from A as a without password:

a@A:~> ssh b@B

then this will work without entering a password

ssh b@B "cd /some/directory; program-to-execute &"

Czechoslovak

The question says that dagorym already set the keys up to not require a password, but it's still hanging – Stuffed

Answer 19

I think this is what you need: At first you need to install sshpass on your machine. then you can write your own script:

while read pass port user ip; do
sshpass -p$pass ssh -p $port $user@$ip <<ENDSSH1
    COMMAND 1
    .
    .
    .
    COMMAND n
ENDSSH1
done <<____HERE
    PASS    PORT    USER    IP
      .      .       .       .
      .      .       .       .
      .      .       .       .
    PASS    PORT    USER    IP    
____HERE

Lalapalooza