How to get the PID of a process that is piped to another process in Bash?

Question

I am trying to implement a simple log server in Bash. It should take a file as a parameter and serve it on a port with netcat.

( tail -f $1 & ) | nc -l -p 9977

But the problem is that when the netcat terminates, tail is left behind running. (Clarification: If I don't fork the tail process it will continue to run forever even the netcat terminates.)

If I somehow know the PID of the tail then I could kill it afterwards.
Obviously, using $! will return the PID of netcat.

How can I get the PID of the tail process?

Exocrine

What happens if you don't use &? tail -f is supposed to just wait there. I don't get what the & is for, though it does look like this is part of a bigger script. Anyway, if you kill the pipe I would think that tail would then die (so long as you didn't background it). – Undershorts

from my understanding of shells, & in this case is used with its meaning as "start to background". At least, that's the behavior I'm seeing right now, trying to apply the solution to my own, similar issue. – Seek

related: unix.stackexchange.com/questions/103472/… – Stenographer

Answer 1

Write tail's PID to file descriptor 3, and then capture it from there.

( tail -f $1 & echo $! >&3 ) 3>pid | nc -l -p 9977
kill $(<pid)

Ruminate

I used a variant: ( tail -f $1 & echo $! >pid ) | nc -l -p 9977 (not sure why using file descriptor 3 would help when finally redirecting to a file) – Darcie

Not sure why but my solution fails after a couple of log lines are output. Probably when the pipe buffer is full. Then the initial process seems to be waiting for the pipe to be processed. – Darcie

Answer 2

Another option: use a redirect to subshell. This changes the order in which background processes are started, so $! gives PID of the tail process.

tail -f $1 > >(nc -l -p 9977) &
wait $!

Myceto

The advantage of this approach is that after the wait, $? also holds the exit status – Fanchie

This seems like the cleanest approach by a long shot. Does it change anything (compared to standard piping) from the perspective of either process other than the start order? – Claudetteclaudia

Actually, the syntax on that redirection is wrong (though the principle is correct). >(foo) is substituted for the name of the new file descriptor, whereas > >(foo) actually redirects output to it. You want the first line to be tail -f $1 > >(nc -l -p 9977) &. – Claudetteclaudia

exactly what I needed! Thanks! Just redirecting the pid didn't work in my case. – Weaks

If you want to capture both stdout and stderr in the subshell use &> for the redirection. Example: tail -f $1 &> >(nc -l -p 9977) &. – Strapper

Answer 3

how about this:

jobs -x echo %1

%1 is for first job in chain, %2 for second, etc. jobs -x replaces job specifier with PID.

Duron

This must be the cleanest + shortest solution to 'how to get pid of any process earlier in chain' Thank You! It also works with '&' getting the pid of process earlier in the chain that is running in background! E.g. dd if=/dev/urandom bs=1M count=1024 | sha1sum & pid=$(jobs - x echo %1) kill -USR1 $pid – Kevon

The answer is wrong. Only one job ID is assigned, for the entire pipeline: shopt -s lastpipe; sleep 1 | cat | jobs -> [1] Running sleep 1 | cat. It has the PID of the 1st command on the pipeline but jobs -l prints all the PIDs: shopt -s lastpipe; sleep 1 | cat | jobs -l; jobs -p -> [1] 5394 Running sleep 1 5395 | cat 5394. – Klara

Answer 4

This works for me (SLES Linux):

tail -F xxxx | tee -a yyyy &
export TAIL_PID=`jobs -p`
# export TEE_PID="$!"

The ps|grep|kill trick mentioned in this thread would not work if a user can run the script for two "instances" on the same machine.

jobs -x echo %1 did not work for me (man page not having the -x flag) but gave me the idea to try jobs -p.

Tojo

Note that you can only get the PID of the first command on the pipeline this way. jobs -l would print PIDs of all commands but you'll have to parse its output to extract them. – Klara

Unlike many of the other comments, this also works in /bin/sh to obtain the PID of the first command (process group leader), e.g. to be able to kill the entire pipeline later on. – Hygiene

Answer 5

Maybe you could use a fifo, so that you can capture the pid of the first process, e.g.:

FIFO=my_fifo

rm -f $FIFO
mkfifo $FIFO

tail -f $1 > $FIFO &
TAIL_PID=$!

cat $FIFO | nc -l -p 9977

kill $TAIL_PID

rm -f $FIFO

Trapper

Yes I have tried It before. The problem about using fifo is the same: pipe never gets terminated so cat stays running even netcat terminates. Also the control stays in the cat line so it never executes kill. – Cuthbertson

That's odd - the script above worked perfectly for me on Mac OS X. Only slight difference was that I omitted the '-p' flag for nc. – Trapper

Maybe its a platform issue (about how to handle pipes). I'm trying it on a linux machine. thanks for your answer anyway! – Cuthbertson

What if you remove the cat and use nc -l -p 9977 < $FIFO? – Northumbria

Answer 6

Finally, I have managed to find the tail process using ps. Thanks to the idea from ennuikiller.

I have used the ps to grep tail from the args and kill it. It is kind of a hack but it worked. :)

If you can find a better way please share.

Here is the complete script:
(Latest version can be found here: http://docs.karamatli.com/dotfiles/bin/logserver)

if [ -z "$1" ]; then
    echo Usage: $0 LOGFILE [PORT]
    exit -1
fi
if [ -n "$2" ]; then
    PORT=$2
else
    PORT=9977
fi

TAIL_CMD="tail -f $1"

function kill_tail {
    # find and kill the tail process that is detached from the current process
    TAIL_PID=$(/bin/ps -eo pid,args | grep "$TAIL_CMD" | grep -v grep | awk '{ print $1 }')
    kill $TAIL_PID
}
trap "kill_tail; exit 0" SIGINT SIGTERM

while true; do
    ( $TAIL_CMD & ) | nc -l -p $PORT -vvv
    kill_tail
done

Exocrine

Shouldn't the PID of the tail command be available in $! so that you could simply do kill $! instead of kill_tail? – Menticide

Answer 7

ncat automatically terminates tail -f on exit (on Mac OS X 10.6.7)!

# simple log server in Bash using ncat
# cf. http://nmap.org/ncat/
touch file.log
ncat -l 9977 -c "tail -f file.log" </dev/null   # terminal window 1
ncat localhost 9977 </dev/null                  # terminal window 2
echo hello > file.log                           # terminal window 3

Office

Answer 8

One way would be to simply do a ps -ef and grep for tail with your script ppid

Engaged

I couldn't get it using the ppid because it is detached when I fork it in a subshell. But I managed to grep it using the args param of the ps program. – Cuthbertson

Answer 9

Have you tried:

nc -l -p 9977 -c "tail -f $1"

(untested)

Or -e with a scriptfile if your nc doesn't have -c. You may have to have an nc that was compiled with the GAPING_SECURITY_HOLE option. Yes, you should infer appropriate caveats from that option name.

Tsimshian

Oh, I have never thought of it. I wish it worked :) But it's still not terminating because of the nature of "tail -f" – Cuthbertson

without the -f, its OK. But I want to serve it in real time. – Cuthbertson

Answer 10

You may store the pid of the tail command in a variable using Bash I/O redirections only (see How to get the PID of a process in a pipeline).

# terminal window 1
# using nc on Mac OS X (FreeBSD nc)
: > /tmp/foo
PID=$( { { tail -f /tmp/foo 0<&4 & echo $! >&3 ; } 4<&0 | { nc -l 9977 ;} & } 3>&1 | head -1 )
kill $PID

# terminal window 2
nc localhost 9977

# terminal window 3
echo line > /tmp/foo

Gurnard

Answer 11

Not an ideal answer, but I found a workaround for a logger daemon I worked on:

#!/bin/sh
tail -f /etc/service/rt4/log/main/current --pid=$$ | grep error

from $info tail:

--pid=PID
          with -f, terminate after process ID, PID dies

Catachresis

Answer 12

You could use the coproc command twice.

The given example translates to:

coproc TAIL { tail -f $1; }; exec {TAIL[1]}<&-
coproc NC { nc -v -l -p 9977; } <&"${TAIL[0]}" >&1
wait $NC_PID; echo "nc exit code: $!"
kill $TAIL_PID; echo "done"

(I've thrown a -v and a couple echo in there for troubleshooting.)

Using coproc feels a lot like using Popen() in various other scripting languages.

Deference

Closing stdin of tail with eg. </dev/null or exec as in the example is nice to do, so that no streams are left dangling around unconnected, but not strictly necessary for the tail command in particular, since tail never reads from stdin when given a filename. Could be left out when the first coproc command is tail. – Deference

Note that coproc is new in Bash 4 (e.g. OSX still uses 3.2). It is available in OSX's zsh though. – Klara

Answer 13

bobbogo answer works but requires a intermediary pid file.

You can leverage the process substitution feature >() which works like| but without waiting for tail to finish

tail -f $1 > >(nc -l -p 9977) & pid=!

Jermayne

Correction: pipe | also doesn't wait for tail to finish. The differences are a bit more interesting. – Balustrade

Answer 14

The --pid option to tail is your best friend here. It will allow you total control of the pipeline running in background. read the tail command options for more resilience in case your file is actively rotated by another process which might leave you tailing a inactive inode. The example below, though not used to process the data demonstrate the "imposed" restriction on the tail and the ability to tell it to exit at any time. This is used for measuring the service pressure on httpd .

  # Set the tail to die in 100 second even if we die unexpectedlly.
sleep 100 & ;  ctlpid=$!
tail -q -n 0 --follow=name --retry --max-unchanged-stats=1 --pid=$ctlpid -f  /var/log/httpd/access_log 2>/dev/null | wc –l > /tmp/thisSampleRate &
…. Do some other work
….  Can kill the pipe at any time by killing $ctlpid 
…. Calculate preassure if /tmp/thisSampleRate is ready

Higgledypiggledy