Ending tail -f started in a shell script

Question

I have the following.

1. A Java process writing logs to the stdout
2. A shell script starting the Java process
3. Another shell script which executes the previous one and redirects the log
4. I check the log file with the tail -f command for the success message.

Even if I have exit 0 in the code I cannot end the tail -f process.

Which doesn't let my script to finish. Is there any other way of doing this in Bash?

The code looks like the following.

function startServer() {
  touch logfile
  startJavaprocess > logfile &

  tail -f logfile | while read line 
  do
    if echo $line | grep -q 'Started'; then
      echo 'Server Started'
      exit 0
    fi
  done
}

Puglia

not really a good answer, but the tail will die on a broken pipe at the first output after your script has ended. – Pyrology

The problem is what happens if the java process doesn't write anything after starting, it seems that the tail is waiting for ever. – Puglia

Does it ever echo 'Server Started'? – Petuu

be careful too though, start your tail before you start the server (with a -n0 as not to read any old entries). Otherwise it might already be out of the 'scrollback' tail gives you (default is five lines I believe) – Pyrology

@Alex Brown, yes ! It echos 'Server Started'. And the loop is also finished. I verified it. – Puglia

@rangalo: if the server doesn't print anything, you can put a timeout on the read, read -t 30 line will wait thirty seconds after each line that's been read, and then, if there's nothing more to read, returns false (breaking out of the loop) – Pyrology

Is is a problem if I touch the file before redirecting the logs to it. I am editing the code with the touch command. – Puglia

See also "How to get the PID of a process that is piped to another process in Bash?" – Esse

Answer 1

The best answer I can come up with is this

1. Put a timeout on the read, tail -f logfile | read -t 30 line
2. Start tail with --pid=$$, that way it'll exit when the bash-process has finished.

It'll cover all cases I can think of (server hangs with no output, server exits, server starts correctly).

Dont forget to start your tail before the server.

tail -n0 -F logfile 2>/dev/null | while read -t 30 line

the -F will 'read' the file even if it doesn't exist (start reading it when it appears). The -n0 won't read anything already in the file, so you can keep appending to the logfile instead of overwriting it each time, and to standard log rotation on it.

EDIT:
Ok, so a rather crude 'solution', if you're using tail. There are probably better solutions using something else but tail, but I got to give it to you, tail gets you out of the broken-pipe quite nicely. A 'tee' which is able to handle SIGPIPE would probably work better. The java process actively doing a file system drop with an 'im alive' message of some sort is probably even easier to wait for.

function startServer() {
  touch logfile

  # 30 second timeout.
  sleep 30 &
  timerPid=$!

  tail -n0 -F --pid=$timerPid logfile | while read line 
  do
    if echo $line | grep -q 'Started'; then
      echo 'Server Started'
      # stop the timer..
      kill $timerPid
    fi
  done &

  startJavaprocess > logfile &

  # wait for the timer to expire (or be killed)
  wait %sleep
}

Pyrology

it didn't work, the $$ was bash -hB process which is always running – Puglia

I see, I figured you were running a script to start your server, not that you had one doing both start and stop. But anyway, that should be solvable... I'll give it some thought and get back to you – Pyrology

I am currently not on the system, but soon give it a try and let you know. – Puglia

Thanks. This works, but only if I start the script before the tail and loop. Otherwise i only get the timeout no grep. The other disadvantage is that I get output from bash for kill and if wait because the sleep has already died. – Puglia

So far this is the most suitable answer for me. – Puglia

Answer 2

Based on the answers I found here, this is what I've come up with.

It directly deals with tail and kills it once we've seen the needed log output. Using 'pkill -P $$ tail' should ensure that the right process is killed.

wait_until_started() {
    echo Waiting until server is started
    regex='Started'
    tail logfile -n0 -F | while read line; do
            if [[ $line =~ $regex ]]; then
                    pkill -9 -P $$ tail
            fi
    done
    echo Server is started
}

Bradstreet

This is a gem! Really looked all over the internet for this. I believe this should be the accepted answer :) – Dichasium

Update: Actually, though this works ok in the command line, somehow it doesn't work as good when run over SSH. What I did is replaced pkill -9 -P -$$ tail with killall tail. A little crude, but does the job. – Dichasium

Answer 3

According to the tail man page, you can get tail to terminate after the a process dies

In BASH, you can get the PID of the last started background process using $! SO if you're using bash:

tail -f --pid=$! logfile

Sphygmograph

The problem is the script is waiting. I do the same trick for the shutdown script and it works because the shutdown script finishes. But the startup script hangs on. – Puglia

This will not stop the tail until the server stops, which might be years from now. Use $$ instead, it'll trigger when the script exits. – Pyrology

@roe, can you please elaborate a bit ? My problem is that neither server nor the script is terminating – Puglia

Answer 4

I have had a similar situation where I need to tail a log for a "started" message within a reasonable time and if it is not found during that time I need to exit. Here is what I ended up doing.

wait_tomcat_start(){
WAIT=60
echo "Waiting for Tomcat to initialize for $WAIT seconds"

# Tail log file, do a while read loop with a timeout that checks for desired log status,
# if found kill the find and break the loop. If not found within timeout: the read -t will
# kill the while read loop and bounce to the OR statement that will in turn kill the tail 
# and echo some message to the console.
tail -n0 -f $SERVERLOG | while read -t $WAIT LINE || (pkill -f "tail -n0 -f" && echo "Tomcat did not start in a timely fashion! Please check status of tomcat!!!")
do
        echo "$LINE"
        [[ "${LINE}" == *"Server startup in"* ]] && pkill -f "tail -n0 -f" && break
done
}

I’m not sure this is very elegant or even the best way to do it, but it works good enough for me. I'd be happy for any opinions :)

Sorilda

Answer 5

Capture the pid of the background process

pid=$!

Use tail's --pid=PID option, so that it terminates after the process having pid $PID terminates.

Mckinney

Answer 6

Rather than exiting the process, you can instead find the process ID of the tail -f process and kill it (a kill -9 would even be safe here if you're sure the log file has finished).

That way, the while read line will terminate naturally and you won't need to exit.

Or, since you're not really using the tail to output to the screen, you could also try the more old-school:

grep -q 'Started' logfile
while [[ $? -ne 0 ]] ; do
    sleep 1
    grep -q 'Started' logfile
done

Shoestring

There might be 'Started' from earlier sessions in there, so I wouldn't bet on your old-school version – Pyrology

No, there won't be because the java process is overwriting the logfile, not appending to it. – Shoestring

right, I guess I'm not paying attention.. cleaner would be while ! grep -q 'Started' logfile; do sleep 1; done – Pyrology

Answer 7

How about using an infinite loop instead of the -f command-line option for tail?

function startServer() {
  startJavaprocess > logfile &

  while [ 1 ]
  do
   if tail logfile | grep -q 'Started'; then
    echo 'Server started'
    exit 0
   fi
  done
}

Gamali

That's not bad but there's a risk you could miss the 'Started' line if, for example, it's written in between loops followed by enough lines to stop it showing up in the next tail. – Shoestring

If you want to eliminate the risk entirely you could grep in the entire file. – Gamali

Answer 8

I had the same problem, couldn't find simple and good solution. I'm not good in Python, but I managed somehow to solve this:

wait_log.py:

#!/usr/bin/env python

from optparse import OptionParser
import os
import subprocess
import time

def follow(file):
    def file_size(file):
        return os.fstat(file.fileno())[6]
    def is_newLine(line):
        return line != None and line.find("\n") != -1;

    file.seek(0, os.SEEK_END)

    while True:
        if file.tell() > file_size(file):
            file.seek(0, os.SEEK_END)

        line_start = file.tell()
        line = file.readline()

        if is_newLine(line):
            yield line
        else:
            time.sleep(0.5)
            file.seek(line_start)

def wait(file_path, message):
    with open(file_path) as file:
        for line in follow(file):
            if line.find(message) != -1:
                break

def main():
    parser = OptionParser(description="Wait for a specific message in log file.", usage="%prog [options] message")
    parser.add_option("-f", "--file", help="log file")

    (options, args) = parser.parse_args()

    if len(args) != 1:
        parser.error("message not provided")

    if options.file == None:
        parser.error("file not provided")

    wait(options.file, args[0])

if __name__ == "__main__":
    main()

Taster

Answer 9

Had a similar issue where the tail process wasnt getting killed when

1. Run through jsch
2. tail wasnt producing any output to jsch and hence to its output stream.

Used the --pid=$! to kill it and started a infinite while loop to echo something in the background before the tail which gets killed when the underlying process is killed and thus kills the tail.

( while true; do echo 'running';  sleep 5; done ) & ( tail -f --pid=$! log-file )

Capitation

Answer 10

My preferred solution for this problem is to put the 'tail' command and its consumer into a subshell, and let the filter logic kill the parent and its children (which includes the tail process). If you look at the process tree, it will be:

startServer (pid=101)
   startServer (pid=102) << This is the subshell created by using parens "(...)"
      tail -f logfile (pid=103) << Here's the tail process
      startServer (pid=104)     << Here's the logic that detects the end-marker

In this approach, the end-marker detection logic (pid 104) looks for its parent PID (102), and all of its children, and kills the whole batch -- including itself. Then the grandparent (pid 101 above) is free to continue.

function startServer() {
  touch logfile
  startJavaprocess > logfile &

  tail -f logfile | while read line 
  do
    if echo $line | grep -q 'Started'; then
      echo 'Server Started'
      mypid=$BASHPID
      pipeParent=$(awk '/^PPid/ {print $2}' /proc/$mypid/status)
      kill -TERM $pipeParent $(pgrep -P $pipeParent)  # Kill the subshell and kids
    fi
  done
}

# To invoke startServer(), add a set of parens -- that puts it in a subshell:
(startServer())

Got

Answer 11

It is possible to background tail -f logfile, send tailpid to the while read loop subshell and implement a trap on EXIT to kill the tail command.

( (sleep 1; exec tail -f logfile) & echo $! ; wait) | (
  trap 'trap - EXIT; kill "$tailpid"; exit' EXIT
  tailpid="$(head -1)"
  while read line 
  do
    if echo $line | grep -q 'Started'; then
      echo 'Server Started'
      exit 0
    fi
  done
)

Careen

Answer 12

Don't use tail - you can get the same 'monitor the newest thing in the file' using read.

Here I use a FIFO instead of the log file:

function startServer() {
  mkfifo logfile
  startJavaprocess > logfile &

  a=""; while [ "$a" != "Started" ]; do read <logfile a; done

  echo "Server Started"
}

Note that this leaves a FIFO hanging around.

Petuu

This will kill the java process (broken pipe) when the fifo is removed, or in the second example when the loop exits. Probably not what you intended. – Pyrology

true, I had spotted the terminate in the second instance, but not the first. simple fix for the first, I'll remove the second until I think of a way around that. – Petuu

Answer 13

This should work and tail should die once the sub shell dies

function startServer() {
  touch logfile
  startJavaprocess &gt logfile &

  while read line 
  do
    if echo $line | grep -q 'Started'; then
      echo 'Server Started'
      exit 0
    fi
  done &lt &lt(tail -f logfile)
}

Try this:

function startServer() {
  while read line 
  do
    if echo $line | grep -q 'Started'; then
      echo 'Server Started'
      return 0
    fi
  done &lt &lt(startJavaprocess | tee logfile)
}

Revelry

Answer 14

Using tail -n0 -f piped to grep is indeed a nice solution, and indeed the first process in the pipe will die when it tries to output to a dead grep process.

But if you're hunting for text that appears near to the last current output of the tail, then grep will already have read the whole input from the tail (in one block) and therefore there won't be any more text output in the log that needs sending down the pipe as grep already read it before it quit (or maybe it was already in the pipe buffer) - at least this is my understanding.

Using -m1 option on grep looks like it'd do exactly what you want and leave the input immediately after the line it matched, but it didn't seem to make a difference or help me in my search for similar functionality. I suspect the pipe buffer still holds all the text output from tail, or some other reason for tail not to have anything left to output. You wanted this post-grep-match text still left to be output next, because its what would kill your tail when it tried (still risky - what happens if its the last line for some reason?), and return control to the calling script.

I found one way round it is to output anything into the end of the log file once the grep has quit; ie.

tail -f logfile | ( grep -q ; echo >> logfile)

I have a theory that (if my guess is right) you could force the pipe to be less buffered to make it work without this, or maybe that adding a huponexit setting command to the appropriate pipe component - i.e. in (probably curly) brackets would help; but I didn't care about appending a blank line to the logfile and it worked ok and its only a smaller test script (so not a long lived logfile that needs to stick to a format for other processing).

shopt -s huponexit would be useful but for the subshell-ness of it.

PS my first post here, would liked to do it as a comment to existing answer rather than re-iterate stuff, but I don't think I can now.

Jabalpur

PS this doesn't rely on too much bash-specific-ness - "<(file)" is not available in older ksh/other shell versions. – Jabalpur

Answer 15

For the original question, why the exit command didn't quit, I got the same issue and finally found the cause.

By using the debug mode of bash, I can see the exit command was called but the process still hung until one more line flush to the log file just after the 'Started'. The wierd thing is when 'Started' came out, even exit had been called, the process is still hooked by something. I guess it was tail -f, until one more line come out, it will really release the hook.

So if you print one more line after it's started, your monitor will quit straight away.

Fetid

Answer 16

Using a combination of answers, I came up with this simple solution. This example calls Tomcat's startup.sh script and then tails the catalina.out log until "Server startup" is logged, and then it stops tailing.

#!/bin/bash

function logUntilStarted() {
    tail -n0 -F /home/tomcat/logs/catalina.out | while read line; do
        if echo $line && echo $line | grep -q 'Server startup' ; then
            pkill -9 -P $$ tail > /dev/null 2>&1
        fi
    done
}

/home/tomcat/bin/startup.sh
logUntilStarted

Geller

Answer 17

Run the previous command with nohup.

In my case, Run java -jar with nohup,such as

nohup java -jar trade.jar xx.jar &

there will no log output,but a new "nohup.out" will created. The original log file trade.log works as well.

Then , tail -f trade.log, the shell will show log info , Ctrl-c can interrupt it ,return to shell.

Pb

the question (and my google search) are looking for ways to terminate that tail -f when the log file is done without shell access. – Symons

Answer 18

tail -n0 --pid=$(($BASHPID+1)) -F logfile | sed -n '/Started/{s/.*/Server Started/p; q}'

When piping, PIDs are sequential, so the pid of the tail will be $BASHPID and the pid of the sed will be $BASHPID+1. The --pid switch will cause tail to exit (properly!) when the sed command quits. This sed command will look for /Started/ and then substitute the whole line (.*) with "Server Started", then quit.

Bisulfate