Linux: Block until a string is matched in a file ("tail + grep with blocking")

Question

Is there some one-line way in bash/GNU tools to block until there's a string matched in a file? Ideally, with timeout. I want to avoid multi-line loop.

Update: Seems like I should have emphasize that I want the process to end when the string is matched.

Disseisin

Answer 1

Thanks both for answers, but the important part was that the process blocks until found, then ends. I found this:

grep -q 'PATTERN' <(tail -f file.log)

-q is not much portable, but I will only use Red Hat Enterprise Linux so it's ok. And with timeout:

timeout 180 grep -q 'PATTERN' <(tail -f file.log)

Disseisin

Not sure why you say that -q isn't portable, it is specified by POSIX. – Militarism

@Militarism The man page for GNU grep suggests that portable scripts should avoid -q, simply because some older flavours of Unix don't have it. They suggest redirecting to /dev/null; you might use grep -m1 'PATTERN' <(tail -f file.log) >/dev/null if this is a concern. – Koski

The problem of this solution is that the process 'tail' will still alive even after grep exit. – Galbanum

@Galbanum How to avoid that? – Zennie

Use grep -q 'PATTERN' <(timeout 10 tail -f file.log) so that the tail process terminates, while including a timeout. – Foliation

Answer 2

I make a variant with sed instead of grep, printing all lines parsed.

sed '/PATTERN/q' <(tail -n 0 -f file.log)

The script is in https://gist.github.com/2377029

Bergamot

Answer 3

Take a look at the --max-count option:

tail -f file.log | grep -m 1 'PATTERN'

It will exit after the first line that matches PATTERN.

---

EDIT: take note of @Karoly's comment below. If the file.log velocity is slow, it's possible that the grep process will block until additional content is added to the file after the matching line.

echo 'context PATTERN line' >> file.log  ## grep shows the match but doesn't exit

will print the matching line, but it will not exit until additional content is appended to the file (even if it doesn't have a newline yet):

echo -n ' ' >> file.log  ## Now the grep process exits

In some cases (such as a high-velocity log file), this isn't a big deal, because new content is probably going to be added to the file soon anyway.

Also note that this behavior does not happen when reading from a console as stdin, so it appears to be a difference in the way grep reads from a pipe:

$ grep -m1 'PATTERN' -      # manually type PATTERN and enter, exits immediately
$ cat | grep -m1 'PATTERN'  # manually type PATTERN and enter, and it hangs

Cupro

it doesn't exit - you first have to wait till something is written in the file... – Vapid

Yes, that was a requirement in the original question: "Block until a string is matched"... The -m 1 argument causes it to exit after the first match of PATTERN – Cupro

yes, intuitively that's how it should work. unfortunately first you need extra lines to appear in the log file to trigger the actual exit. test it yourself. I see in the grep output PATTERN highlighted, but the process doesn't exit. Note: this is with bash, your shell might be more clever. – Vapid

Indeed, looks like you are correct. I was testing with something that had a high velocity of logs (android logcat, IIRC), so I didn't notice the hang. I'm not sure why grep works that way, because the man page says it should stop immediately when it hits the max-count. – Cupro

@Cupro This subtle difference described in your answer/comments confused me for days. Is there any other way to IMMEDIATELY exit from a stream with sleep infinity in the end of script if keyword is grep ed? – Dislocation

This was the best answer so far. – Gobo

Answer 4

tail -f file | grep word | head -n1

Will post snip with async timeout

For now: How to include a timer in Bash Scripting?

The linked answer defines a 'run_or_timeout' function that does what you are looking for in a very bash-savvy way

Haw

The -1 syntax is discouraged, IIRC. Use -n 1. – Hwahwan

This way, tail does not end when the string is found. – Zennie

That's a surprise to me. Will need to check later – Haw

@Ondra Žižka okay... grep --line-buffered is one part of the equation. Also, it appears now as if running head -n0 will abort on the next line after the first match. I'll try to figure more out later – Haw

Answer 5

$ tail -f path | sed /pattern/q

or, if you want to suppress the output of non-matching lines:

$ tail -f path | sed -n '/pattern/{p; q;}'

A simple-minded way to add a timeout is to do:

$ cmd& sleep 10; kill $! 2> /dev/null

(Suppress the errors from the kill so that if the process terminates before the time expires, you don't get the "No such process" warning). Note that this is not at all robust, since it is possible that cmd will terminate and the pid count will wrap around and some other command will have that pid by the time the timer expires.

Lope

This works, but sed's buffer (or something else) causes it to need one extra line to trigger, which may never come (this line tends to be the last one in the log). – Zennie

And the sed in the second command gives me 'unterminated adress of regex' error. – Zennie

@Ondra Žižka: so the extra line required is the same effect as what I'm up to a.t.m. -- I need to be leaving now – Haw

Answer 6

wait for file to appear

while [ ! -f /path/to/the.file ] 
do sleep 2; done

wait for string to apper in file

while ! grep "the line you're searching for" /path/to/the.file  
do sleep 10; done

https://superuser.com/a/743693/129669

Strong

Answer 7

I had a similar requirement and came up with the following.

The one-liner you are after is the line which starts with "timeout ...." and the rest of the code is the prep work needed to provide the one-liner with the information it will need and to clean up afterwards.

##
## Start up the process whose log file we want to monitor for a specific pattern.
##
touch file_to_log_nohup_output.log
nohup "some_command" "some_args" >> file_to_log_nohup_output.log 2>&1 &
my_cmd_pid=$!


## Specify what our required timeout / pattern and log file to monitor is
my_timeout=10m
my_logfile="/path/to/some_command's/log/file.txt"
my_pattern="Started all modules."


## How does this work?
## - In a bash sub shell, started in the background, we sleep for a second and
##   then execute tail to monitor the application's log file.
## - Via the arguments passed to it, tail has been configured to exit if the
##   process whose log file it is monitoring dies.
## - The above sub shell, is executed within another bash sub shell in which
##   we identify the process id of the above sub shell and echo it to stdout.
## - Lastly, in that sub shell we wait for the sub shell with tail running in
##   it as a child process, to terminate and if it does terminate, we redirect
##   any output from its stderr stream to /dev/null.
## - The stdout output of the above sub shell is piped into another sub shell
##   in which we setup a trap to watch for an EXIT event, use head -1 to read
##   the process id of the tail sub shell and finally start a grep process
##   to grep the stdout for the requested pattern. Grep will quit on the first
##   match found. The EXIT trap will kill the process of the tail sub shell
##   if the sub shell running grep quits.
##
## All of this is needed to tidy up the monitoring child processes for
## tail'ing + grep'ing the application log file.
##
## Logic of implementing the above sourced from: http://superuser.com/a/1052328


timeout ${my_timeout} bash -c '((sleep 1; exec tail -q -n 0 --pid=$0 -F "$1" 2> /dev/null) & echo $! ; wait $! 2>/dev/null ) | (trap "kill \${my_tail_pid} 2>/dev/null" EXIT; my_tail_pid="`head -1`"; grep -q "$2")' "${my_cmd_pid}" "${my_logfile}" "${my_pattern}" 2>/dev/null &


##
## We trap SIGINT (i.e. when someone presses ctrl+c) to clean up child processes.
##
trap 'echo "Interrupt signal caught. Cleaning up child processes: [${my_timeout_pid} ${my_cmd_pid}]." >> "file_to_log_nohup_output.log"; kill ${my_timeout_pid} ${my_cmd_pid} 2> /dev/null' SIGINT
wait ${my_timeout_pid}
my_retval=$?
trap - SIGINT


## If the time out expires, then 'timeout' will exit with status 124 otherwise
## it exits with the status of the executed command (which is grep in this
## case).
if [ ${my_retval} -eq 124 ]; then
    echo "Waited for [${my_timeout}] and the [${my_pattern}] pattern was not encountered in application's log file."
    exit 1
else
    if [ ${my_retval} -ne 0 ]; then
        echo "An issue occurred whilst starting process. Check log files:"
        echo "  * nohup output log file: [file_to_log_nohup_output.log]"
        echo "  * application log file: [${my_logfile}]"
        echo "  * application's console log file (if applicable)"
        exit 1
    else
        info_msg "Success! Pattern was found."
        exit 0
    fi
fi

I've implemented the above into a stand alone script which can be used to run a command wait for its log file to have the required pattern, with a timeout.

Available here: run_and_wait.sh

Balm