Trying to find a simple way for watching for recent events (from less than 10 minutes), I've tried this:
awk "/^$(date --date="-10 min" "+%b %_d %H:%M")/{p++} p" /root/test.txt
but it doesn't work as expected...
Log files are in form :
Dec 18 09:48:54 Blah
Dec 18 09:54:47 blah bla
Dec 18 09:55:33 sds
Dec 18 09:55:38 sds
Dec 18 09:57:58 sa
Dec 18 09:58:10 And so on...
You can match the date range using simple string comparison, for example:
d1=$(date --date="-10 min" "+%b %_d %H:%M")
d2=$(date "+%b %_d %H:%M")
while read line; do
[[ $line > $d1 && $line < $d2 || $line =~ $d2 ]] && echo $line
done
For example if d1='Dec 18 10:19' and d2='Dec 18 10:27' then the output will be:
Dec 18 10:19:16
Dec 18 10:19:23
Dec 18 10:21:03
Dec 18 10:22:54
Dec 18 10:27:32
Or using awk if you wish:
awk -v d1="$d1" -v d2="$d2" '$0 > d1 && $0 < d2 || $0 ~ d2'
[[ "Jan 18 10:27:32" < "Feb 18 10:27:32" ]] && echo "older" and [[ "Feb 18 10:27:32" < "Mar 18 10:27:32" ]] && echo "older" give inconsistent results. –
Belenbelesprit date function by using printf "%(...)T" -1. This is a lot quicker than using a fork to date. For using endianess property of date, you could correct month entry by using associative array. See my answer! –
Purpleness Here is nice tool range is any you wish from -10 till now
sed -n "/^$(date --date='10 minutes ago' '+%b %_d %H:%M')/,\$p" /var/log/blaaaa
2017-06-28 14:00 so I modified the sed command as follows: sed -n "/^$(date --date="60 minutes ago" "+%Y-%m-%d %H:%M")/,\$p" /var/log/blah.log to get the last 60 minutes of the log –
Tryptophan This answer is something long, because there is 3 different way on thinking: 1) perl quick or exact, 2) pure bash and 3) perl script in bash function.
Simple and efficient:
perl -MDate::Parse -ne 'print if/^(.{15})\s/&&str2time($1)>time-600' /path/log
This version print last 10 minutes event, upto now, by using time function.
You could test this with:
sudo cat /var/log/syslog |
perl -MDate::Parse -ne '
print if /^(\S+\s+\d+\s+\d+:\d+:\d+)\s/ && str2time($1) > time-600'
Note that first representation use only firsts 15 chars from each lines, while second construct use more detailed regexp.
As a perl script: last10m.pl
#!/usr/bin/perl -wn
use strict;
use Date::Parse;
print if /^(\S+\s+\d+\s+\d+:\d+:\d+)\s/ && str2time($1) > time-600
Meaning not relative to current time, but to last entry in logfile:
There is two way for retrieving end of period:
date -r logfile +%s
tail -n1 logfile | perl -MDate::Parse -nE 'say str2time($1) if /^(.{15})/'
Where logically, last modification time of the logfile must be the time of the last entry.
So the command could become:
perl -MDate::Parse -ne 'print if/^(.{15})\s/&&str2time($1)>'$(
date -r logfile +%s)
or you could take the last entry as reference:
perl -MDate::Parse -E 'open IN,"<".$ARGV[0];seek IN,-200,2;while (<IN>) {
$ref=str2time($1) if /^(\S+\s+\d+\s+\d+:\d+:\d+)/;};seek IN,0,0;
while (<IN>) {print if /^(.{15})\s/&&str2time($1)>$ref-600}' logfile
Second version seem stronger, but access to file only once.
As a perl script, this could look like:
#!/usr/bin/perl -w
use strict;
use Date::Parse;
my $ref; # The only variable I will use in this.
open IN,"<".$ARGV[0]; # Open (READ) file submited as 1st argument
seek IN,-200,2; # Jump to 200 character before end of logfile. (This
# could not suffice if log file hold very log lines! )
while (<IN>) { # Until end of logfile...
$ref=str2time($1) if /^(\S+\s+\d+\s+\d+:\d+:\d+)/;
}; # store time into $ref variable.
seek IN,0,0; # Jump back to the begin of file
while (<IN>) {
print if /^(.{15})\s/&&str2time($1)>$ref-600;
}
There is a very quick pure bash script:
Warning: This use recent bashisms, require $BASH_VERSION 4.2 or higher.
#!/bin/bash
declare -A month
for i in {1..12};do
LANG=C printf -v var "%(%b)T" $(((i-1)*31*86400))
month[$var]=$i
done
printf -v now "%(%s)T" -1
printf -v ref "%(%m%d%H%M%S)T" $((now-600))
while read line;do
printf -v crt "%02d%02d%02d%02d%02d" ${month[${line:0:3}]} \
$((10#${line:4:2})) $((10#${line:7:2})) $((10#${line:10:2})) \
$((10#${line:13:2}))
# echo " $crt < $ref ??" # Uncomment this line to print each test
[ $crt -gt $ref ] && break
done
cat
Store this script and run:
cat >last10min.sh
chmod +x last10min.sh
sudo cat /var/log/syslog | ./last10min.sh
Simply replace line 10, but you have to place filename in the script and not use it as a filter:
#!/bin/bash
declare -A month
for i in {1..12};do
LANG=C printf -v var "%(%b)T" $(((i-1)*31*86400))
month[$var]=$i
done
read now < <(date -d "$(tail -n1 $1|head -c 15)" +%s)
printf -v ref "%(%m%d%H%M%S)T" $((now-600))
export -A month
{
while read line;do
printf -v crt "%02d%02d%02d%02d%02d" ${month[${line:0:3}]} \
$((10#${line:4:2})) $((10#${line:7:2})) $((10#${line:10:2})) \
$((10#${line:13:2}))
[ $crt -gt $ref ] && break
done
cat
} <$1
As commented by ajcg, this could be nice to put efficient perl script into a bash function:
recentLog(){
perl -MDate::Parse -ne '
print if/^(.{'${3:-15}'})\s/ &&
str2time($1)>time-'$((
60*${2:-10}
)) ${1:-/var/log/daemon.log}
}
Usage:
recentLog [filename] [minutes] [time sting length]
filename of log fileminutes max before now of lines to showtime sting length from begin of lines (default 15)../last10min.sh: line 6: printf: (': invalid format character ./last10min.sh: line 7: month[$var]: bad array subscript ./last10min.sh: line 9: printf: (': invalid format character ./last10min.sh: line 10: printf: (': invalid format character ./last10min.sh: line 17: [: 1809485400: unary operator expected ./last10min.sh: line 17: [: 1809544700: unary operator expected ./last10min.sh: line 17: [: 1809553300: unary operator expected` –
Callaghan tail -n1 logfile | perl -MDate::Parse -nE 'say str2time($1) if /^(.{15})/' or date -r logfile +%s. –
Purpleness Date::Parse –
Copperas You can match the date range using simple string comparison, for example:
d1=$(date --date="-10 min" "+%b %_d %H:%M")
d2=$(date "+%b %_d %H:%M")
while read line; do
[[ $line > $d1 && $line < $d2 || $line =~ $d2 ]] && echo $line
done
For example if d1='Dec 18 10:19' and d2='Dec 18 10:27' then the output will be:
Dec 18 10:19:16
Dec 18 10:19:23
Dec 18 10:21:03
Dec 18 10:22:54
Dec 18 10:27:32
Or using awk if you wish:
awk -v d1="$d1" -v d2="$d2" '$0 > d1 && $0 < d2 || $0 ~ d2'
[[ "Jan 18 10:27:32" < "Feb 18 10:27:32" ]] && echo "older" and [[ "Feb 18 10:27:32" < "Mar 18 10:27:32" ]] && echo "older" give inconsistent results. –
Belenbelesprit date function by using printf "%(...)T" -1. This is a lot quicker than using a fork to date. For using endianess property of date, you could correct month entry by using associative array. See my answer! –
Purpleness In bash, you can use the date command to parse the timestamps. The "%s" format specifier converts the given date to the number of seconds since 1970-01-01 00:00:00 UTC. This simple integer is easy and accurate to do basic arithmetic on.
If you want the log messages from the last 10 minutes of actual time:
now10=$(($(date +%s) - (10 * 60)))
while read line; do
[ $(date -d "${line:0:15}" +%s) -gt $now10 ] && printf "$line\n"
done < logfile
Note the ${line:0:15} expression is a bash parameter expansion which gives the first 15 characters of the line, i.e. the timestamp itself.
If you want the log messages from the last 10 minutes relative to the end of the log:
$ lastline=$(tail -n1 logfile)
$ last10=$(($(date -d "$lastline" +%s) - (10 * 60)))
$ while read line; do
> [ $(date -d "${line:0:15}" +%s) -gt $last10 ] && printf "$line\n"
> done < logfile
Dec 18 10:19:16
Dec 18 10:19:23
Dec 18 10:21:03
Dec 18 10:22:54
Dec 18 10:27:32
$
Here's a mild performance enhancement over the above:
$ { while read line; do
> [ $(date -d "${line:0:15}" +%s) -gt $last10 ] && printf "$line\n" && break
> done ; cat ; } < logfile
Dec 18 10:19:16
Dec 18 10:19:23
Dec 18 10:21:03
Dec 18 10:22:54
Dec 18 10:27:32
$
This assumes the log entries are in strict chronological order. Once we match the timestamp in question, we exit the for loop, and then just use cat to dump the remaining entries.
[pid 26481 on' ./test.sh: line 8: [: -gt: unary operator expected date: invalid date [pid 17563 on' ./test.sh: line 8: [: -gt: unary operator expected Dec 18 10:19:16 Dec 18 10:19:23 Dec 18 10:21:03 Dec 18 10:22:54 Dec 18 10:27:32 –
Callaghan date can't parse the passed date string correctly. What output do you get from date --date="Dec 18 10:19:16"? What output do you get from date --version? Which version of centos are you using? I just installed 6.5 in a VM and it is working for me. –
Belenbelesprit tail -n1 /root/test.txt" +%s date: invalid date `Dec 02 15:44:57 vv' –
Callaghan ${line:0:15} gives the first 15 characters of $line. gnu.org/software/bash/manual/html_node/… –
Belenbelesprit date gets invoked for every line. The python solution is likely more performance-optimal, so long as it doesn't have to spawn external processes for each iteration. –
Belenbelesprit paste <(sed 's/^\(.\{15\}\).*$/\1/' logfile | date -f - +%s ) logfile , use the commande time myscript.sh to evaluate the cost of your script... –
Purpleness date can take a -f parameter. –
Belenbelesprit In python, you could do as follows:
from datetime import datetime
astack=[]
with open("x.txt") as f:
for aline in f:
astack.append(aline.strip())
lasttime=datetime.strptime(astack[-1], '%b %d %I:%M:%S')
for i in astack:
if (lasttime - datetime.strptime(i, '%b %d %I:%M:%S')).seconds <= 600:
print i
Put the lines from the file into a stack (a python list). pop the last item and get difference between the successive date items until you get the difference as less than 600 seconds.
Running on your input, I get the following:
Dec 18 10:19:16
Dec 18 10:19:23
Dec 18 10:21:03
Dec 18 10:22:54
Dec 18 10:27:32
You can pass days, hours, minutes or seconds as a parameter and it will search for the expression and on the file specified (or directory, in which case it will append '/*' to the name):
In your case just call the script like so: $0 -m 10 "expression" log_file
Note: Also if you know the location of 'ruby' change the shebang (first line of the script), for security reasons.
#! /usr/bin/env ruby
require 'date'
require 'pathname'
if ARGV.length != 4
$stderr.print "usage: #{$0} -d|-h|-m|-s time expression log_file\n"
exit 1
end
begin
total_amount = Integer ARGV[1]
rescue ArgumentError
$stderr.print "error: parameter 'time' must be an Integer\n"
$stderr.print "usage: #{$0} -d|-h|-m|-s time expression log_file\n"
end
if ARGV[0] == "-m"
gap = Rational(60, 86400)
time_str = "%b %d %H:%M"
elsif ARGV[0] == "-s"
gap = Rational(1, 86400)
time_str = "%b %d %H:%M:%S"
elsif ARGV[0] == "-h"
gap = Rational(3600, 86400)
time_str = "%b %d %H"
elsif ARGV[0] == "-d"
time_str = "%b %d"
gap = 1
else
$stderr.print "usage: #{$0} -d|-h|-m|-s time expression log_file\n"
exit 1
end
pn = Pathname.new(ARGV[3])
if pn.exist?
log = (pn.directory?) ? ARGV[3] + "/*" : ARGV[3]
else
$stderr.print "error: file '" << ARGV[3] << "' does not exist\n"
$stderr.print "usage: #{$0} -d|-h|-m|-s time expression log_file\n"
end
search_str = ARGV[2]
now = DateTime.now
total_amount.times do
now -= gap
system "cat " << log << " | grep '" << now.strftime(time_str) << ".*" << search_str << "'"
end
© 2022 - 2024 — McMap. All rights reserved.