How to add date string to each line of a continuously written log file

Question

Having a long running program that continuously writes to a logfile - how is it possible, disregarding any buffering issues, to add a date string to each line written to that file using a linux script?

I would imagine something like this:

tail -f logfile | ADD_DATE_TO_EACH_LINE > logfile2

The input would be something like that:

abc
def
ghi
jkl

The output should be similar to that:

2011-06-16 18:30:59 abc
2011-06-16 18:31:00 def
2011-06-16 18:35:21 ghi
2011-06-16 18:40:15 jkl

Pie

Answer 1

With perl:

command 2>&1 | perl -pe 'print scalar(localtime()), " ";'

With gawk:

command 2>&1 | awk '{ print strftime(), $0; fflush() }'

Replace command with tail -f logfile for your specific example. Or, perhaps you could just redirect the original program's stdout/stderr to the above pipe.

Symbolism

The benefit you get with this answer is that it doesn't spawn a new date process with each line, but both work well. – Symbolism

perl -pne ?? You don't want -n as well as -p. – Ogawa

Thanks! I don't really know perl. I just cut and paste other crud from the internet. :) Last time I used perl for real was over a decade ago! I usually use the gawk provided answer above, but included the perl one too in case the OP was not on linux. – Symbolism

@StevePrentice When I do the first command, it hangs. I'm running tail -f test-text.rtf 2>&1 | perl -pe 'print scalar(localtime()), " ";' > test-text.rtf Any idea why? – Kain

Answer 2

Try

tail -f logfile | while read line; do echo `date` "$line" ; done

Zaffer

+1 Just one note though: using date +"%y-%m-%d %H:%M:%S" instead of simply date would result in the exact output he expects; and you could also add the redirect to the output file in the end. (It's actually two notes) :) – Xiomaraxiong

Thank you, that works. And if you add " signs to $line the shell will not expand * signs to the directory content ;) – Pie

the problem with this solution is : when the input buffer given to the 'while read line' loop is too big, some lines will be truncate. – Remitter

Answer 3

You can try this

cat /etc/motd | xargs -d"\n" -I {} date +"%Y-%m-%d %H:%M:%S {}"

Example output:

2013-02-26 15:13:57 
2013-02-26 15:13:57 The programs included with the Debian GNU/Linux system are free software;
2013-02-26 15:13:57 the exact distribution terms for each program are described in the
2013-02-26 15:13:57 individual files in /usr/share/doc/*/copyright.
2013-02-26 15:13:57 
2013-02-26 15:13:57 Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
2013-02-26 15:13:57 permitted by applicable law.

Knecht

@Vouze You can fix that with sed in between, or actually rather instead of the useless use of cat; sed 's/%/%%/g' /etc/motd | xargs ... – Rinehart

Answer 4

A tool exists for that exact purpose, it’s ts (see man ts)

For example, with your logfile:

tail -f logfile | ts '%Y-%m-%d %H:%M:%S'

Also works with any software writing on the standard output, of course:

./blabla | ts '%Y-%m-%d %H:%M:%S'

If needed, you can add sub-second precision, with %.S instead of %S

Errand

ts is in the package moreutils on Ubuntu, and similar, installed via sudo apt install moreutils. – Bollen

Answer 5

A little lengthy, but here is what I came up with:

tail -f logfile | sed -u 's/%/%%/g' | xargs -I {} date +"%Y-%m-%d %H:%M:%S {}"

Memento

That's a very good solution. But what's the sed -u 's/%/%%/g' for? Do the % signs have a special meaning for xargs? – Pie

@Pie This method works by putting the entire line into the format string for date, where % has a special meaning. For example if the line in the log contained %Y then it would get replaced by the year, but changing it to %%Y with sed will correctly output %Y. – Memento

I have one additional remark: If there are single quotes (') in the input stream xargs terminates with following error message: xargs: unmatched single quote; by default quotes are special to xargs unless you use the -0 option therefore I changed the command to tail -f logfile | sed -u 's/%/%%/g' | tr "\n" "\0" | xargs -0 -I {} date +"%Y-%m-%d %H:%M:%S {}". – Pie

Answer 6

Can you configure the long running program to write it's output to the standard output and not to the logfile? In this case it would be easy to pipe the output to a script that first writes the current timestamp and then the entry.

If that is impossible, it may help to periodically (e.g. every second) read the logfile content, copy each line to another file (adding the current timestamp) and then deleting the logfile. This may, however, impose losing logfile entries that are written between reading and deleting the file :(

Fluorosis

I cannot configure the program to log to standard output because it is (resp. can be used as) an interactive program. But actually executing tail -f is in my opinion more or less the same as directly writing to stdout. Actually I'm looking for that "script that first writes the current timestamp and then the entry". – Pie

you could use a FIFO then. The Program will write everything to the named pipe (fifo) and you can read from it the same time, append the date and write to the real logfile. – Antitank

Answer 7

Or you can use python ...

cat /dev/urandom | python -c "from __future__ import print_function; import sys; import datetime; map(lambda x: print(datetime.datetime.now(), x), [line for line in sys.stdin.readlines()])"

Or use gnu screen

screen -a ls -lh -L -U -X command

First you need to enable logging and timestamp on your ~/.screenrc.

logfile /tmp/screen-%S-%n.log
logtstamp on

Marci

Answer 8

I've seen elsewhere a suggestion, if timeliness is important, that one should avoid calling other programs from BASH and use a built-in:

printf '%(%F %T)T\n'

%T is the same format strings from date (see man date), AFAICT. Example:

output="Err, Miss Crabapple, I just ate my worm!"; \
printf '\n%(%F %s)T'; printf "%s \n" " $output"

will output:

2021-03-28 1616894012 Err, Miss Crabapple, I just ate my worm!

I've just noticed a comment on that link has a suggestion to use gawk for timestamp top speeds.

Bollen