How to correctly parse text file using rsyslog and imfile

Question

Good day

I want to import text files into rsyslog, using the imfile file input module. However, rsyslog does not parse the content of the text files as I expected and I am struggling to find documentation on exactly how it is done. To test the setup, I am reading from a text file with imfile and then writing the logs to another text file with omfile.

The text file's contents are logs in the "standard" syslog format:

<PRI>TIMESTAMP HOSTNAME MESSAGE

The example text file (example_file.txt.) that I want to import into Rsyslog looks like this:

<34>Feb 15 12:12:12 hostname1 tag1: message1
<34>Feb 16 12:12:12 hostname2 tag2: message2
<34>Feb 17 12:12:12 hostname3 tag3: message3

My config file for rsyslog in rsyslog-d looks like follows:

module(load = "imfile")
input(type = "imfile" file = "/home/.../Desktop/example_file.txt" Tag = "example")
action(type = "omfile" file = "/home/.../Desktop/example_output.log")

The resulting output in example_output.log looks like this:

Feb 15 17:10:21 username example <34>Feb 15 12:12:12 hostname1 tag1: message1
Feb 15 17:10:21 username example <34>Feb 16 12:12:12 hostname1 tag2: message2
Feb 15 17:10:21 username example <34>Feb 17 12:12:12 hostname1 tag3: message3

As you can see, all of the content from example_file.txt was placed in the MSG field of the resulting log in example_output.log, instead of using the field information and placing them in the correct places, e.g. TIMESTAMP, HOSTNAME, TAG, MSG. I have played around with different formats in the .txt file, or even saving the .txt file as a .log file, but rsyslog places the whole content in the MSG field every time.

My question then:

How can I tell rsyslog and imfile that my .txt content is actually logs and to parse them correctly?

Take into account:

1. I am working on the Up-Board with Linux v4.4.0-ubi4-amd64 (UbiLinux)

2. I am using rsyslog8.24 (newest stable version)

3. I have already read through:

   -Rsyslog official documentation,

   -Imfile official documentation,

   -Rainer Gerhards's syslog parsing in rsyslog (http://www.rsyslog.com/doc/syslog_parsing.html),

   -and even the documentation for the BSD Syslog protocol RFC3164 (http://www.ietf.org/rfc/rfc3164.txt)

Decline

Answer 1

You can use templates to extracts fields out of messages. Here is an example template.

template(name="structured-format" type="list") {
   constant(value="{")
       property(outname="pri" name="msg" field.number="1" field.delimiter="32" format="jsonf") 
       constant(value=", ")
       property(outname="hostname" name="msg" field.number="4" field.delimiter="32" format="jsonf") 
       constant(value=", ")
       property(name="msg" format="jsonf")
   constant(value="} \n")
}

You can use this template in output like this.

action(type = "omfile" file = "/home/.../Desktop/example_output.log" template="structured-format") 

Output would look like this:

{"pri":"<34>", "hostname":"hostname1", "msg":"<34>Feb 15 12:12:12 hostname1 tag1: message1"}

That said, I have not yet figured out how to exclude parsed fields from msg and only add the remaining to msg field. Hope you find the pointers helpful.

Headmost

Answer 2

How can I tell rsyslog and imfile that my .txt content is actually logs and to parse them correctly?

This can be done using templates as below (update rsyslog.conf with below rules)

#Define a template of type string which just formats output to be send
#                  to remote as line read from file. Named "tpl1" here.

template(name="tpl1" type="string"
     string="%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"
    )


#Load imfile module to read file with log messages
#Specify the input (which file to read) and action (where to send log messages 
#along with template to be used)

module(load="imfile")
input(type="imfile" file="/var/log/FileWithLogMessages.log" Tag="GiveSomeTag")
action(type="omfwd" target="192.168.0.1" Port="514" Protocol="udp" 
template="tpl1")

Refer :

Templates : https://rsyslog.readthedocs.io/en/latest/configuration/templates.html

Sol