Logstash: How to use date/time in a filename as an imported field

Question

I have a bunch of log files that are named as 'XXXXXX_XX_yymmdd_hh:mm:ss.txt' - I need to include the date and time (separate fields) from the filename in fields that are added to Logstash.

Can anyone help?

Thanks

Nicely

Answer 1

Use a grok filter to extract the date and time:

filter {
  grok {
    match => [
      "path",
      "^%{GREEDYDATA}/[^/]+_%{INT:date}_%{TIME:time}\.txt$"
    ]
  }
}

Depending on what goes instead of XXXXXX_XX you might prefer a stricter expression. Also, GREEDYDATA isn't very efficient. This might yield better performance:

filter {
  grok {
    match => [
      "path", "^(?:/[^/]+)+/[^/]+_%{INT:date}_%{TIME:time}\.txt$"
    ]
  }
}

Bibeau

Thanks. Is there any way I can replace the @timestamp value with the time contained within each line of the log file? Each line starts "08:55:43.23" (no quotes). I'd like to take the date from the file name + time in the log file – Nicely

Create a field (possibly with the mutate filter) that concatenates the date field picked up from the filename with the time from the log message and use the date filter to populate the @timestamp field. – Embattle

Can what you posted above be used in conjunction with what I currently have? i.e. can I put the match statement below this? grok { match => [ "message", "%{TIME:timestamp},%{WORD:agent},%{NUMBER:agentid},%{NUMBER:campaignid},%{CISCO_REASON:campaign_name},%{NUMBER:unknown1},%{NUMBER:unknown2},%{NUMBER:unknown3},%{NUMBER:unknown4},%{NUMBER:unknown5},%{NUMBER:unknown6},%{NUMBER:unknown7},%{NUMBER:unknown8},%{CISCO_REASON:Status}" ] } – Nicely

Sure. Your two grok filters parse different fields so they're not in conflict and they don't have an order dependency. – Embattle

Should your suggestion work in Grok Debugger? I'm seeing 'No Matches' when using the following input "C:/inetpub/tslogs/ACTrace_ACTAT_150202_100034.txt" (no quotes) – Nicely

The first example works with Windows paths. The second one doesn't (but could be made to work with a minor change). – Embattle

Sorry to sound like I need to be spoon fed, but are you able to help me with the syntax? I've tried #logstash to no avail. I need the date from the filename and time from each event in the log... – Nicely

Just prepend the second expression with "C:" and it'll match Windows paths too. – Embattle