How can I debug logstash even when configuration is OK, on Windows?

Question

I have the following configuration for my logstash importing a few CSV files:

input {
  file {
    path => [
        "C:\Data\Archive_ATS_L1\2016-10-08-00-00_to_2016-10-09-00-00\S2KHistorian\Historian\S2KEventMsg_Table.csv",
        "C:\Data\Archive_ATS_L1\2016-10-09-00-00_to_2016-10-10-00-00\S2KHistorian\Historian\S2KEventMsg_Table.csv",
        "C:\Data\Archive_ATS_L1\2016-10-10-00-00_to_2016-10-11-00-00\S2KHistorian\Historian\S2KEventMsg_Table.csv",
        "C:\Data\Archive_ATS_L1\2016-10-11-00-00_to_2016-10-12-00-00\S2KHistorian\Historian\S2KEventMsg_Table.csv",
        "C:\Data\Archive_ATS_L1\2016-10-12-00-00_to_2016-10-13-00-00\S2KHistorian\Historian\S2KEventMsg_Table.csv",
        "C:\Data\Archive_ATS_L1\2016-10-13-00-00_to_2016-10-14-00-00\S2KHistorian\Historian\S2KEventMsg_Table.csv",
        "C:\Data\Archive_ATS_L1\2016-10-14-00-00_to_2016-10-15-00-00\S2KHistorian\Historian\S2KEventMsg_Table.csv"
    ]
    start_position => "beginning"    
  }
}
filter {
  csv {
      separator => ","
      columns => ["MessageCode","SourceGuid","DateTimeGenerated","Code1","Code2","Code3","Code4","LanguageCode", "AlarmSeverity", "Message", "Guid1", "Guid2", "Guid3", "Guid4", "MessageOrigin", "RequestId", "Bool1", "Bool2", "Bool3", "Bool4", "Bool5", "Bool6", "Bool7", "Bool8", "Code5", "Code6", "Bool9", "Bool10", "Bool11", "Code7"]
  }
}
output {  
    elasticsearch {
        action => "index"
        hosts => "localhost"
        index => "S2K"
        workers => 1
    }
    stdout {}
}

I launch logstash with this command line:

logstash.bat –f ..\conf\logstash.conf --verbose

Usually I see the data that's being imported into Elasticsearch in the console. But all I get this time is one line that says "Pipeline main started" and it stays like that.

How can I check from logstash if data was imported? I tried using Elasticsearch by running: curl http://localhost:9200/_aliases

This usually gives the list of indices. But the index I have in this config (called S2K) does not get listed.

I'm new to ELK so how can I check if logstash is doing it's job? Please note that I'm using Windows 7.

Weakkneed

you can take a look to this discuss.elastic.co/t/logstash-not-reading-file-in-windows/41723 – Anemograph

I think the problem might be that you have already read the files with logstash, then you'll have to change the path of the sincedb file, which saves where logstash has read files cf elastic.co/guide/en/logstash/current/… – Anemograph

Answer 1

To debug logstash you need to do two things: add stdout in config, and run logstash in a proper way.

1 step: Add this config in your logstash conf file (ex.: /etc/logstash/conf.d/config.conf)

output {
  stdout {
    codec => rubydebug {
      metadata => true # Here, we will print metadata in console
    }
  }
}

2 step: Run logstash to see output with command

sudo /usr/share/logstash/bin/logstash  -f /etc/logstash/conf.d/config.conf

And you will get something like this:

{
            "log" => {
        "file" => {
            "path" => "***\\spring.log"
        }
    },
        "appName" => "my-service",
      "@metadata" => {
        "ip_address" => "***",
              "type" => "_doc",
              "beat" => "filebeat",
           "version" => "7.12.0"
    },
      "log_level" => "INFO",
     "serverName" => "***",
            "pid" => "6236",
         "thread" => "main",
        "message" => "***",
    "serviceName" => "***",
           "tags" => [
        [0] "beats_input_codec_plain_applied"
    ],
          "input" => {
        "type" => "log"
    },
     "@timestamp" => 2021-01-03T10:22:07.644Z,
       "@version" => "1",
          "class" => "***"
}

Finally, after debug you can run it like sudo systemctl start logstash

Hope, it would help you, this approach helped me to save my time

Tunny

Answer 2

Stdout Ruby Debug is your friend here.

This will output everything to screen so you'll need to push the screen output to a file (example code at the bottom)

https://www.elastic.co/guide/en/logstash/current/plugins-outputs-stdout.html

What goes in the .conf file within the output section

output { stdout { codec => rubydebug } }

This is the example of how you would run the conf and push the screen output to another file for debugging.

logstash -r -f yourconfig.conf > debugfile.out

Just change yourconfig.conf and debugfile.out for whatever names you want and please remember to remove the rubydebug codec from your conf file when done debugging!

Villenage

The debugfile.out gives me this: {:timestamp=>"2016-10-17T09:54:33.091000-0400", :message=>"Pipeline main started"} but how do I know if the data was imported. Shouldn't I be seeing the data that's imported, in the log itself too? I put the rubydebug in the output – Weakkneed

Ray, as per the comments from baudsp on your question, what's occurred here is that the files have already been processed so nothing has come out to the screen. For debugging purposes and replaying of data you can define the sincedb location. Passing it out to /dev/null will mean that each time you start the process again all of the data will be imported. If at all possible please apply the answer credits to baudsp as he has seen the correct answer to your question – Villenage

If your happy for the data to be reloaded (this will create dupes unless you plan to drop your index) then the parameter sincedb_path => "/dev/null" within the input filter will assist. If you want to view if an item from each file is loaded within the index is an element called "path" that should contain the filename and a quick kibana report for path and count should give you a count of how many entries were pulled in (provided path isn't analysed!) – Villenage

None of these suggestions are working. My elasticsearch database does not have the index after running this. I even tried sincedb_path => "/dev/null" and all I get is the 'Pipeline main started'. I'm using Windows 7 and to check if the index was created, I run Powershell with the command : Invoke-WebRequest -Uri "localhost:9200/S2K/_search?pretty. The index name is S2K in my config. – Weakkneed

Would you be able to test with just 1 CSV file please? It sounds as though its not picking them up – Villenage

That's what I'm actually doing now, one file. Same results. 1) Can the encoding of the file play a role here? 2) The first line contains data, no headers, can this be the problem? My CSV is Unicode encoded. – Weakkneed

1) It certainly can! 2) This hasn't been a problem for my files. – Villenage

Let us continue this discussion in chat. – Weakkneed

Answer 3

You may be able to use the line codec to change the charset of how the line is read (default is UTF-8) instead of having to change the files themselves

Villenage

How do you do that? I'm not sure what you mean. – Weakkneed

Within the input filter you can put. codec => line { charset => "xxxxx" } apologies, all of this is off a mobile so I'm unable to format correctly. The web address for available codecs is elastic.co/guide/en/logstash/current/plugins-codecs-line.html – Villenage

Wow, there are a lot of possible values on that page. When I open my file in Notepad on my Windows machine, all it tells me is that it is a Unicode encoding. I'm not sure which one to pick from your link. Very confusing. Is there an easier way to figure this out? – Weakkneed