Getting IP address of Logstash-forwarder machine

Question

I've setup the Elasticsearch, Logstash, Kibana log viewing tools on my systems. There are 2 machines in my configuration now (Amazon EC2 instances):

• 54.251.120.171 - Logstash-server where ELK is installed
• 54.249.59.224 - Logstash-forwarder - sends "/var/logs/messages" log to Logstash-server

On logstash-server, this what my configs (in different files) look like :-

input {
  lumberjack {
    port => 5000
    type => "logs"
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}




filter {
 if [type] == "syslog" {

    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}


output {
  elasticsearch { host => localhost }
  stdout { codec => rubydebug }
}

On logstash-forwarder, this what my config file looks like, it forwards the /var/log/messages, /var/log/secure logs to logstash-server

{
    "network": {
        "servers": [ "54.251.120.171:5000" ],
        "timeout": 15,
        "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt"
    },

    "files": [
        {
        "paths": [
                "/var/log/messages",
                "/var/log/secure"
        ],
        "fields": { "type": "syslog" }
        }
    ]
}

This is what my Kibana interface looks like after it has fetched the indexed logs from Elasticsearch.

So my question is, I need a way to retrieve IP address of the logstash-forwarder i.e. 54.249.59.224 in case there's a log event.

The reason why I'm asking this is in a real scenario, we might have many logstash-forwarders (say 10), with all 10 of them sending logs to our logstash-server. So I need someway to tag all the log events, so that I can identify which logstash-server has sent which log event.

I'll need to use the IP address (maybe some other information as well) to search for log events in the Kibana interface.

Can someone please help me to do this? :)

Or incase someone has a better idea how to do this effectively in a different way, you're most welcome!

Lithia

Answer 1

You will need to modify the Lumberjack input filter and server to do this. See the following diffs:

https://github.com/stanhu/logstash-input-lumberjack/commit/0861c9d95caa46370f17a82353710bc78ea2c82e

https://github.com/stanhu/logstash-forwarder/commit/b711d273a24ab3fe1b4a7b6005d2f26983cac859

Until these changes get merged into logstash-forwarder and logstash-input-lumberjack, the easiest way to use this right now is to modify your installed files directly. If you're using the current Debian packages, they are here:

/opt/logstash/lib/logstash/inputs/lumberjack.rb
/opt/logstash/vendor/bundle/jruby/1.9/gems/jls-lumberjack-0.0.20/lib/lumberjack/server.rb

To the input filter configuration file (e.g. /etc/logstash/conf.d/01-lumberjack-input.conf), add:

client_address_field => "client_address"

Your logstash config should look something like this:

input {
  lumberjack {
    host => "localhost"
    port => 5000
    type => "logs"
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
    client_address_field => "client_address"
  }
}

Restart logstash on the machine, and check whether the client_address field is now being added to each message.

Pindaric

Answer 2

This is what the 'host' field (which you're also copying to 'received_from') is for.

Sahaptin

As you can see in the kibana snap I've attached, the received_from field is reporting the the host name to be 'eva-all', but I expect it to report the IP address of the host i.e. '54.249.59.224' Can you point me how to modify the code to do this. Sorry I'm still not able to understand fully the syntax or the capabilities of grok and filters. – Lithia

I added another machine whose IP is '46.137.246.88', so basically the Kibana is getting logs from 2 machines (both have host-names to be eva-all), so I'm not able to differentiate as to which log is from which machine. – Lithia

As you've shown in your logstash-forwarder config, you can send fields over from the shipper. Add another field and give it a value that will let you differentiate it from the other machine. You might also consider not giving your machines the same hostnames. – Sahaptin

I get it till this point, I can add add {"uniqueId" : "env1"} and similarly {"uniqueId" : "env2" } in the config of logstash shipper machines. So now I can differentiate based on "env1" or "env2". How do I dynamically get the IP address of that machine and add a field like {"ipaddr" : ip_addr_value}, so that I don't need to edit the logstash-forwarder file everytime I add a new machine to our system. – Lithia