How to gather logs to Elasticsearch

Question

I have logs of web apps in different servers (many machines). How can I gather these logs in a system where I have Elastic search and Kibana installed. When I searched I only found tutorials that show setup where logs, logstash, beats, elasticsearch and kibana are all together.

Denude

You need to read more about ELK stack, simply Logstash can read logs from your log files and filter them the format you want and then index them to Elasticsearch , then you can use Kibana to visualize this logs. – Dynah

Answer 1

Since you have many machines which produce logs, you need to setup ELK stack with Filebeat, Logstash, Elasticsearch and Kibana.

• You need to setup filebeat instance in each machine.

It will listen to your log files in each machine and forward them to the logstash instance you would mention in filebeat.yml configuration file like below:

#=========================== Filebeat inputs =============================

filebeat.inputs:

- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /path_to_your_log_1/ELK/your_log1.log
    - /path_to_your_log_2/ELK/your_log2.log

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["private_ip_of_logstash_server:5044"]

• Logstash server listens to port 5044 and stream all logs through logstash configuration files:

  input {
        beats { port => 5044 }      
  }
  filter {
      # your log filtering logic is here
  }
  output {
          elasticsearch {
              hosts => [ "elasticcsearch_server_private_ip:9200" ]
              index => "your_idex_name"
          }
  }
  

• In logstash you can filter and split your logs into fields and send them to elasticsearch.

• Elasticsearch saves all the data we send through logstash in indexes.
• All data in elasticsearch database can be readable through Kibana. We can create dashboards with many types of charts based on our data using kibana.

Below is the basic architecture for ELK with filebeat:

Atomicity

Answer 2

• You need to install Filebeat first which collects logs from all the web servers.
• After that need to pass logs from Filebeat -> Logstash.
• In Logstash you can format and drop unwanted logs based on Grok pattern.
• Forward logs from Logstash -> Elasticsearch for storing and indexing.
• Connect Kibana with Elasticsearch to add Index and view logs in Matrix based on selected Index.

Kempe

Answer 3

As mentioned in other answers you will need to install Filebeat on all of your instances to listen of your log file and ship the logs. Your Filebeat configuration will depend on your log format (for example log4j) and where you want to ship it (for example: Kafka, Logstash, Elasticsearch).

Config example:

filebeat.inputs:
- type: log
  paths:
    - /var/log/system.log
  multiline.pattern: '^\REGEX_TO_MATCH_YOUR_LOG_FORMAT'
  multiline.negate: true
  multiline.match: after

output.elasticsearch:
  hosts: ["https://localhost:9200"]
  username: "filebeat_internal"
  password: "YOUR_PASSWORD"

Also Logstash is not mandatory if you don't want to use it, logs can be sent directly to Elasticsearch, but you will need to setup ingest pipeline in your Elasticsearch cluster to process incoming logs, more on ingest piplelines here.

Also one more useful link: Working With Ingest Pipelines In ElasticSearch And Filebeat

Rosenda

Answer 4

In order to grab all your web application logs you need to setup ELK stack. Right now you have elastic search setup which is just a database where all logs data are saved. In order to view those logs, you need Kibana which is UI and then you need Logstash and Filebeat to read those logs of your application and transfer it to Logstash or directly to Elasticsearch.
If you want a proper centralized logs system then I recommend you to use Logstash with Filebeat also. As you have different servers than on each server you install Filebeat and on your main server where you have kibana and Elasticsearch install Logstash and point all Filebeats to that server.

FileBeats are lightweight data shippers that we install as agents on servers to send specific types of operational data to Logstash and then logstash do the filter and send that logs data to elasticsearch.

Check How To Setup ELK follow the instruction on this website. Also, look FileBeat + ELK Setup

Newish

Answer 5

You can use Splunk and Splunk forwarder to gather all the logs together. Use Splunk forwarder in your web servers to forward all the logs to your centralized server which has Splunk.

Intake

Answer 6

If you don't want to add another tool to Elasticsearch and Kibana stack you can directly send logs to Elasticsearch, but you should be careful while constructing your pipeline to have more stable system.

To gather logs you can use python or another language but for python I would use this library: https://elasticsearch-py.readthedocs.io/en/master/

There is also another medium tutorial for python:

https://medium.com/naukri-engineering/elasticsearch-tutorial-for-beginners-using-python-b9cb48edcedc

If you prefer other languages to push your logs to elasticsearch, for sure you can use them too. I just suggested python because I am more familiar with it and also you can use it to create a fast prototype before make it live product.

Assuan