fluentd elasticsearch plugin - The client is unable to verify that the server is Elasticsearch

Question

I want to send some nginx logs from fluentd to elasticsearch , however, fluentd is unable to start due to following error message:

The client is unable to verify that the server is Elasticsearch. Some functionality may not be compatible if the server is running an unsupported product.

[error]: #0 unexpected error error_class=Elasticsearch::UnsupportedProductError error="The client noticed that the server is not Elasticsearch and we do not support this unknown product."

This is my fluentd config :

<source>
  @type tail  
    <parse>    
      @type nginx 
    </parse>  
  path /tmp/lab4/nginx/access.log  
  pos_file /tmp/lab4/nginx/access.po
  tag nginx.access
</source>

<match nginx.**>
 @type elasticsearch
 scheme http
 host 192.168.1.154 
 port 9200 
 with_transporter_log true
 @log_level debug
</match>

If I do a curl http://192.168.1.154:9200 , I can see a response from Elasticsearch with the system version and other info .

For reference I am using :

• fluentd version 1.14.5
• fluentd elastic-search-plugin 5.2.0
• elastic-search 7.12.0

Any idea on what I am doing wrong ?

Australia

In the above snapshot, the elasticsearch client gem version (used by fluent-plugin-elasticsearch) is 8.0.0. You are using ElasticSearch v7.12.0 which is evaluated as unsupported. See: github.com/elastic/elasticsearch-ruby/blob/…. So, it looks like you need to install an equivalent supported version. – Detritus

thanks for pointing out the validation, I tried upgrading my Elasticsearch to version 8 , but couldn't make it work ... In the end, what I did is upgrade Elasticsearch to version 7.17 ,and downgrade my fluentd Elasticsearch plugin to v 7.12 sudo fluent-gem install elasticsearch -v 7.12 – Australia

You're welcome! Glad you could make it work! You might want to post a detailed answer on how you made it work if someone else encounters this issue. – Detritus

Thanks @Detritus — your comment is very helpful, you should post an answer, you deserve the rep! – Loveridge

@DarraghEnright: Glad it was helpful! :) Answer posted. Thank you! – Detritus

Answer 1

for anyone who is facing the issue in docker, the below steps solved the issue for me:

• need to build the fleutd with the "elasticsearch gem" as per the version of the elasticsearch being used, like below: Dockerfile:

FROM fluent/fluentd
RUN gem install elasticsearch -v 7.6
RUN gem install fluent-plugin-elasticsearch
RUN gem install fluent-plugin-rewrite-tag-filter
RUN gem install fluent-plugin-multi-format-parser

• Mention the es version in the out plugin of es in fluent.conf:

@type elasticsearch
host 10.10.13.21
port 9200
verify_es_version_at_startup false
default_elasticsearch_version 7

Diploid

Answer 2

In that snapshot, the elasticsearch client gem version (used by fluent-plugin-elasticsearch) is 8.0.0. You are using ElasticSearch v7.12.0 which is evaluated as unsupported.

See https://github.com/elastic/elasticsearch-ruby/blob/ce84322759ff494764bbd096922faff998342197/elasticsearch/lib/elasticsearch.rb#L110-L119.

So, it looks like you need to install an equivalent supported version.

Detritus