ElasticSearch 8 errors with Action/metadata line [1] contains an unknown parameter [_type] status:400

Question

I am trying to setup EFK (ElasticSearch 8, FluentD and Kibana) stack on K8S cluster (on-premises)

I followed this link to install elasticsearch and installed it using helm charts and followed this link to install fluentd

Output of fluentd and elasticsearch pods

[root@ctrl01 ~]#  kubectl get pods
NAME                                                     READY   STATUS    RESTARTS   AGE
elasticsearch-master-0                                   1/1     Running   0          136m

[root@ctrl01 ~]#  kubectl get pods -n kube-system
NAME                                                            READY   STATUS    RESTARTS   AGE
fluentd-cnb7p                                                   1/1     Running   0          107m
fluentd-dbxjk                                                   1/1     Running   0          107m

However, elasticsearch log was piled up with the following warning messages

2021-10-18 12:13:12 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2021-10-18 12:13:42 +0000 error_class="Elasticsearch::Transport::Transport::Errors::BadRequest" error="[400] {\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"Action/metadata line [1] contains an unknown parameter [_type]\"}],\"type\":\"illegal_argument_exception\",\"reason\":\"Action/metadata line [1] contains an unknown parameter [_type]\"},\"status\":400}" plugin_id="out_es"
2021-10-18 12:13:12 +0000 [warn]: suppressed same stacktrace

Conf file (tailored output)

2021-10-18 12:09:10 +0000 [info]: using configuration file: <ROOT>
  <match fluent.**>
    @type null
  </match>
  <source>
    @type tail
    @id in_tail_container_logs
    path /var/log/containers/*.log
    pos_file /var/log/fluentd-containers.log.pos
    tag kubernetes.*
    read_from_head true
    format json
    time_format %Y-%m-%dT%H:%M:%S.%NZ
  </source>
  <source>
    @type tail
    @id in_tail_minion
    path /var/log/salt/minion
    pos_file /var/log/fluentd-salt.pos
    tag salt
    format /^(?<time>[^ ]* [^ ,]*)[^\[]*\[[^\]]*\]\[(?<severity>[^ \]]*) *\] (?<message>.*)$/
    time_format %Y-%m-%d %H:%M:%S
  </source>

I am not sure which 'type' field it refers to. I am unable to find an example of ElasticSearch 8 for match and source directives to compare

It seems type field is not supported from ES 8 onwards but I am not sure on that. Kindly let me know the reason for the error

Langbehn

Using type in request was deprecated in version 7.X and removed in version 8.X, you can read more in this documentation link. Also, Elasticsearch 8 is still in Alpha, a lot of things may not work with it yet. – Prong

@Prong Can you suggest me how to change the conf file ? – Langbehn

Unfortunately no, I do not use fluentd, you should check their documentation. But again, Elasticsearch 8 is in alpha, you should stay with version 7 unless you need to test if your applications will work with version 8. – Prong

the anslike,update you fluent.conf set https://mcmap.net/q/1322449/-how-do-i-connect-aws-ecs-to-elasticsearch-with-fluentbit – Cardholder

Answer 1

I faced similar errors when I tried to use elasticsearch 8.2.3 with fluentBit 1.9.5. I could see elastic was sending logs but could not see any data in kibana webpage due to which could not create indices and saw the above error in fluent-bit pod logs. I followed this github issue and added Suppress_Type_Name On under outputs: section in my fluent-bit helm chart values.yaml file and it worked fine after that.

      [OUTPUT]
          Name  es
          Match *
          Host  {{ .Values.global.backend.es.host }}
          Port  {{ .Values.global.backend.es.port }}
          Logstash_Format Off
          Retry_Limit False
          Type  _doc
          Time_Key @timestamp
          Replace_Dots On
          Suppress_Type_Name On
          Index {{ .Values.global.backend.es.index }}
      {{ .Values.extraEntries.output }}

Disintegration

what is the consequence of doing this? – Celt

Answer 2

I was working on the same issue for a few days and I found a solution but just a workaround, not the optimal solution.

If you set TypeName as null for ElasticsearchSinkOptions, you don't face this issue.

Unfortunately, you can't set it from appsettings.json. At least I couldn't find a way.

In background, Serilog.Sinks.ElasticSearch library use this property as _type in HTTP header. But the '_type' header, as leandrojmp pointed out in the comment, it is no longer available in version 8.2 of ElasticSearch.

Uriah

worked for me thx. Maybe write in your answer that you need to set documentType="" in the elastic search target <target name="elastic" xsi:type="ElasticSearch" index="TestService-${date:format=yyyy.MM.dd}" uri="localhost:9200" layout ="${MicrosoftConsoleLayout}" documentType=""> </target> – Pollinosis

Answer 3

FluentD (and Fluent Bit) Elasticsearch output plugin puts a _type field in logs when transferring them to Elasticsearch. However, this field was a special field of Elasticsearch and it was removed from Elasticsearch in version 8.

Therefore, if your Elasticsearch installation has a version of 8 or later, you need to suppress FluentD (or Fluent Bit) to not put the _type field in logs by setting Suppress_Type_Name configuration parameter as On (default is Off).

[OUTPUT]
    Name  es
    Host  192.168.2.3
    Port  9200
    ...
    ... (other es output plugin parameters)
    ...
    Suppress_Type_Name On

By using this configuration parameter, FluentD (or Fluent Bit) will not include a _type field in the final log and Elasticsearch will not complain.

Reference:

https://docs.fluentbit.io/manual/pipeline/outputs/elasticsearch#action-metadata-contains-an-unknown-parameter-type

Grenada

thank you very much, it was applied with the latest version of elasticsearch 8.14.3. – Kandace