EFK system is build on docker but fluentd can't start up

Question

I want to build the efk logger system by docker compose. Everything is setup, only fluentd has problem.

fluentd docker container logs

2022-02-15 02:06:11 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"

2022-02-15 02:06:11 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.0.3'

2022-02-15 02:06:11 +0000 [info]: gem 'fluentd' version '1.12.0'

/usr/local/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require': cannot load such file -- elasticsearch/transport/transport/connections/selector (LoadError)

my directory:

 my project/
 ├─ fluentd/
 │  ├─ conf/
 │  │  └── fluent.conf
 │  └── Dockerfile
 └── docker-compose.yml

docker-compose.yml:

version: "3"
services:
  web:
    image: httpd
    ports:
      - "8010:80"
    depends_on:
      - fluentd
    logging:
      driver: "fluentd"
      options:
        fluentd-address: 127.0.0.1:24224
        fluentd-async: 'true'
        tag: httpd.access

  fluentd:
    build: ./fluentd
    volumes:
      - ./fluentd/conf:/fluentd/etc
    links:
      - "elasticsearch"
    ports:
      - "24224:24224"
      - "24224:24224/udp"

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.13.1
    environment:
      - discovery.type=single-node
    expose:
      - 9200
    ports:
      - "9200:9200"

  kibana:
    image: docker.elastic.co/kibana/kibana:7.13.1
    links:
      - "elasticsearch"
    ports:
      - "5601:5601"

Dockerfile:

# fluentd/Dockerfile

FROM fluent/fluentd:v1.12.0-debian-1.0
USER root
RUN ["gem", "install", "fluent-plugin-elasticsearch", "--no-document", "--version", "5.0.3"]
USER fluent

fluent.conf:

<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>

<match *.**>
  @type copy

  <store>
    @type elasticsearch
    host elasticsearch
    port 9200
    logstash_format true
    logstash_prefix fluentd
    logstash_dateformat %Y%m%d
    include_tag_key true
    type_name access_log
    tag_key @log_name
    flush_interval 1s
  </store>

  <store>
    @type stdout
  </store>
</match>

Pantomime

Hi! Welcome to StackOverflow! Do you see in docker build logs that ElasticSearch plugin is installed or not? You can verify by running gem list via docker exec in a running container. – Antiperistalsis

@Antiperistalsis so weird. I really lack fluent-plugin-elasticsearch. It is not installed success by Dockerfile, but why? – Pantomime

Does this mean RUN ["gem", "install", "fluent-plugin-elasticsearch", "--no-document", "--version", "5.0.3"] invalid? – Pantomime

Try: RUN gem install fluent-plugin-elasticsearch -v 5.0.3 --no-document – Antiperistalsis

Answer 1

UPDATE: New version of the fluent-plugin-elasticsearch fixed that issue, so you could install fluent-plugin-elasticsearch >= 5.20

If you can't use a newer plugin version or have other concerns, you could still use the old solution:

Seems to be elasticsearch 8.0 broke this feature. You could wait for a fix, and while waiting just add something like that:

FROM fluent/fluentd:v1.12.0-debian-1.0
USER root
RUN gem uninstall -I elasticsearch && gem install elasticsearch -v 7.17.0
RUN ["gem", "install", "fluent-plugin-elasticsearch", "--no-document", "-- 
 version", "5.0.3"]
USER fluent

Up

Thank you so much. I will try this later :) – Pantomime

It work. but I cant understand if I build the elasticsearch service in docker-compose. Why build this again in Dockerfile? – Pantomime

Is it for updating elasticsearch version 7.13.1 to 7.17.0? – Pantomime

@HenryKao Its, not a standalone elasticsearch server, it's just a client for ruby – Up

I see. thanks for your answer λ – Pantomime

same here... created issue: github.com/fluent/fluentd/issues/3639 – Saliva

Reinstalling of stock elasticsearch with certain one was crucial for my case. Thanks a lot! – Ir

Answer 2

I had the same issue, I solved it by updating the fluent-plugin-elasticsearch version in Dockerfile to 5.2.0.

Subcritical

Answer 3

I faced the same problem, but I used to make exactly the same image where everything works to this day. I can't figure out what has changed.

But if you need to urgently solve the problem, use my in-person image:

docker pull kurraj/fluentd_castom:latest

my dockerfile:

FROM fluent/fluentd:v1.14-1

USER root

RUN gem update --system && \
gem install fluent-plugin-elasticsearch --source http://rubygems.org

Pert

Mee too. this problem is still puzzling me :( – Pantomime

Thank you for your solution. I will try this later :) – Pantomime