How to run fluentd in docker within the internal network

Question

I have the following configuration in my docker-compose file:

 fluentd:
    build: ./fluentd
    container_name: fluentd
    expose:
    - 24224
    - 24224/udp
    depends_on:
    - "elasticsearch"
    networks:
    -  internal

 public-site:
    build: ./public-site
    container_name: public-site
    depends_on:
    - fluentd
    logging:
      driver: fluentd
      options:
        tag: public-site
    networks:
    -  internal

networks:
  internal:

When I start the app using docker-compose up, then the webserver exists with the error message ERROR: for public-site Cannot start service public-site: failed to initialize logging driver: dial tcp 127.0.0.1:24224: connect: connection refused.

On the other hand, when I publish the ports from fluentd (ports: 24224:24224), it works. The problem is that I don't want to publish those ports on the host, since it bypasses the linux firewall (i.e. it exposes the fluentd port to everyone, see here).

This is confusing, since exposing a port should make it available for every container in the network. I am using an internal network betweem fluentd and the webserver, so I would expect that the exposed ports of fluentd are enough (which isn't the case).

When I connect to the webserver container, I can ping and resolve the fluentd container, so there is a connection. For some reasons however, at startup it won't accept a fluentd config with no published ports.

Vestigial

I have exactly the same problem. Have you found a way to solve it ? – Scan

Answer 1

The communication to 127.0.0.1 is always problematic if you're in a container. I found this explanation in the docs that performs way better than I would do:

To use the fluentd driver as the default logging driver, set the log-driver and log-opt keys to appropriate values in the daemon.json file, which is located in /etc/docker/ on Linux hosts or C:\ProgramData\docker\config\daemon.json on Windows Server. For more about +configuring Docker using daemon.json, see +daemon.json.

The following example sets the log driver to fluentd and sets the fluentd-address option.

 {
   "log-driver": "fluentd",
   "log-opts": {
     "fluentd-address": "fluentd:24224"
   }
 }

src: https://docs.docker.com/config/containers/logging/fluentd/

EDIT: this works until you want to have an application on the host communicating with the dockerized fluentd (then it's a pain)

Husbandry

When I use the fluentd address, I get instead: Cannot start service public-site: failed to initialize logging driver: dial tcp: lookup fluentd: Temporary failure in name resolution . What can I do to help docker resolve fluentd? I have tried with depends_on and links, but both won't work. – Vestigial

can you do another test? try to run a docker-compose up fluentd wait some time and then docker-compose up public-site. I think there might be a problem with the resources being ready to be accessed – Husbandry

Then I am getting this error: Cannot start service public-site: failed to initialize logging driver: dial tcp: lookup fluentd on 192.168.65.1:53: no such host – Vestigial

I found this issue (github.com/docker/compose/issues/6764) that would explain the behavior you're experiencing. – Husbandry

Answer 2

I am facing the same error with you. After check the example config in fluent official site, I was able to connect fluentd through links.

Below is my configuration that works:

version: "3.5"

networks:
  test:

services:
  flog:
    container_name: flog
    image: mingrammer/flog:0.4.3
    command: -t stdout -f apache_common -d 1s -l
    logging:
      driver: "fluentd"
      options:
        fluentd-address: localhost:24224
    links: 
      - fluentd
    networks:
      - test

  fluentd:
    container_name: fluentd
    image: moonape1226/fluentd-with-loki-plugin:v1.13-1
    ports:
      - "24224:24224"
      - "24224:24224/udp"
    volumes:
      - ./config/fluentd/fluent.conf:/fluentd/etc/fluent.conf
    networks:
      - test

Brenn

Answer 3

I have facing issue, I have solve using using static ip address.

logging:
  driver: fluentd
  options: 
    fluentd-address: 172.24.0.5:24224

Ceballos

Answer 4

I have fixed the failed to initialize logging driver: dial tcp 127.0.0.1:24224: connect: connection refused. error by specifying the IP and port while running my docker container and connecting to fluentd as in their specifying address example

docker run -p 127.0.0.2:8080:80/tcp --log-driver=fluentd --log-opt fluentd-address=192.168.0.11:24224 -i -t mycontainer:v2

Changing the IP from the default one to the current IP took from ipconfig (example in this case 192.168.0.11) fixed the error. In fluentd config I've had to also specify this IP in the bind section.

Metamerism