Fluentd SSL/TLS secured TCP output plugin to generic receiver (Logstash)?

Question

I've been looking for a while for fluentd output plugin for tcp which is also ssl secured that doesn't force my receiver to be from a specific kind. In my case, my receiver is logstash.

Here are a few of the plugins which came close (close but no cigar):

Forward Output - not supporting ssl connection.

Secure Forward Output - sends data only to another fluentd receiver.

Some were https plugins and some were specific service plugins (which required a token/user/password of some kind).

Is there any other plugin i can use? maybe with some workaround?

Qadi

Answer 1

After spending days on searching for an existing plugin, I decided that there is none and I shall write it myself!

fluent-plugin-loomsystems A fluentd output plugin for secured TCP forwarding :)

To add the plugin to your fluentd agent, use the following command:

gem install fluent-plugin-loomsystems

To match events and send them anywhere you'd like, simply add the following code to your fluentd configuration file.

<match **>
  @type loomsystems
  host <your-beloved-host>
</match>

After a restart of Fluentd, all flunetd events will be sent to your host.

The plugin opens a new ssl connection by default but can be configured to send on a non secured tcp mode.

<match tag-life.**>
  @type loomsystems
  host <your-beloved-host>
  use_ssl false
</match>

I welcome you to star, suggest, and contribute the plugin, enjoy :)

Qadi

Answer 2

@dorony : I was trying to use it but unable to make it work. I am running openshift-3.6.0 locally on docker containers. I added below configuration in fluentd.conf:

<match **>
    @type loomsystems
    host 172.17.0.1
    port 4000
    use_ssl false
</match>

And below configuration in my logstash input.conf. However i am unable to receive any logs in logstash. Even there is no connection. I am not getting any error in fluentd logs.

input {
   tcp {
     codec => fluent
     port => 4000
   }
}

Musical

Thanks it worked :) . Added this match on top and it started working. – Musical

That's great!! Also I personally didn't use the fluent codec in Logstash, because it removed the metadata from my sent messages. – Qadi

Yes I also removed this codec as it doesn't work with the current fluentd message structure. – Musical