Fluentd to Logstash output plugin

Question

I am trying to read from the scribe server using flunetd and output those logs to be stored in logstash for now. I know it's very stupid to log the scribe_central logs to another central logger, but we need this to be done in our current architecture.

Does anyone know if there is any plugin to do that? I searched Google but could not find any.

Abran

Answer 1

For Scribe <-> Fluentd, there is fluent-plugin-scribe:

For Fluentd <-> Logstash, a couple of options:

1. Use Redis in the middle, and use fluent-plugin-redis and input_redis on Logstash's side. So it would be Fluentd -> Redis -> Logstash. This is what Logstash recommends anyway with log shippers + Logstash.
2. Alternatively, you can use Fluentd's out_forward plugin with Logstash's TCP input. Logstash has Fluentd codec to handle the input coming from Fluentd.

Monetta

Since this post is almost a year old, do you know if there's a new options to do it ? – Match

What are you looking for specifically? The outlined solution still works. – Monetta

The solution Kiyoto Tamura mentioned, about using Fluentd forwarding to Logstash, is not working. I have tried without success. Fluentd keeps saying "no nodes are available". – Comeback

please notice that the out_forward is not secured (ssl), and the out_secure_forward is only between fluentd sender and fluentd receiver. – Smiga

Answer 2

You can forward it directly to your logstash tcp input :)

I wrote a flunetd output plugin for forwarding fluentd events to a generic receiver through a secured tcp connection (can be configured for non-secured as well).

To add the plugin to your fluentd agent, use the following command:

gem install fluent-plugin-loomsystems

Also see full original answer, good luck.

Smiga

Answer 3

I got it working with http output from fluentd and http input for logstash, below the configuration snippets:

fluentd:

<match **>
    @type http
    endpoint_url http://logstash-box:8080/
    http_method put
    serializer json
    raise_on_error false
</match>

Logstash:

input {
    http {
        port => 8080 
    }
}

Obs: You may have to install the plugin gem install fluent-plugin-out-http

Corduroy

504 Gateway Time-out when trying to push to logstash. – Nonperishable