Fluentd: Event forwarding to Elasticsearch taking long when using copy plugin

Question

I am setting up my fluentD configuration and for certain events, I need to push them to both loggly and elasticsearch. I am using copy plugin for that but see a considerable difference in the time taken by the fluentD call to return - time taken by a test POST call made through postman. It takes around 5sec if I use elasticsearch match inside copy vs <1s for direct elasticsearch match. I am using the following configuration.

<match testapp.elastic>
  @type elasticsearch
  host localhost
  port 9200
  index_name fluentdtest
  type_name fluentdtest
 </match>

 #Common match for loggly and mongo
 <match logapp.**>
    type copy
    <store>
            @type elasticsearch
            host localhost
            port 9200
            index_name fluentdtest
            type_name fluentdtest
    </store>


    <store>
      type forest
      subtype loggly
    <template>
        loggly_url ##myurl##/tag/${tag}
      </template>
    </store>
</match>

Radcliff

I see that you are using "host local" in the copy plugin definition - hope that's either a typo or you have "local" defined somewhere. – Vedi

Oh yes thats a typo in the post. Thanks for pointing out - will correct it – Radcliff