Best approach for sending logs from ECS Fargate into Elasticsearch

Question

We have a setup with multiple containers running NodeJS services(node:11-alpine docker image) deployed in AWS ECS Fargate.

We already have a running ElasticSearch instance collecting logs from non-Fargate application. I would like to pass the logs from the Fargate containers into this ElasticSearch instance, but I have a hard time to figure out what is the best approach.

1) It seem one way is to stream the logs from Cloudwatch --> Lambda --> ElasticSearch. It seem a bit overkill - isn't there another way to do this?

2) I was hoping i could run a Logstash docker instance that could collect the logs from the containers but I am not sure if this is possible when running Fargate?

3) Should I install something like FileBeat on each container and let that send the logs?

Any help is appreciated.

Gerick

How about leveraging FunctionBeat which is fit for this very purpose? – Procreant

Answer 1

1. It seems one way is to stream the logs from Cloudwatch --> Lambda --> ElasticSearch. It seem a bit overkill - isn't there another way to do this?

If you're looking for an AWS-based managed solution, that is one of the ways. You don't really need to write a Lambda function, AWS does it for you. Although, you have to bear the cost of Lambda and CloudWatch.

There is another solution that is recommended by AWS and that is the use of fluent-bit as a sidecar container to export logs directly to Elasticsearch/OpenSearch from other containers running within a service. Using this solution, you save money by not using AWS CloudWatch. This solution also provides better results with regard to the loss of logs upon failure.

2. I was hoping I could run a Logstash docker instance that could collect the logs from the containers but I am not sure if this is possible when running Fargate?

Yes, that is possible if you run that container along with the other container.

3. Should I install something like FileBeat on each container and let that send the logs?

You can use Fluent Bit, Filebeat, Fluentd, Functionbeat, or Logstash as you like.

Note: If you're thinking of running your own logs exporter container like Logstash, Fluent Bit, etc, don't enable CloudWatch logging to save money as you're not going to use that.

Elidaelidad

Thanks so much for your help. Could you explain more in details about 2). Is it also possible for a Fargate setup (and not EC2)? – Gerick

@KasperF, yes, but you have to run your Logstash container in your AWS Fargate task. You cannot have a separate task for Logstash for all your Fargate containers. While defining the Fargate task, add your Logstash container with your desired configurations (AWS ECR Image might help you in that). – Elidaelidad

Please have a look at the accepted answer of #61159470. It's relatively very inexpensive solution. – Earleenearlene

@PythonEnthusiast Thanks. By the way, I have already mentioned that in the 3rd question's answer. – Elidaelidad