Can Kinesis Firehose do filtering?

Question

So we have 100 different types of messages coming into our Kinesis stream. We only want to save 4 types. I know Kinesis can transform messages, but can it filter as well? How is this done?

Strut

A few clarification questions: 1. What's the source of the data? 2. is this a kinesis firehose or kinesis data stream? 3. Any reason you can't filter at the source? – Diaconate

1. The source is a Kinesis Data Stream. 2. This is a Firehose that I am using. 3. No, we want to pick off specific types of messages for customers. There might be a lot of these. – Strut

Answer 1

Filtering is just a transform in which you decide not to output anything. You indicate this by sending the result with a value "Dropped" as per the documentation.

You can find at this post an example of transform, and the logic includes several things: letting records just pass through without any transform (status "OK"), transforming and outputting a record (again, status "OK"), dropping -or filtering- a record (status "Dropped"), and communicating an error using the status "ProcessingFailed"

Ilyse