Athena can only see the first JSON record written to Firehose by Kinesis Analytics

Question

I am using Kinesis Analytics to read in JSON from Kinesis Firehose. I am successfully filtering out some of the records and writing a subset of the JSON properties to another Firehose.

I wanted to execute an Athena query on the data being written to S3 via the destination Firehose. However, the JSON records written to the files in S3 do not have any newlines. Consequently, when I query the data using Athena, it only returns the first record in each file.

When I write records to the source Firehose, I manually insert a newline between records, but Analytics doesn't seem to do this when writing to the destination.

Is there a way to get Analytics to write out a separator or newline between records, so Athena can see all of the records?

Perfectionism

Looks like for now the only way is to add a lambda function to the firehose :( – Selinaselinda

you need to check the answer here. #48226972 – Reflect

Answer 1

Enabling "New line delimiter" in the Destination setting of the firehose delivery stream resolves the issue for me.

Sokoto

"new line delimiter" is only available when dynamic partitioning is enabled. – Sherrillsherrington