AWS Kinesis Firehose - dynamic partitioning by timestamp other than epoch

Question

My Firehose reads from Eventbridge events that look something like:

{
  "detail": {
    "key1": "some value",
    "key2": "some value",
    "Timestamp": "2022-01-21T19:01:05Z"
  }
}

I'd like to perform dynamic partition when saving the events to files in s3. I tried to extract the year like so:

.detail.Timestamp| strftime("%Y")

I know this would have worked if Timestamp field was given in the epoch format, but now I get the error (saved to a file by firehose):

{
  . . .
  "errorCode": "DynamicPartitioning.MetadataExtractionFailed",
  "errorMessage": "strftime/1 requires parsed datetime inputs",
  . . .
}

What's the right syntax to extract year (and other time parts) from a timestamp like my events contain?

Presto

Answer 1

Answering my own question:

I realized that the way to select keys to be used for partitioning is to use jq syntax. So using | strftime("%Y") is the way to extract year from epoch time. To extract it from my time format, I needed to add another step of | strptime("%Y-%m-%dT%H:%M:%SZ").

The full solution would be: .detail.Timestamp| strptime("%Y-%m-%dT%H:%M:%SZ")| strftime("%Y").

After having understood that this is actually jq, the answer from here How to to extract the year from JSON date using jq is what completed the puzzle for me.

Presto