I am trying to configure a Kinesis Analytics application with the following settings: Input stream is a Kinesis Firehose which is taking stringified JSON values The SQL is a simple passthrough (i...

I have bucket which is used as destination for a Kinesis Firehose stream. Firehose automatically creates date-based prefixes on that bucket using the yyyy/mm/dd/HH format. Then I created a crawle...

I am trying to set up firehose to send data from a kinesis stream to a redshift cluster. Firehose successfully inserts the data to my s3 bucket, but I am receiving the following error when firehose...

Reading over the documentation of both offerings (Firehose and Streams), it sounds like Firehose is "near" real-time with a potential of 60 seconds delay between producing a message to emitting it,...

Many AWS reference architectures for serverless real-time analytics, suggest pushing processed data from Lambda to S3 through Kinesis Firehose. e.g. https://aws.amazon.com/blogs/big-data/create-rea...

I'm attempting to price out a streaming data / analytic application deployed to AWS and looking at using Kinesis Firehose to dump the data into S3. My question is, when pricing out the S3 costs fo...

While doing a proof of concept for our new ETL pipeline, I figured out some problems using partition projection in AWS Athena. Created the following table in glue: CREATE EXTERNAL TABLE `test_inter...

Objective Solution or workaround for the problem. Problem The Terraform API Gateway integration with Firehose below works if Firehose is created separately in advance. resource "aws_api_gateway...

The AWS docs makes it look as if you can specify DataFormatConversionConfiguration for a AWS::KinesisFirehose::DeliveryStream in cloudformation, but is not documentation on where the property is su...

We have a firehose that sends records to an Elasticsearch Service cluster. Our cluster filled up and some records failed over to S3. The documentation at https://docs.aws.amazon.com/firehose/latest...

Account A is the application account where I created Kinesis stream and I want to create Firehose in Account B to read from Account A Kinesis stream. Is this possible? I tried to follow the steps f...

I'm researching AWS services related to mobile analytics as we are using AWS as our backend infra and now would like to understand how to best utilize Pinpoint and/or Kinesis to end up with events ...

I am trying to catch Cloudwatch logs for my firehose to find any errors when sending data to S3 destination. I created a cloudformation template with logging details "CloudWatchLoggingOptions&...

Using Amazon Kinesis Data Firehose with AWS PrivateLink tells Firehose VPC endpoint keeps the traffic between VPC and Firehose within AWS. You can use an interface VPC endpoint to keep traffic b...

I read the document of official AWS Kinesis Firehose but it doesn't mention how to handle duplicated events. Does anyone have experience on it? I googled someone use ElasticCache to do filtering, d...

I am using Firehose and Glue to ingest data and convert JSON to the parquet file in S3. I was successful to achieve it with normal JSON (not nested or array). But I am failed for a nested JSON arr...

I'm currently sending a series of xml messages to aws kinesis stream, I've been using this on different projects, so I'm pretty confident that this bit works. Then I've written a lambda to process ...

I'm using Kinesis Firehose to copy application logs from CloudWatch Logs into S3 buckets. Application logs are written to CloudWatch A Kinesis subscription on the log group pulls the log events i...

I would like to be able to send data sent to kinesis firehose based on the content inside the data. For example if I sent this JSON data: { "name": "John", "id": 345 } I would like to filter t...

I'm trying to use Firehose API (JS) and I keep getting the following error: "InvalidArgumentException: Firehose is unable to assume role arn:aws:iam::XXXXXXXXXX:role/NAME. Please check the rol...

So we have 100 different types of messages coming into our Kinesis stream. We only want to save 4 types. I know Kinesis can transform messages, but can it filter as well? How is this done?

I'm setting up a Kinesis Firehose delivery stream to S3, and I noticed you can set a custom KMS key to be used for encrypting the files on S3. However, if the S3 bucket already has KMS encryption ...

The story I have ECS tasks that run docker containers that produce stdout/stderr output. The tasks are configured to use the awslogs driver to send the output to CloudWatch. There is a subscriptio...

Using AWS Firehose I am converting incoming records to parquet. In one example, I have 150k identical records enter firehose, and a single 30kb parquet gets written to s3. Because of how firehose p...

I'm trying to figure out if there is a service on GCP which would allow consuming a stream from Pub/Sub and dump/batch accumulated data to files in Cloud Storage (e.g. every X minutes). I know that...