How API Gateway talk to Firehose VPC endpoint

Question

Using Amazon Kinesis Data Firehose with AWS PrivateLink tells Firehose VPC endpoint keeps the traffic between VPC and Firehose within AWS.

You can use an interface VPC endpoint to keep traffic between your Amazon VPC and Kinesis Data Firehose from leaving the Amazon network.

When API Gateway invokes PutRecord API of Firehose via the AWS integration, does the traffic goes through the Firehose VPC endpoint or goes to the Internet?

Updates

The Introducing Amazon API Gateway Private Endpoints shows the diagram where EC2 and Lambda are in VPC. "All publicly available endpoints" go to the Internet and not sure of if API Gateway recognize if Firehose private endpoint exists or not and route the traffic there.

Quarterstaff

Its all internal traffic. Check this article. Everything between API gateway and AWS resources is contained within AWS bubble. – Kurman

@Marcin, I read the article a few times and still not sure of the case when AWS resources is in VPC. In the article, only EC2 and Lambda is in a VPC with a comment "At re:Invent 2017, we announced endpoint integrations inside a private VPC. With this capability, you can now have your backend running on EC2 be private inside your VPC". Firehose as in "any other AWS services" is not in the VPC in the article. – Quarterstaff

The private integration for APi gateway is to have the api gateway endpoint in side your vpc. This way you make your api gateway only accessible from withing VPC. Without this, api gateway endpoints are public by default. When you create firehose, it is not created in your vpc. You are not specifying any vpc subnets when you create firehose stream. – Kurman

Thanks @Marcin, the API gateway itself is Internet facing. To step away from APIGW and Firehose temporarily, AWS document says if we create a DynamoDB private endpoint, somehow magically the access to a DynamoDB will go to the endpoint (docs.aws.amazon.com/amazondynamodb/latest/developerguide/…). Will the same happen to APIGW to Firehose as happens with EC2 to DynamoDB in the AWS document? – Quarterstaff

Sorry for late reply. Hope the answer clarifies some questions you may have about API gateway and Kinesis. – Kurman

Answer 1

The example with DynamoDB and EC2 is about Gateway VPC Endpoints. For API Gateway, there is no Gateway VPC Endpoint. Instead there are Interface VPC Endpoints (AWS PrivateLink) and Gateway private integrations.

Before I begin, it needs to be mentioned that there are three API Endpoint types Choice of the API Gateway endpoint type has important consequences when working with VPC.

Interface VPC Endpoint for API Gateway

It allows e.g. EC2 instance in private and public subnets to access your API Gateway using AWS internal network, without going over Interent. For this to work the API Gateway endpoint must be configured as private.

In this case API Gateway works with kinesis as usual. No need to do anything except to setup AWS integration for it. For instance, a private EC2 instance (in private subnet) will be able to access the private API Gateway endpoint through the interface VPC endpoint, and subsequently access the Kinesis:

Private EC2 instance -> Interface VPC Endpoint for API Gateway -> API Gateway (private) -> Kinesis

Important thing to know here is that once you create Interface VPC Endpoint for API Gateway in your VPC, you will not be able to connect to a regional or edge-optimized API gateway, even in a public subnet. Only private API gateway will be accessible from inside VPC when the interface is present.

Gateway private integration

This allows your public (i.e. regional or edge-optimized) API Gateway to access a private EC2 instance in a private subnet. This is done by creating (e.g. internal) NLB in your VPC which you connect to a VPC Link which in turn you associate with an API method in the API Gateway.

VPC Link works at method level, thus your public API can have one method (e.g. /private) to access private EC2 instance through the VPCLink, and second method (e.g. /kinesis) to access kinesis as usual using AWS integration.

Accessing private EC2 instance looks as follows:

API Gateway (/private method) -> VPCLink -> NLB -> private EC2 instance.

Accessing Kinesis:

API Gateway (/kinesis) -> Kinesis (through AWS integration)

You can also have your private EC2 instance communicate with Kinesis. In this case you need VPC Interface Endpoint for Kinesis if you are not using NAT gateway:

API Gateway (/private) -> VPCLink -> NLB -> private EC2 instance -> Interface VPC Endpoint for Kinesis -> Kinesis (AWS integration)

Hope this clarifies how API Gateway and Kinesis can interact.

p.s. AWS conventions by naming different things in a similar way causes a lot of headache.

Kurman

Thanks Marcin. ...I am still not certain if API Gateway AWS Integration (integration_type =AWS, connection_type=INTERNET) invoking AWS API will go to the Kinesis VPC Endpoint as "Gateway private integration" is only available for NLB with integration type HTTP or HTTP_PROXY and connection type VPC_LINK. It is not AWS integration to invoke Kinesis. – Quarterstaff

Perhaps the only way to verify is by taking away IGW from VPC and see what happens... – Quarterstaff

@Quarterstaff Gateway private integration is only for accessing your private resource in VPC (e.g. your ec2 instance). Kinesis is not yours. Gateway private integration is not used for kinesis. Whether there is IGW or not is irrelevant, as kinesis is not in your VPC. API Gateway can always communicate with Kinesis. – Kurman

@Quarterstaff IGW is only relevant for your instance to access kinesis, not whether API gateway can access kinesis. – Kurman

I see. So Kinesis VPC endpoint has been irrelevant with API Gateway from the beginning. I think I have some misunderstanding about VPC endpoint. – Quarterstaff

Kinesis VPC endpoint is only important for your EC2 instance in private subnet to access Kinesis. API gateway using AWS integration works "outside" of VPC. – Kurman