How to support SSE with API Gateway in AWS while maintaining an external authentication layer in FastAPI application?

Question

I have a FastAPI application that uses Server Sent Events (SSE) for streaming the response of a generative AI model, similar to the API of OpenAI. The application is deployed using the following architecture:

• FastAPI application hosted by Gunicorn with uvicorn worker
• EKS that runs the dockerized FastAPI application
• ALB controlled by the ingress controller installed on the EKS
• API Gateway that adds an authentication layer to all of the services hosted on the EKS

When I run the FastAPI application with the SSE endpoint locally, everything works perfectly. However, when deploying the application with the above-mentioned stack, the SSE response is not streamed back, but returned when the stream completes with all the chunks at once.

After investigating, I discovered that the issue occurs when I add the API Gateway layer, which I need for authentication. The response isn't streamed anymore, and the content-length header is added when passing through the API Gateway. This makes it look like the API Gateway is waiting for the response to fully complete before adding the header and sending it back to the client.

Another problem I encountered is that the request times out after 30 seconds due to the API Gateway, while the SSE response could take longer than that.

I am looking for a solution to support SSE while keeping the authentication layer outside of the application code. Any suggestions or guidance on how to achieve this would be greatly appreciated!

Coady

Answer 1

This is how API Gateway works, full stop, there is no streamed responses and the response timeout is at 30 seconds. You've already concluded this. The mistake comes in with your expectation:

which I need for authentication

You don't need APIGW for authentication, You can directly verify users authentication and authorize them in your application. Using APIGW for authentication, is really an abuse of what it can do as it is not meant for that purpose, which is why you are running into all sorts of issues. For the cost of APIGW, adding it in also does not make sense.

We should investigate why you believe you can't add authentication into your application. What's stopping you from doing that?

Lowdown

It's not that we cannot add authentication to our application, it just makes it very complicated for us because we have 20+ applications with different tech stacks (Go, .NET, python, and others) that can be accessed from the internet and needs authentication. Implementing an authentication layer for each application and maintaining it will be very costly. We can develop a slim application very quickly and deploy them on EKS with service and configuring ingress route, it also allows internal apps to access those applications without authentication. We would like to hear about other options. – Coady

Well then in that case your options are using the ALB OpenID authentication configuration or using a K8s ingress controller that includes authentication handling. – Lowdown

@WarrenParad where can I learn more about this idea of authentication with API Gateway as an "abuse"? We are currently exploring that option – Darell

It's just the extension of KISS, if you can do something in code, don't deploy another infrastructure resource to do that. It's just better to focus on what can't be done with the layer you are at. Also in all but the most simple of cases, the authorizer in APIGW needs to be customized which means you are just writing code anyway. So now you have application code in two places, the ALB (as required by OP) and a lambda function. APIGW is a proxy for services that don't have their own HTTP interface. If you already have one, you don't need AWS APIGW. – Lowdown

Answer 2

Sadly this is not possible. I've tried using a lambda linked to an API Gateway, but even sending the keep alive header did not kept the connection to my lambda

Needlecraft