ALB is not propagating response headers correctly

Question

I have a lambda target behind an ALB. My lambda is a python lambda.

def handler(event, context):
response = {
    "statusCode": 200,
    "statusDescription": "200 OK",
    "isBase64Encoded": False,
    "headers": {
        "Content-Type": "text/html; charset=utf-8"
    }
}

On hitting my url using curl though, I receive a

< HTTP/1.1 200 OK
< Server: awselb/2.0
< Date: Sat, 06 Apr 2019 04:46:50 GMT
< Content-Type: application/octet-stream
< Content-Length: 0
< Connection: keep-alive

Note Content-Type is an octet-stream, which causes browsers to download the response as a file instead of displaying it. I tried adding additional headers "Foo":"Bar" to the response and they don't show up in the curl response. ALB seems to be eating my lambda supplied headers. How can I fix this?

Prefrontal

Add a response body to your JSON as body. See if it gets returned. – Paulettepauley

@sqlbot Yes, body gets returned fine, and any changes I make body content reflect immediately in the response. With an octet-stream response though, the body text does not get displayed in a browser, instead gets downloaded as a file. – Prefrontal

Understood -- I just wanted to verify that the response in general was indeed coming from the expected place. – Paulettepauley

Answer 1

Turns out I had multivalue headers turned on for my target group. With that setting turned on, my lambdas needs to return a response with field multiValueHeaders set instead of headers. So my lambda code needed to be:

def handler(event, context):
response = {
    "statusCode": 200,
    "statusDescription": "200 OK",
    "isBase64Encoded": False,
    "multiValueHeaders": {
        "Content-Type": ["text/html; charset=utf-8"]
    }
}

More information on AWS' release blog post.

Prefrontal

Thank you so much! This was exactly my problem! – Bud

Interesting, in my case it was turned off and I had to turn it On so it would work! – Notability

Answer 2

I had similar issue, even I was setting content-type correct in my Lambda but it was always returned as octet-stream. The reason was I had multiValueHeader enabled in my ALB target group whereas from Lambda I was just returning headers as Map.

After changing target group from multi valued header to false it works like a charm.

Primordial

Answer 3

AWS Application Load Balancer transforms all response headers to lowercase, you need to check your headers carefully. Unfortunately, you can not change or modify the headers are manipulated by the ALB. You can refer below link for HTTP Headers:

https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/x-forwarded-headers.html

For Request Tracing for Your Application Load Balancer :

https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-request-tracing.html

Also, you need to check the limitations for the Load Balancer:

• The load balancer updates the header when it receives an incoming request, not when it receives a response.

• If the HTTP headers are greater than 7 KB, the load balancer rewrites the X-Amzn-Trace-Id header with a Root field.

• With WebSockets, you can trace only until the upgrade request is successful.

Mortgage

Can you point me to documentation about lowercasing of headers? I do not have http/2 enabled on my alb. my question also is not about x-forwarded headers. – Prefrontal

Answer 4

In my case, I had two registered targets in my Target Group (HTTP 80). Both to the same EC2 instance, but one on port 5000 (for running outside of Docker), and the other on port 5001 (for running inside Docker).

I removed the port 5000 one and problem solved. Despite not having anything listening on port 5000.

The problem manifested as some files (a stylesheets.css and a js recource file) incorrectly having ContentType text/html set in the response headers.

The 'enable multi value headers' discussions around Target Groups was a red herring in my case because that option is only available for a Lambda target type.

Martellato