How to serve binary data from AWS API Gateway with proxy integration?

Question

I am building a serverless web site with AWS API Gateway (APIG) and Lambda functions. I must use proxy integration because the Lambda function behind has to receive http request headers. At the same time, some binary data need to be served. In my case, the favicon.ico file. Other people might want to serve dynamically created PDF or Excel files. APIG has binary support for this purpose. Encode data with base64 and configure that MIME content type so it can get decoded before it gets served to client. However, this doesn't work with proxy integration. Proxy integration just skips the integration response part.

I tried to redirect the request of favicon.ico to S3 endpoint, but browsers show strange behaviors. Because the ico file is from a different and redirected domain, not the same domain.

Encoding it with base64 and letting the client browser decode it is not an option since this isn't standard and might not work in all browsers.

I assume I can't do anything else until AWS adds a new feature for this. Has anyone looked into this problem? Any idea or suggestion?

Fortification

Answer 1

Answering my own question here. You should go to AWS forums for AWS questions. Not mamy AWS users here.

The answer: Binary support with proxy integration DOES WORK and it works for both incoming and outgoing response.

There are 3 factors related to this:

1. the MIME types in binary support settings for APIG (I set this on console)
2. the "isBase64Encoded" value in both incoming and outgoing JSON
3. the "Content-type" value in outgoing JSON

The answer above means that yes, you can accept files and yes again, you can spit out files with proxy integration.

When user 'posts' with body and if the MIME type you set matches it, APIG encodes the whole body part in base64 and indicates this with "isBase64Encoded" value. The same thing happens with outgoing responses too. If you want to response with binary data, you encode it in base64 and set that value true in outgoing JSON.

For incoming requests, it depends on only one factor, the MIME type you set. For outgoing, the both conditions, indicator in JSON and MIME type must match.

For simplicity, I just set the MIME type to */*. Whenever user submits anything in the body, let APIG encode then I decode. Whenever I respond with binary, I just set the indicator and encode in base64. I do not do this with other types such as test/html (not compressed).

Fortification

I'm trying to return a PDF in one of my endpoints, and I encoded it as base64 and set "isBase64Encoded" to true. However, the PDF I get back seems like it doesn't get decoded properly. If I open up the file in a text editor, decode it manually, and save it, then I can view the PDF. Any ideas why that's happening? – Phonetic

JPL, check your MIME settings for binary support. – Fortification

Since my last comment I have added binary support for all types, /, which works, but it messes up non-binary types such as JSON. If I type in application/pdf or even application/*, API Gateway doesn't recognize those and will not treat the type properly. I'm not sure if the format is wrong or if there are other settings. – Phonetic

Then, I guess you are not setting "Content-type" in your response correctly. The MIME setting in API Gateway must match the content type in your response. You are responsible to set that content type in the response. API gateway has no way to figure out what type your response is unless you explicitly set it. – Fortification

I'm also having an issue with API Gateway simply returning the base64 content without decoding it first. I've set Content-Type header in my response and the MIME setting in the Binary Support console--unfortunately, it's still not working. – Chine

Apparently, passing the Accept header in the request is also necessary for Lambda Proxy Integration. – Chine

@Chine That's where I've ended up, too. It's fine for curl but not helpful as a link for a browser. :( Did you ever figure out how to make it work without the Accept header? – Pictish

Ah, finally... I had set application/pdf as the "Binary Media Types" in the API's "Settings" section. That required the Accept header. But changing the media type to */* means the Accept header is not needed (by Chrome at least). Nice. Though this may interfere with returning JSON. One thing at a time. – Pictish

@RodneyGitzel I am now working with AppEngine. Much less of a headache, more control over the inputs/outputs. I hope you figure things out with Lambda. – Chine

Answer 2

For how to resolve "application/pdf" header not work and avoid using */*: the "Accept" header in the request must be exactly match with the Binary Media Type on gateway.

• if Accept is "*/*", the BinearyMediaType on gateway must be "*/*" (application/pdf will not work).
• if Accept is "application/pdf" (or image/png...), the BinearyMediaType on gateway must be "application/pdf" (*/* will not work.).

Nahtanha

Answer 3

I see the correct answer being spread throughout several answers and comments.

1. You need to make sure that not only you follow documentation on how to return binary from Lambda Proxy https://docs.aws.amazon.com/apigateway/latest/developerguide/lambda-proxy-binary-media.html
2. You also have to configure Binary Media Type at API Gateway (please, don't use */* - this will break your non-binary responses) https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-payload-encodings-configure-with-console.html
3. And the most overlooked part is that your clients, have to provide an Accept header as part of their request. Value for Accept header should match one you are using as a return Content-Type from your Lambda.

Albertalberta

Not only must the Accept-Header include the media type, but it must be at the first position. But I do not have control over the headers the clients send. The docs suggest using a wildcard for the binary types, but as you already pointed out that breaks all non-binary responses. – Ciel

Answer 4

In case of Lambda proxy integration, there are two requirements:

• The binary should be base64-encoded on the function's response, include the 'isBase64Encoded' attribute set to 'true' and a 'Content-Type' header to indicate to the client the type of data that it returns. See example in [1].

• The 'binaryMediaTypes' on the API settings must support such 'Content-Type' so the payload could be handle as binary [2].

Note: For proxy integrations, API Gateway passes the entire request through to your backend, and you do not have the option to modify the passthrough behaviors [3].

If you use '/' wildcard as the binary media type and the response from the Lambda is flagged as not base64 encoded, then the Lambda response will not be treated as binary, and the response will be returned in text format ( e.g. base64 encoded in case you encoded the file). Please note if your client is not prepared to handle the text response, it may see your binary as corrupted..

API Gateway applies a base64 decode for mime-type registered for binary support, if the payload received by the backend integration is base64 encoded.

Returning binary with a mapping template(in general):

The issue with returning binary with a mapping template if that if you need to select "CONVERT_TO_BINARY" in the Integration response, this will be applied before the mapping template is applied and this will result in the failure. Mapping template are meant to be used only and exclusively for application/json payloads. Using it with other content types and binary does not work well as mapping templates were not meant to work with those.

The mapping template you mentioned will not work. If you want to have a binary file as response, the CONVERT_TO_BINARY will be applied before the mapping template, hence this will not work. This is a limitation of mapping templates and hence it would be recommend to use Lambda Proxy Integration, or retuning from your Lambda template only the base64 encoded payload, no JSON or other data in it. Using $util.base64Decode will also not result in a valid binary file as API Gateway would still try to interpret it as application/json and hence the payload will be corrupted.

Following are the options you can consider:

1- Return the application/pdf as base64 encoded string from your lambda and then apply CONVERT_TO_BINARY in the Integration Response Passthrough. You will also need to statically specify the Content-Type response header in the integration response and method responses. Leave the response mapping template empty so the body is passed through and converted to a proper binary file.

2- Use Lambda proxy integration and return the payload as you were doing, with the headers, body as base64 encoded payload and 'isBase64Encoded': true. API Gateway will take care of setting the proper headers and return the file as binary.

Rydder

Answer 5

I ran into a similar situation with Node lambda function returning binary blob(PDF) from the service layer.

1. Configured / as a Binary Media Type on gateway. (I tried to use application/pdf, but did not work?)
2. Make sure the response body from the service layer not transformed into string (I am using request, and by default it gives me string). I send encoding: null along with the request
3. When i get the Buffer data from the service layer, i use Buffer to convert response body into base64 encoding.
4. In the lambda output, I set isBase64Encoded to true

Reference: here

Margitmargo