AWS Lambda TooManyRequestsException: Rate Exceeded

Question

When attempting to execute an Amazon Web Services (AWS) Lambda function (many times) we were seeing the error:

AWS Lambda TooManyRequestsException: Rate Exceeded

How do we resolve this?

Toul

can you clean your lambda, from CLI aws lambda delete-function --function-name ... – Settee

Are you hitting the documented default safety limit of 100 concurrent invocations? – Concretize

This safety limit has been replaced: docs.aws.amazon.com/lambda/latest/dg/… – Janayjanaya

Answer 1

As noted by Michael, this is the error message you will see when you reach the documented default "safety" limit of 100 concurrent invocations:

"AWS Lambda has a default safety throttle of 100 concurrent executions per account per region. If you wish to submit a request to increase the throttle of 100 concurrent executions you can visit our Support Center..."

The solution was to open a support ticket providing the following info:

Limit increase request 1 
Service: Lambda 
Region: EU (Ireland) 
Limit name: concurrent requests (average duration * average TPS) 
New limit value: 2000

And then in the body of the ticket/request try to estimate your usage pattern:

Expected average requests per second: 200 
Expected peak requests per second: 2000 
Expected function duration: 2 seconds 
Function memory size: 1000mb 
Invocation Type: Request-response 
Event Source: Api Gateway & Lambda<->Lambda

It can take a while to get a response from AWS Support unless you are paying for premium support, so its best to load test your app during development/staging and request to increase concurrent invocations before you launch your app!

In our case it took 45 hours from initial support request to get the invocation limit increase.

The AWS support people are lovely, it just took what felt like ages (almost two days!) to get the service level increase, which would have been fatal if our launch had been public!

Toul

Surely if you have a business-critical public platform you pay for corporate support contract and get immediate responses? – Dishwasher

@Dishwasher yes, if you have a Lambda that is already in production and "BAU" then execs will see the benefit of paying for a support contract, but while developing the app it was considered a "toy" in 2016. Obviously now (in 2019) that Lambda & Serverless has been proven it's easy to get approval for support. :-) – Toul

Ah. Yes that sounds a frustrating position. – Dishwasher

Looks like they have changed this now: docs.aws.amazon.com/lambda/latest/dg/… – Janayjanaya

Answer 2

I got this problem when I set Reserve concurrency below 100. And use aws-sdk to send over 100 requests at a time.

In fact, When reserved concurrency is relatively low, any concurrency request over your concurrency setting (e.g. send 21 request when concurrency is 20) will throw out TooManyRequestsException. I am not sure is my account problem or just as @nelsonic memtioned: default limit.

But when I increase my concurrency no less than 100, if not works, keep increasing, then problem solved. Hope this can help.

Sesquiplane

Answer 3

This error happens when your lambda concurrency value is low.

In your lambda configuration settings concurrency should be set to : Use unreserved account concurrency (Ex: 20,000 -> this will be the value that is given for your account)

Even your provisioned concurrency for a given lambda version will use the capacity from your concurrency value

Ref:

Reserved concurrency – guarantees the maximum number of concurrent instances for your function. Once the function reserves concurrency, other functions cannot use it. Lambda lets you configure reserved concurrency for your function free of charge.

Provisioned concurrency – lets you request a certain number of execution environments which get initialized and prepared to respond immediately to the invocations of the function. This option incurs additional charges.

3d