Rate exceeded in throttling is thrown on DescribeJobFlows

Question

Several days ago all went well, but now, not just one, but three of our servers are throwing these exceptions:

{ "Error" => { "Message" => "Rate exceeded", "Code" => "Throttling", "Type" => "Sender" } }

This happens when retrieving data from Amazon, while calling DescribeJobFlows on Amazon::Coral::ElasticMapReduceClient, and shows as NFO Exception Retriable invalid response.

Plenish

You must be hitting the server too hard it looks like. Try these links with suggestions or other information and then please let us know what works for you. sesblog.amazon.com/post/TxKR75VKOYDS60/… forums.aws.amazon.com/thread.jspa?messageID=366822# – Hypoglossal

Thank you very much.Now ,I have know the reason without the links.My colleages told me .Thanks any way.The reason is that there are so many tasks.We have 5 servers and aws must receive our data during 5 hours.But it is not enough to upload one server task to AWS in an hour.So time is out. I will read the links and then tell u which one can solve my problem.Thanks. – Plenish

Answer 1

Every AWS API has dynamic throttling protection. It's kinda sad but best way it to retry/sleep between API calls :(

There is a configuration for the boto3 client to change max_retries before throwing the exception. There is also the general reference on how to use retries.

Good luck!

Postmillennialism

Answer 2

You should add some retry logic to your code. See this other Stackoverflow question for some ways of implementing that in Python:

Need advice to add exponenital back-off logic in AWS API in python

Tetrad