Performance testing - receiving high rate of HTTP 429 Too Many Requests

Question

I have a requirement to have a website hosted on Azure App Service, fronted by Azure Front Door Premium, to have <5% error rate and <15s average response time over a 1 hour test with 55,000 virtual users.

I'm using Tricentis NeoLoad and the results look as follows:

Sample log response:

HTTP/1.1 429 Too Many Requests
Cache-Control: no-store
Content-Length: 101
Content-Type: text/plain
X-Azure-Ref: 0uIzqZQAAAAB4EU9yZuEFRL55FVqEGkG6U1lEMDNFREdFMTIxMwA1MGYyYmQ2Ni1kM2QwLTRkY2EtODk2ZC0xNDE1NjMyN2U1ODc=
Date: Fri, 08 Mar 2024 03:57:43 GMT
Connection: close

Diagnostics are enabled, however there is no log of this in the log analytics workspace. I've checked the subscription limits and we look to be well under the 100,000 requests per second limit, the 75Gbps limit, and the various other limits.

I've opened an Azure Support ticket and they are investigating, but I thought I'd put this out there for anyone that has been down this road before, and also just to track this issue and provide an answer should I get one from Azure Support.

Update:

The WAF is enabled, in blocking mode, with the default rule set and default bot manager rule set. There are no custom rules.

Requests over the test period

This correlates correctly from the NeoLoad side which recorded 25,682,719 requests, 11,438,803 errors.

Diagram

Is there any custom rate limiting configured on the Front Door WAF policy? – Coccidioidomycosis

Also can you verify the number of requests Front Door is seeing during the load test using the log analytics? – Coccidioidomycosis

@Coccidioidomycosis Updated the question with answers to those questions. Thanks. – Diagram

Answer 1

Azure support have come back with the following information:

Azure Front Door is a global service and has a limit of 100K request per second per profile globally across 192 edge locations. If the traffic exceeds this limit, the customer traffic is rate limited.

When traffic is concentrated around one or more big geographical regions, AFD has a regional upper limits for the safety of platform and customers and to prevent outages. In such cases, AFD allows requesting to increase the regional upper limit appropriately based on the available capacity in that region.

In this case, the client’s traffic is concentrated with one/two POPs with 50K RPS. This is beyond the default limit in the capacity in PoPs and cannot be supported in short notice. However, we may be able to provide up to 15K RPS capacity per PoP after 1.5 months. Please check back after 1.5 months.

Diagram