AWS load balancer returns a 403 response?

Question

When a call comes to a particular URL, AWS load balancer returns a 403 response. Once the browser cache is cleared, it will start working. it happens occasionally. What is the reason? No such response was noted in the WAF log. Is this because of the Application Load Balancer?. Has anyone faced such issues recently from AWS?.

Condottiere

Do you use the URL of ALB or anything else in front of your ALB? – Monophthong

@AshBlake Yes, the required URLs are CNAMED to the alb – Condottiere

You can change to use A record. And do you attach the WAF to the ALB? I think the reason is from the browser itself – Monophthong

Yes @AshBlake I attached the WAF to ALB – Condottiere

@AshBlake Any solution on this?? – Condottiere

There is 3 reasons that I am thinking you should check: 1. The browser (10%), 2. Your App (10%) 3. WAF (80%) You can read the log if you see the headers are different between the cache and without cache. And the WAF is denied of not, you should check it carefully – Monophthong

Answer 1

In case you are using AWS WAF along with ALB, it is likely that the WAF is blocking the request. Below documentation can be helpful.

[1] Troubleshoot your Application Load Balancers - HTTP 403: Forbidden - https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-troubleshooting.html#http-403-issues

In case you have access logs enabled, check the "Actions taken" field in the access logs. If it has the value "waf", it means The load balancer forwarded the request to AWS WAF to determine whether the request should be forwarded to the target. If this is the final action, AWS WAF determined that the request should be rejected. Check below documentation for reference.

[2] Access logs for your Application Load Balancer - Actions taken - https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html#actions-taken

On the other hand, if is also possible that the target is sending HTTP 403 and ALB is forwarding it back to the client. Check if the "target_status_code" field is 403 in the access logs. You will also see "elb_status_code" to be same because ALB will send the same code to the client.

In such scenario, the value of "action" field will be "forward".

PFB documentation for ALB access logs format/syntax.

[3] Access logs for your Application Load Balancer - Syntax - https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html#access-log-entry-syntax

I hope this helps!

Soiree

Thank you Aditya!. Helped me pinpoint a critical issue with the WAF configuration. – Ravelment

You are welcome Siraj! – Soiree

In my case, whitelisting the particular IP addresses (via custom WAF rule) solved the issue. – Blackball