AWS elastic beanstalk 100.0 % of the requests are erroring with HTTP 4xx

Question

My AWS Elastic Beanstalk account keeps getting down with the error: "Environment health has transitioned from Ok to Severe. 100.0 % of the requests are erroring with HTTP 4xx" On a daily basis.

When looking at the server logs, it keeps getting down after access to several odd web pages (which do not exist). Part of the log:

/var/log/httpd/error_log-XXX
[XXX] [:error] [pid XXX] [client XXXX] script '/var/www/html/w.php' not found or unable to stat
[XXX] [:error] [pid XXX] [client XXX] script '/var/www/html/sheep.php' not found or unable to stat
[XXX] [:error] [pid XXX] [client XXX] script '/var/www/html/qaq.php' not found or unable to stat
[XXX] [:error] [pid XXX] [client XXX] script '/var/www/html/db.init.php' not found or unable to stat
[XXX] [:error] [pid XXX] [client XXX] script '/var/www/html/db_session.init.php' not found or unable to stat
[XXX] [:error] [pid XXX] [client XXX] script '/var/www/html/db__.init.php' not found or unable to stat
[XXX] [:error] [pid XXX] [client XXX] script '/var/www/html/wp-admins.php' not found or unable to stat
[XXX] [:error] [pid XXX] [client XXX] script '/var/www/html/m.php' not found or unable to stat
[XXX] [:error] [pid XXX] [XXX] script '/var/www/html/db_dataml.php' not found or unable to stat
...
[XXX] [XXX] [pid XXX] XXX: Graceful restart requested, doing restart

Does anyone know whats going on? Thanks!

Cicelycicenia

Someone is attempting to hack the server using URLs for things like WordPress logins. Perhaps there's some configuration options to enable firewall blocking of these. If not, one hack would be to add fake pages for those URLs that just echo "no thanks hacker" so they return 200 status. – Amalita

Answer 1

I suspect it's some kind of attack (DDoS).

Maybe someone is scanning your website on different ports and looking for a specific file (w.php) which could be a backdoor or something similar. As the file does not exist it throws errors.

I recommend the following steps:

1. Ensure all of your data is backed up on the server.
2. Re-install the instance from scratch.
3. Ensure the instance is secured per any CIS benchmark.
4. Ensure Apache is secured per any Apache CIS benchmark.
5. Ensure the VPS provider is using an IPS/IDS to monitor your instances, if not find another provider that does.
6. Ensure that all relevant logs are sent to a central syslog server that is not the same as the web server instance. This will improve the integrity of the logs.
7. You might want to install the Snort IPS/IDS solution just to see if another attack is launched.
8. Install a file integrity monitoring solution such as AIDE and monitor config files for changes.

https://benchmarks.cisecurity.org/downloads/multiform/

Caird

Answer 2

Same thing happenend to me. AWS Elastic Beanstalk was checking if a root query was working. This means that a request at the root "/" must respond with a HTTP code 200.

You maybe just need to fix that.

Keeter

Thanks, mate that's really helped, it been there for long, and every time my team leader sees it, asking me when u going to fix that. LOL – Bonnie

How did you fix it? – Sateen

@Keeter How can I define root "/" in configuration. I had the same issue. Here is the link : #75405937 – Contractive

Answer 3

Change the Target group -> Health check settings-> Path to a valid api endpoint of your service

Kathrinkathrine

This may have fixed my issue, but needed to find "Target Group". The whole path is: EC2 -> side nav item Load Balancing: Target Groups -> tab Health Checks -> button Edit -> Health check path – Olympus

Answer 4

I got the same error. Changing the Health check configurations in the elastic beanstalk to a valid API solved it.

You can find the Health check configurations in the elastic beanstalk environment's configurations -> Load balancer -> in the Processes section mark and edit your process -> scroll to the Health check section.

Drench

Answer 5

I received 2 errors while creating an application on Elastic Beanstalk. Some may get any one of the error, some may get both the errors on the rare case. In my scenario, I had to resolve my beanstalk by performing both the solutions provided below.

1. ELB connectivity and health issue:

The Solution for the above problem is to go to the Beanstalk made Ec2 instance and in the target groups, select the group and under that click on the health->edit the health configuration->under that change the path by mentioning your endpoint given in your eclipse or STS tool for the URL requests.

Steps to perform this operation are as follows: From Elastic Beanstalk-> Open Ec2 Service-> Scroll down on Left Navigation pane to find the target groups-> click the checkbox and under that find the Health option-> click health->click edit->change the path by adding your endpoint I.E (/health-I used)-> after all the changes-> Save changes-> go back to your Elastic Beanstalk-> your status will turn into OK. That's all, you are good to go.

2. My AWS Elastic Beanstalk account keeps getting down with the error: "Environment health has transitioned from Ok to Severe. 100.0 % of the requests are erroring with HTTP 4xx"

The solution for the above problem is by adding the security group Inbound rules of the ec2 instance(of Elastic Beanstalk.). you have to add the server port(anything you used in your crud application: 5000,8080,8090,etc).

Steps to perform this operation are as follows: From Elastic Beanstalk->open ec2 Instance->open Security groups->click on Inbound rules->Edit Inbound rules-> Add rule -> [Custom TCP, port no:5000(I used), traffic: Anywhere(0.00.000)]->save changes. open your beanstalk again, the status will turn OK. Hope this will be useful for you people. Have a Good Day.

Aruabea

Answer 6

By default the target group on an EC2 created from EBS is http, in health check protocol if you added a certificate change it to https

Subsocial

Answer 7

I was having the same issue when I was deploying my spring boot application to elasticbeans talk. I was getting "4xx" errors. I fixed it by updating the security groups inside the ec2-instance created by this. it was due to permission issue not accessible from local machine

Lilalilac

Answer 8

In my case my application had outgrown the instance type that was defined during the initial setup.

I upgraded my EB environment's instance type from t3.medium to t3.large and that resolved the issue.

Fortepiano

Answer 9

I experienced the same issue on my ELB servers. Looking at the logs, we were getting a POST to usage.php every second or so. I also saw POST to app_stats.php every few seconds. Since neither of these files exist on my server, it was resulting in a ton of 404 errors which in turn triggered the AWS server status messages. I added the files that would just echo "sorry hacker" and that seemed to remove the flood of 404 errors and restored my server state to healthy.

Soft