To many long running Apache processes in READ status without requests after a certain time

Question

I am using /server-status to monitor the Apache processes. When starting they look like this:

_____W_K__K____________C_K________C_____________W_..............
................................................................
................................................................

But after few hours of running look like this:

R_KCR___KR__RKRR_RRRKRRRRRRKRR_RRCK____R_RRRR_RRRKRRRKRRRRRRRRR_
R_RRRR_R.RR.R_R.R_R..CKRRRRW.K_RCRKRR_R_.._R._.RK_KRK_.RRR.KK_.R
..RR............................................................

There is too many "Read" (R) status which took a long time and I don't know what they do, because they even have no requests. (Mention that there are additional "." statuses which I skipped from the example above, in total I have 2000 positions available.) In the process list I have approx 40 "R" processes like this one:

Srv  PID   Acc        M CPU  SS   Req Conn Child Slot   Client       VHost Request
15-2 21291 0/37/11158 R 0.03 7468 2   0.0  1.93  198.35 82.78.95.105

The report's header look like this:

Server Version: Apache/2.4.10 (Debian) mod_fcgid/2.3.9 OpenSSL/1.0.1t
Server MPM: prefork
Server Built: Sep 15 2016 20:44:43
Current Time: Thursday, 12-Jan-2017 08:38:46 EET
Restart Time: Wednesday, 11-Jan-2017 00:51:18 EET
Parent Server Config. Generation: 3
Parent Server MPM Generation: 2
Server uptime: 1 day 7 hours 47 minutes 27 seconds
Server load: 0.34 0.35 0.39
Total accesses: 1599556 - Total Traffic: 29.9 GB
CPU Usage: u18.87 s6.81 cu0 cs0 - .0224% CPU load
14 requests/sec - 274.0 kB/second - 19.6 kB/request
90 requests currently being processed, 27 idle workers

I have a private server with: 4X3.6GHZ Xeon processor, 2X512 GB SSD, 32 GB memory, Debian 3.16.36-1, Apache 2.4.10, PHP 5.6.29, mod_fcgid, php-fpm, php5-cgi, prefork.

Some of my apache2.conf settings:

Timeout 14400
KeepAlive On
MaxKeepAliveRequests 1000
KeepAliveTimeout 3
HostnameLookups Off

My mpm_prefork.conf settings:

<IfModule mpm_prefork_module>
  StartServers 50
  MinSpareServers 25
  MaxSpareServers 50
  ServerLimit 2000
  MaxRequestWorkers 2000
  MaxConnectionsPerChild 1000
</IfModule>

Some of my fcgid.conf settings:

FcgidMaxRequestLen 1073741824
FcgidOutputBufferSize 1073741824
FcgidMaxProcesses 200
FcgidMaxProcessesPerClass 100
FcgidMinProcessesPerClass 0
FcgidProcessLifeTime 30
FcgidConnectTimeout 30
FcgidIOTimeout 14400
FcgidBusyTimeout 14400
FcgidIdleTimeout 3
FcgidIdleScanInterval 1

I have some long runing cronjobs in the servers, this is why I need that big "14400" (=4 hours) timeout on Apache and PHP processes.

Questions:

1. Why there are so many "R" statuses without requests?
2. What they do, how can I find out?
3. Why they run so long?

---

Update 1:

A have modified the timeouts to 7200 (=2 hours), this is the minimum for me to can run my cronjobs. However I did not have problems with my cronjobs.

FcgidIOTimeout 7200
FcgidBusyTimeout 7200

Also for Apache:

Timeout 7200

Based on PHP and mod_fcgid: ap_pass_brigade failed in handle_request_ipc function I did this:

FcgidOutputBufferSize 0

Based on mod_fcgid: ap_pass_brigade failed in handle_request function I did this:

FcgidMaxRequestsPerProcess 500

After few hours the server stopped responding:

Service Unavailable: The server is temporarily unable to service your request due to maintanance downtime or capacity problems. Please try again later.

In the logs I found some erros like this two:

[fcgid:warn] (104)Connection reset by peer: mod_fcgid: ap_pass_brigade failed in handle_request_ipc function
[fcgid:warn] (32)Broken pipe: mod_fcgid: ap_pass_brigade failed in handle_request_ipc function

And a bunch of like this (maybe for each requests):

[fcgid:warn] mod_fcgid: too much processes, please increase FCGID_MAX_APPLICATION

Questions 2:

4. What are these errors mean? How can I correct them?
5. Why the server stops responding? Probably because of errors...
6. Are the mod_fcgid, php-fpm, php5-cgi, prefork modules fully compatible and effective? This was the default configurations when I got the new server.

Claw

The key here is your comment "I have some long runing cronjobs in the servers, this is why I need that big "14400" (=4 hours) timeout on Apache and PHP processes." Stop them, problem persists? Commands like lsof will show you active connections for those children. And on another note, if you have mod_fcgid you don't need prefork and can switch to a threaded mpm like event which will scale much better. Open slots reflect your spareservers directives. OPTIONS requests are from something of yours, apache does not ping itself. – Langmuir

in any case, try reducing your timeouts to normal values and you will see this behaviour stops. – Langmuir

Your php scripts are hanging there forever because for some reason mod_fcgid can't spawn more processes, for what I read "FCGID_MAX_APPLICATION" is an internal table which can just be changed on source and re-compilation. Which version of mod_fcgid are you using? If not the last maybe you should upgrade. My recommendation? Migrate to mod_proxy_fcgi -> php-fpm, this will let you have apache with mpm_event and php-fpm is quite versatile and allows you very different isolated setups. – Langmuir

mod_fcgid version: 1:2.3.9-1+b1 – Claw

Can you get a backtrace of one that is old and has a Req field of 2 or more? – Lucan

prefork and fastcgi is a weird combination. – Lucan

How does the length of a cron job relate to the TImeout in Apache? – Lucan

the php cronjob runs nightly and it need aprox 1-2 hours to terminate. if i set the timeout to less than 60*60*2=7200, than the cronboj will be killed without termination. it's a maintanance and calculation cronjob and i need it – Claw

Haven't you considered that this is a sign of DoS attack (SlowRead)? When did such behavior start? Or it was from day 1? – Jehias

why don't you call your cron jobs from the command line? (rather than going via apache) – Imtiaz

presumably they're called from cron – Lucan

To be clear: all of the mentioned cronjobs are PHP scripts, that's why I need to use a big timeout for PHP. – Claw

@BuzoganyLaszlo FWIW, I don't believe the php max_execution_time has any effect on crons, to quote the phpdoc for max_execution_time: When running PHP from the command line the default setting is 0. php.net/manual/en/info.configuration.php#ini.max-execution-time – Huzzah

I have also set the max_input_time to a high value. – Claw

Answer 1

Problem solved.

The solution was to lower the Apache's Timeout value to a number like 15. As I realized for running a long PHP script (even for hours) do not need to this timeout to be hight, it's enought that the PHP's max_execution_time to be big enought.

Update

I have also upgraded to PHP 7.1 with FastCGI under PHP-FPM, and I have changed the Apache's MPM mode to event as esra-s suggested, and it runs like hell. Many thanks!

Claw

Can you elaborate on this? Are you saying PHP 7.1 can now run under Apache event mpm, and is thread-safe? – Beery