High CPU mod_fcgid: can't apply process slot

Question

I get the following warning:

mod_fcgid: can't apply process slot for /var/www/cgi-bin/cgi_wrapper/cgi_wrapper

As soon as this warning appears server CPU raises to 97% - 101%

The server gets around 140K pageviews daily. This warning appears mostly around 11:00-12:00 Besides the 140K pageviews calculated by Analytics it also makes requests from third party websites that are displaying a text-ad on their website. Around 2700 websites are displaying one advertisement and are making a total of 22139 requests per hour together to the server.

I tried: chmod 755 /var/log/httpd but there isn't a folder "httpd" in /var/log

I tried to adjust MaxSpareServers and MaxClients to whatever possible high values. But doesn't effect at all. After Apache has restarted it seems to be fine for a while (server is still responding little bit slow and warnings are no longer appearing in the log file). But after a while the same warning appears and CPU raises again.

If someone has suggestions please let me know, thanks all.

Spontaneous

I honestly have no idea why this was migrate away from serverfault. – Preglacial

Answer 1

This warning tells you that the FastCGI process pool is exhausted, and it has a global limit of FcgidMaxProcesses and a per-script limit of FcgidMaxProcessesPerClass (see http://httpd.apache.org/mod_fcgid/mod/mod_fcgid.html#fcgidmaxprocesses).

The size of this pool may be tuned with the following information :

• in the case where your application is CPU bound and you don't want it to slow down, you will set FcgidMaxProcesses to the number of CPU cores on your server; the server average load will tend to this number at peak hours. Your server will be serving clients without any slow down but will also reject some requests to keep up. Rejects are logged "can't apply process slot" on the server side and seen as 503 errors on the client side
• in the (common) case where your application is not totally CPU-bound, for instance it waits a fair amount on its database server for replies (or other I/O or remote resources), you may have FcgidMaxProcesses a bit higher than your number of CPU cores. As a rule of thumb I start with twice the number of CPU cores then check the peak average load for a whole day (a week is better). Then adjust until it peaks at avg. load = number of CPU cores.

In any case the default value of FcgidMaxProcesses (1000) is outrageous. You run a dumb 'ab -c100 -n10000 <proper-url-which-calls-in-fastcgi' from a poor 3G access and kill the server in 30 sec, tested and approved.

Just to be sure, let me state it another way : the "can't apply process slot" error is a normal error, because your server has a fixed amount of resources, and those limits may be hit. If you push that limit to high wrt. to your server resources, the warning might never appear again, but your server will crumble under the load.

And BTW, MaxSpareServers and MaxClients are not related to your problem, they regulate the Apache process pool, not the FastCGI process pool, which is another topic.

Breadstuff

Thank you for your explanation regarding this error. I disabled my display network 2 days ago to see what would happen, and the server is running just fine now. I contacted a company that is specialized in optimizing servers, and load balancing with Nginx. I think it's just to much for one server to handle 650K requests p/d. I will reply here with the status after this company has set up a load balanced enviroment/network or how you would call it. – Spontaneous

I'm working in this kind of 'optimization company'. Throwing a Nginx might help but it's not an automatic solution. To this day, I have only known 1 case (out of hundreds, really) where I could not configure Apache in order it's not the bottleneck. Apache is very performant and predictable, but it's not the hype. Most people saying 'use Nginx instead' simply don't have a clue about Apache wich is not complicated and quite well documented. – Breadstuff

@Breadstuff This is my experience, too. I work at an ISP with A LOT of apaches, almost no nginx here. – Preglacial

I have a dedicated server, eight core cpu / eight thread, 64GB ram and about 10 sites with 10/20 user online (around 8000/9000 session at month). I see a lot of 503, i prefer to slow down things, but never show a 503... what i can do? My settings: FcgidIdleTimeout 180 FcgidProcessLifeTime 300 FcgidMaxProcesses 300 FcgidMaxProcessesPerClass 90 FcgidMinProcessesPerClass 0 FcgidConnectTimeout 600 FcgidIOTimeout 600 FcgidIdleScanInterval 60 FcgidMaxRequestsPerProcess 10000 FcgidInitialEnv PHP_FCGI_MAX_REQUESTS 10000 – Treachery

Wondering if it's possble to queue requests that surpass FcgidMaxProcessesPerClass to handle short bursts. – Whortleberry