apache server reached MaxClients setting, consider raising the MaxClients setting

Question

I am running centos 5.5 with 768mb ram. i keep getting server reached MaxClients setting, consider raising the MaxClients setting in the logs also apache runs really slow. when i look at cacti graphs it shows the server is not even using all the resources.. here is the current configuration

<IfModule prefork.c>
StartServers       8
MinSpareServers    5
MaxSpareServers    10
ServerLimit        1024
MaxClients         768
MaxRequestsPerChild  4000
</IfModule>

<IfModule worker.c>
StartServers         2
MaxClients         150
MinSpareThreads     25
MaxSpareThreads     75 
ThreadsPerChild     25
MaxRequestsPerChild  0
</IfModule>



free -m
             total       used       free     shared    buffers     cached
Mem:           768        352        415          0          0         37
-/+ buffers/cache:        315        452
Swap:            0          0          0



top - 11:03:54 up 41 days, 11:53,  1 user,  load average: 0.05, 0.03, 0.00
Tasks:  35 total,   1 running,  34 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni, 99.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.3%st
Mem:    786432k total,   389744k used,   396688k free,        0k buffers
Swap:        0k total,        0k used,        0k free,    38284k cached

I have tried the following but the server responds very slowly

<IfModule worker.c>
#StartServers         2
#MaxClients         150
#MinSpareThreads     25
#MaxSpareThreads     75
#ThreadsPerChild     25
#MaxRequestsPerChild  0

StartServers    20
MaxClients      1024
ServerLimit     1024
MinSpareThreads 128
MaxSpareThreads 768
ThreadsPerChild 64
MaxRequestsPerChild 0
</IfModule>

free -m
             total       used       free     shared    buffers     cached
Mem:           768        324        443          0          0         37
-/+ buffers/cache:        286        481
Swap:            0          0          0

@regilero

I have updated to

<IfModule prefork.c>
  StartServers       12
  MinSpareServers    12
  MaxSpareServers    12
  MaxClients         50
  MaxRequestsPerChild  300
</IfModule>

using top i see

Tasks:  36 total,   1 running,  35 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.3%sy,  0.0%ni, 99.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:    786432k total,   613180k used,   173252k free,        0k buffers
Swap:        0k total,        0k used,        0k free,    76488k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                                                         
    1 root      20   0 10364   92   60 S  0.0  0.0   1:09.53 init                                                                                                                                                                            
    2 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kthreadd/808                                                                                                                                                                    
    3 root      20   0     0    0    0 S  0.0  0.0   0:00.00 khelper/808                                                                                                                                                                     
  124 root      16  -4 12620    8    4 S  0.0  0.0   0:00.00 udevd                                                                                                                                                                           
  533 root      20   0 95504 5692  228 S  0.0  0.7   4:02.94 memcached                                                                                                                                                                       
  546 root      20   0  5924  332  276 S  0.0  0.0   6:54.51 syslogd                                                                                                                                                                         
  557 root      20   0  101m 1456  868 S  0.0  0.2  13:18.64 snmpd                                                                                                                                                                           
  570 root      20   0 62640  316  208 S  0.0  0.0   2:39.56 sshd                                                                                                                                                                            
  579 root      20   0 21656   24   20 S  0.0  0.0   0:00.00 xinetd                                                                                                                                                                          
  589 root      20   0 12072   12    8 S  0.0  0.0   0:00.05 mysqld_safe                                                                                                                                                                     
  940 mysql     20   0  559m 164m 3832 S  0.3 21.5 209:33.88 mysqld                                                                                                                                                                          
 1015 root      20   0 20880  200  132 S  0.0  0.0   0:10.48 crond                                                                                                                                                                           
 1023 root      20   0 46748    4    0 S  0.0  0.0   0:00.00 saslauthd                                                                                                                                                                       
 1024 root      20   0 46748    4    0 S  0.0  0.0   0:00.00 saslauthd                                                                                                                                                                       
 3605 root      20   0 62832 2168  636 S  0.0  0.3   0:02.58 sendmail                                                                                                                                                                        
 3613 smmsp     20   0 57712 1648  504 S  0.0  0.2   0:00.01 sendmail                                                                                                                                                                        
17610 root      20   0 85932 3312 2600 S  0.0  0.4   0:00.02 sshd                                                                                                                                                                            
17612 mcmap     20   0 86072 1760 1012 S  0.0  0.2   0:00.17 sshd                                                                                                                                                                            
17613 mcmap     20   0 12076 1656 1292 S  0.0  0.2   0:00.01 bash                                                                                                                                                                            
17637 root      20   0 45052 1432 1120 S  0.0  0.2   0:00.00 su                                                                                                                                                                              
17638 root      20   0 12180 1800 1324 S  0.0  0.2   0:00.08 bash                                                                                                                                                                            
17740 root      20   0  246m 9264 4516 S  0.0  1.2   0:00.19 httpd                                                                                                                                                                           
18264 apache    20   0  282m  43m 4940 S  0.0  5.7   0:00.56 httpd                                                                                                                                                                           
18514 apache    20   0  279m  40m 4832 S  0.0  5.3   0:01.47 httpd                                                                                                                                                                           
18518 apache    20   0  273m  36m 4396 S  0.0  4.7   0:00.45 httpd                                                                                                                                                                           
18528 apache    20   0  251m  13m 3660 S  0.0  1.8   0:00.41 httpd                                                                                                                                                                           
18529 apache    20   0  278m  40m 4340 S  0.0  5.3   0:00.99 httpd                                                                                                                                                                           
18530 apache    20   0  278m  40m 4268 S  0.0  5.3   0:00.67 httpd                                                                                                                                                                           
18548 apache    20   0  272m  33m 3516 S  0.0  4.4   0:00.28 httpd                                                                                                                                                                           
18552 apache    20   0  280m  42m 3684 S  0.0  5.5   0:00.48 httpd                                                                                                                                                                           
18553 apache    20   0  271m  33m 3768 S  0.0  4.3   0:00.45 httpd                                                                                                                                                                           
18555 apache    20   0  274m  36m 3672 S  0.0  4.7   0:00.58 httpd                                                                                                                                                                           
18572 apache    20   0  247m 9020 2856 S  0.0  1.1   0:00.01 httpd                                                                                                                                                                           
18578 apache    20   0  280m  42m 3684 S  0.0  5.6   0:00.76 httpd                                                                                                                                                                           
18589 apache    20   0  246m 5452  676 S  0.0  0.7   0:00.00 httpd                                                                                                                                                                           
18588 root      20   0 12624 1216  932 R  0.0  0.2   0:00.06


free -m
             total       used       free     shared    buffers     cached
Mem:           768        578        189          0          0         74
-/+ buffers/cache:        504        263
Swap:            0          0          0

Just added current picture of cacti result last 4 hours. busy periods are monday tuesday. So i will wait till next week to see further results of the config change. but it looks like an improvement as before i only had max 10 threads available. Looking at this do you think i can make more improvment?

free -m
             total       used       free     shared    buffers     cached
Mem:           768        619        148          0          0         49
-/+ buffers/cache:        570        197
Swap:            0          0          0

NEW TEST

On a 2GB Ram VPS box i have now set prefork to

StartServers      20
MinSpareServers   20
MaxSpareServers   20
ServerLimit  256
MaxClients   256
MaxRequestsPerChild  4000

today morning my memcache server died from

Nov 20 09:28:40 vps22899094 kernel: Out of memory: Kill process 12517 (memcached) score 81 or sacrifice child
Nov 20 09:28:40 vps22899094 kernel: Killed process 12517, UID 497, (memcached) total-vm:565252kB, anon-rss:42940kB, file-rss:44kB

What should the optimal values be to set in apache?

#/etc/sysconfig/memcached

PORT="11211"
USER="memcached"
MAXCONN="1024"
CACHESIZE="1024"
OPTIONS="-l 127.0.0.1"

/etc/my.cnf

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
user=mysql
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0

bind-address=127.0.0.1

#script
thread_concurrency=2
query_cache_size = 16M
query_cache_type=1
query_cache_limit=5M

# MyISAM #
#key-buffer-size                = 32M
#myisam-recover                 = FORCE,BACKUP

# SAFETY #
#max-allowed-packet             = 16M
#max-connect-errors             = 1000000

# CACHES AND LIMITS #
tmp-table-size                 = 32M
max-heap-table-size            = 32M
#query-cache-type               = 0
#query-cache-size               = 0
max-connections                = 50
thread-cache-size              = 16
#open-files-limit               = 65535
#table-definition-cache         = 1024
#table-open-cache               = 2048

# INNODB #
#innodb-flush-method            = O_DIRECT
#innodb-log-files-in-group      = 2
#innodb-log-file-size           = 5M
#innodb-flush-log-at-trx-commit = 1
#innodb-file-per-table          = 1
#innodb-buffer-pool-size        = 921M
# LOGGING #
log-error                      = /var/log/mysqld.log
log-queries-not-using-indexes  = 1
slow-query-log                 = 1
slow-query-log-file            = /var/log/mysqld-slow.log

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

Mandel

i am trying to debug connection drops on our apache server as well, clients fail to get connected to the server. Can you tell me which tool is this that is generating such useful graphs? – Milson

Note that as Apache 2.4, MaxClients has been renamed to MaxRequestWorkers, which describes more accurately what it does. – Nicolas

Answer 1

When you use Apache with mod_php apache is enforced in prefork mode, and not worker. As, even if php5 is known to support multi-thread, it is also known that some php5 libraries are not behaving very well in multithreaded environments (so you would have a locale call on one thread altering locale on other php threads, for example).

So, if php is not running in cgi way like with php-fpm you have mod_php inside apache and apache in prefork mode. On your tests you have simply commented the prefork settings and increased the worker settings, what you now have is default values for prefork settings and some altered values for the shared ones :

StartServers       20
MinSpareServers    5
MaxSpareServers    10
MaxClients         1024
MaxRequestsPerChild  0

This means you ask apache to start with 20 process, but you tell it that, if there is more than 10 process doing nothing it should reduce this number of children, to stay between 5 and 10 process available. The increase/decrease speed of apache is 1 per minute. So soon you will fall back to the classical situation where you have a fairly low number of free available apache processes (average 2). The average is low because usually you have something like 5 available process, but as soon as the traffic grows they're all used, so there's no process available as apache is very slow in creating new forks. This is certainly increased by the fact your PHP requests seems to be quite long, they do not finish early and the apache forks are not released soon enough to treat another request.

See on the last graphic the small amount of green before the red peak? If you could graph this on a 1 minute basis instead of 5 minutes you would see that this green amount was not big enough to take the incoming traffic without any error message.

Now you set 1024 MaxClients. I guess the cacti graph are not taken after this configuration modification, because with such modification, when no more process are available, apache would continue to fork new children, with a limit of 1024 busy children. Take something like 20MB of RAM per child (or maybe you have a big memory_limit in PHP and allows something like 64MB or 256MB and theses PHP requests are really using more RAM), maybe a DB server... your server is now slowing down because you have only 768MB of RAM. Maybe when apache is trying to initiate the first 20 children you already reach the available RAM limit.

So. a classical way of handling that is to check the amount of memory used by an apache fork (make some top commands while it is running), then find how many parallel request you can handle with this amount of RAM (that mean parallel apache children in prefork mode). Let's say it's 12, for example. Put this number in apache mpm settings this way:

<IfModule prefork.c>
  StartServers       12
  MinSpareServers    12
  MaxSpareServers    12
  MaxClients         12
  MaxRequestsPerChild  300
</IfModule>

That means you do not move the number of fork while traffic increase or decrease, because you always want to use all the RAM and be ready for traffic peaks. The 300 means you recyclate each fork after 300 requests, it's better than 0, it means you will not have potential memory leaks issues. MaxClients is set to 12 25 or 50 which is more than 12 to handle the ListenBacklog queue, which can enqueue some requests, you may take a bigger queue, but you would get some timeouts maybe (removed this strange sentende, I can't remember why I said that, if more than 12 requests are incoming the next one will be pushed in the Backlog queue, but you should set MaxClient to your targeted number of processes).

And yes, that means you cannot handle more than 12 parallel requests.

If you want to handle more requests:

• buy some more RAM
• try to use apache in worker mode, but remove mod_php and use php as a parallel daemon with his own pooler settings (this is called php-fpm), connect it with fastcgi. Note that you will certainly need to buy some RAM to allow a big number of parallel php-fpm process, but maybe less than with mod_php
• Reduce the time spent in your php process. From your cacti graphs you have to potential problems: a real traffic peak around 11:25-11:30 or some php code getting very slow. Fast requests will reduce the number of parallel requests.

If your problem is really traffic peaks, solutions could be available with caches, like a proxy-cache server. If the problem is a random slowness in PHP then... it's an application problem, do you do some HTTP query to another site from PHP, for example?

And finally, as stated by @Jan Vlcinsky you could try nginx, where php will only be available as php-fpm. If you cannot buy RAM and must handle a big traffic that's definitively desserve a test.

Update: About internal dummy connections (if it's your problem, but maybe not).

Check this link and this previous answer. This is 'normal', but if you do not have a simple virtualhost theses requests are maybe hitting your main heavy application, generating slow http queries and preventing regular users to acces your apache processes. They are generated on graceful reload or children managment.

If you do not have a simple basic "It works" default Virtualhost prevent theses requests on your application by some rewrites:

  RewriteCond %{HTTP_USER_AGENT} ^.*internal\ dummy\ connection.*$ [NC]
  RewriteRule .* - [F,L]

Update:

Having only one Virtualhost does not protect you from internal dummy connections, it is worst, you are sure now that theses connections are made on your unique Virtualhost. So you should really avoid side effects on your application by using the rewrite rules.

Reading your cacti graphics, it seems your apache is not in prefork mode bug in worker mode. Run httpd -l or apache2 -l on debian, and check if you have worker.c or prefork.c. If you are in worker mode you may encounter some PHP problems in your application, but you should check the worker settings, here is an example:

<IfModule worker.c>
  StartServers           3
  MaxClients           500
  MinSpareThreads       75
  MaxSpareThreads      250 
  ThreadsPerChild       25
  MaxRequestsPerChild  300
</IfModule>

You start 3 processes, each containing 25 threads (so 3*25=75 parallel requests available by default), you allow 75 threads doing nothing, as soon as one thread is used a new process is forked, adding 25 more threads. And when you have more than 250 threads doing nothing (10 processes) some process are killed. You must adjust theses settings with your memory. Here you allow 500 parallel process (that's 20 process of 25 threads). Your usage is maybe more:

<IfModule worker.c>
  StartServers           2
  MaxClients           250
  MinSpareThreads       50
  MaxSpareThreads      150 
  ThreadsPerChild       25
  MaxRequestsPerChild  300
</IfModule>

Picayune

the thing is there isn't much traffice in sense of many users. cacti does not even show 10% cpu or ram usage, everything is on low usage on cacti graphs. i will change the values you mentioned above and let you know what happens. – Mandel

@sharif: you have a 5 minutes gap with cacti, which is a long silen time, check load average history also. Now you have only 9 parallel requests busy and they are maybe stalled (application problem?), and apache gets more and more incoming requests and do not create new childrens. Check all the access logs available (even the default VH and also errors logs), could also be the internal dummy connections problems. – Picayune

So you have something somewhere which set this 300, isn't it? – Picayune

that was my mistake sorry – Mandel

i do have internal dummy connections. how do i stop this? ::1 - - [17/Oct/2013:18:03:04 +0100] "OPTIONS * HTTP/1.0" 302 304 "-" "Apache/2.2.3 (CentOS) (internal dummy connection)" 0/525 ::1 - - [17/Oct/2013:18:03:05 +0100] "OPTIONS * HTTP/1.0" 302 304 "-" "Apache/2.2.3 (CentOS) (internal dummy connection)" 0/851 ::1 - - [17/Oct/2013:18:03:06 +0100] "OPTIONS * HTTP/1.0" 302 304 "-" "Apache/2.2.3 (CentOS) (internal dummy connection)" 0/789 . I will see what cacti shows over the next day – Mandel

I have also just noticed the following pop up. [Thu Oct 17 18:07:40 2013] [error] server reached MaxClients setting, consider raising the MaxClients setting – Mandel

Check also the other apache status data that you may have in cacti, like the number of requests received. I've added some input about internal dummy conn – Picayune

updated question. i have read about the dummy connections and think its not a big issue with those. i am just running one site no virtual hosts. – Mandel

sharif: get all infos (as for example sharing a memcached server here has an impact on memroy, is the mySQL server also on the same hardxare? is your apache using worker module or prefork module?) then ask questions on servfault. – Picayune

@regiero. i have memcache, mysql5.1 apache all one one server. its not a busy server and my main problem was threads which you helped me out with. however with the above settings just killed memcached all of a sudden. i have posted all config above – Mandel

@sharif: time to ask new questions. This one is closed for me. – Picayune

Very detailed questino and answer. Should add the tag/keyword "thread-safety" for google to find this? – Nonmoral

Answer 2

Here's an approach that could resolve your problem, and if not would help with troubleshooting.

1. Create a second Apache virtual server identical to the current one

2. Send all "normal" user traffic to the original virtual server

3. Send special or long-running traffic to the new virtual server

Special or long-running traffic could be report-generation, maintenance ops or anything else you don't expect to complete in <<1 second. This can happen serving APIs, not just web pages.

If your resource utilization is low but you still exceed MaxClients, the most likely answer is you have new connections arriving faster than they can be serviced. Putting any slow operations on a second virtual server will help prove or disprove. Use the Apache access logs to quantify.

Maronite

Answer 3

Did you consider using nginx (or other event based web server) instead of apache?

nginx shall allow higher number of connections and consume much less resources (as it is event based and does not create separate process per connection). Anyway, you will need some processes, doing real work (like WSGI servers or so) and if they stay on the same server as the front end web server, you only shift the performance problem to a bit different place.

Latest apache version shall allow similar solution (configure it in event based manner), but this is not my area of expertise.

Hagai

do i have to use nginx. i have been using just apache for very long time. not so familiar with nginx – Mandel

apache is running wordpress but my main content runs in a iframe all php making google map api requests. www.spectrumasa.com/mcmap is the content creating this issue. – Mandel

Answer 4

I recommend to use bellow formula suggested on Apache:

MaxClients = (total RAM - RAM for OS - RAM for external programs) / (RAM per httpd process)

Find my script here which is running on Rhel 6.7. you can made change according to your OS.

#!/bin/bash

echo "HostName=`hostname`"

#Formula
#MaxClients . (RAM - size_all_other_processes)/(size_apache_process)
total_httpd_processes_size=`ps -ylC httpd --sort:rss | awk '{ sum += $9 } END { print sum }'`
#echo "total_httpd_processes_size=$total_httpd_processes_size"
total_http_processes_count=`ps -ylC httpd --sort:rss | wc -l`
echo "total_http_processes_count=$total_http_processes_count"
AVG_httpd_process_size=$(expr $total_httpd_processes_size / $total_http_processes_count)
echo "AVG_httpd_process_size=$AVG_httpd_process_size"
total_httpd_process_size_MB=$(expr $AVG_httpd_process_size / 1024)
echo "total_httpd_process_size_MB=$total_httpd_process_size_MB"
total_pttpd_used_size=$(expr $total_httpd_processes_size / 1024)
echo "total_pttpd_used_size=$total_pttpd_used_size"
total_RAM_size=`free -m |grep Mem |awk '{print $2}'`
echo "total_RAM_size=$total_RAM_size"
total_used_size=`free -m |grep Mem |awk '{print $3}'`
echo "total_used_size=$total_used_size"
size_all_other_processes=$(expr $total_used_size - $total_pttpd_used_size)
echo "size_all_other_processes=$size_all_other_processes"
remaining_memory=$(($total_RAM_size - $size_all_other_processes))
echo "remaining_memory=$remaining_memory"
MaxClients=$((($total_RAM_size - $size_all_other_processes) / $total_httpd_process_size_MB))
echo "MaxClients=$MaxClients"
exit

Cobbler