Login/Register
APC and child pid XXXXX exit signal Segmentation fault [closed]
Asked Answered
Solved phplinuxapache2apcxcache
F

1

4

First I had xCache installed on server, I had xCache on lot of my servers but in this one after couple of days you go nothing except a blank page with this error in apache error log:

 child pid XXXXX exit signal Segmentation fault

As far as I know this mean some sort of memory corruption.

So I removed xCache from php.ini and installed APC. But things are worse now. After one hour or less/more server become unresponsive. I can clearly see in my website's server-status that there are lot of connections in W mode. And they dont have any thing in common. Some of them are from Joomla (first page) and some from vBulletin (forums) and some other are about other custom codes. If I let it for more time I will going to have this error in Apache error log:

 Server reached MaxClients setting, consider raising the MaxClients setting

Apache is in mod_php and mod_ruid2 mode.

Here is result of server-status when hanged:

Server uptime: 52 minutes 57 seconds
Total accesses: 31025 - Total Traffic: 279.0 MB
CPU Usage: u418.14 s64.28 cu0 cs0 - 15.2% CPU load
9.77 requests/sec - 89.9 kB/second - 9.2 kB/request
398 requests currently being processed, 27 idle workers

WWWWWWWWWWWWWWWWWWWWWWWWW.......................................
WWWWWWWWWWWWWWWWWWWWWWWWW.......................................
WWWWWWWWWWWWWWWWWWWWWWWWW.......................................
WWWWWWWWWWWWWWWWWWWWWWWWW.......................................
WWWWWWWWWWWWWWWWWWWWWWWWW.......................................
WWWWWWWWWWWWWWWWWWWWWWWWW.......................................
WWWWWWWWWWWWWWWWWWWWWWWWW.......................................
WWWWWWWWWWWWWWWWWWWWWWWWW.......................................
WWWWWWWWWWWWWWWWWWWWWWWWW.......................................
WWWWWWWWWWWWWWWWWWWWWWWWW.......................................
WWWWWWWWWWWWWWWWWWWWWWWWW.......................................
WWWWWWWWWWWWWWWWWWWWWWWWW.......................................
WWWWWWWWWWWWWWWWWWWWWWWWW.......................................
WWWWWWWWWWWWWWWWWWWWWWWWW.......................................
WWWWWWWWWWWWWWWWWWWWWWWWW.......................................
WW___WW_WW_W_WWW__WWWW_W_.......................................
_W___W___WW_W_W_W_W______.......................................

Here is error log for xCache (lot of these lines):

[Thu Mar 28 09:50:38 2013] [notice] child pid 1607 exit signal Aborted (6)
httpd: /usr/src/xcache-2.0.0/utils.c:589: xc_sandbox_error_cb: Assertion `sandbox != ((void *)0)' failed.
[Thu Mar 28 09:51:57 2013] [notice] child pid 1643 exit signal Aborted (6)
httpd: /usr/src/xcache-2.0.0/utils.c:589: xc_sandbox_error_cb: Assertion `sandbox != ((void *)0)' failed.
[Thu Mar 28 09:52:32 2013] [notice] child pid 1724 exit signal Aborted (6)
httpd: /usr/src/xcache-2.0.0/utils.c:589: xc_sandbox_error_cb: Assertion `sandbox != ((void *)0)' failed.
[Thu Mar 28 09:55:21 2013] [notice] child pid 1672 exit signal Aborted (6)
httpd: /usr/src/xcache-2.0.0/utils.c:589: xc_sandbox_error_cb: Assertion `sandbox != ((void *)0)' failed.
[Thu Mar 28 09:55:56 2013] [notice] child pid 1471 exit signal Aborted (6)
httpd: /usr/src/xcache-2.0.0/utils.c:589: xc_sandbox_error_cb: Assertion `sandbox != ((void *)0)' failed.
[Thu Mar 28 09:56:49 2013] [notice] child pid 1753 exit signal Aborted (6)
[Thu Mar 28 09:58:29 2013] [notice] child pid 1810 exit signal Segmentation fault (11)
[Thu Mar 28 09:58:29 2013] [notice] child pid 1877 exit signal Segmentation fault (11)
httpd: /usr/src/xcache-2.0.0/utils.c:589: xc_sandbox_error_cb: Assertion `sandbox != ((void *)0)' failed.
[Thu Mar 28 10:01:34 2013] [notice] child pid 1844 exit signal Segmentation fault (11)
[Thu Mar 28 10:01:34 2013] [notice] child pid 1913 exit signal Aborted (6)
[Thu Mar 28 10:01:34 2013] [notice] child pid 1941 exit signal Segmentation fault (11)
[Thu Mar 28 10:01:36 2013] [notice] child pid 2007 exit signal Segmentation fault (11)
[Thu Mar 28 10:01:37 2013] [notice] child pid 2035 exit signal Segmentation fault (11)
[Thu Mar 28 10:01:37 2013] [notice] child pid 2036 exit signal Segmentation fault (11)

Error log for APC before reaching max client (from one restart to other restart):

[Thu Mar 28 19:17:10 2013] [notice] Apache/2.2.24 (Unix) mod_ssl/2.2.24 OpenSSL/1.0.0-fips DAV/2 PHP/5.2.17 configured -- resuming normal operations
[Thu Mar 28 19:21:40 2013] [notice] child pid 1356 exit signal Segmentation fault (11)
[Thu Mar 28 19:22:41 2013] [notice] child pid 1520 exit signal Segmentation fault (11)
[Thu Mar 28 19:22:42 2013] [notice] child pid 1330 exit signal Segmentation fault (11)
[Thu Mar 28 19:22:42 2013] [notice] child pid 1518 exit signal Segmentation fault (11)
[Thu Mar 28 19:22:46 2013] [notice] child pid 1967 exit signal Segmentation fault (11)
[Thu Mar 28 19:22:46 2013] [notice] child pid 1968 exit signal Segmentation fault (11)
[Thu Mar 28 19:22:49 2013] [notice] child pid 1939 exit signal Segmentation fault (11)
[Thu Mar 28 19:22:50 2013] [notice] child pid 2034 exit signal Segmentation fault (11)
[Thu Mar 28 19:22:51 2013] [notice] child pid 2062 exit signal Segmentation fault (11)
[Thu Mar 28 19:22:57 2013] [notice] child pid 2091 exit signal Segmentation fault (11)
[Thu Mar 28 19:22:58 2013] [notice] child pid 2090 exit signal Segmentation fault (11)
[Thu Mar 28 19:25:34 2013] [notice] caught SIGTERM, shutting down
[Thu Mar 28 19:25:37 2013] [notice] SSL FIPS mode disabled
[Thu Mar 28 19:25:37 2013] [warn] RSA server certificate CommonName (CN) `localhost' does NOT match server name!?
[Thu Mar 28 19:25:37 2013] [warn] Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)
[Thu Mar 28 19:25:37 2013] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu Mar 28 19:25:38 2013] [notice] SSL FIPS mode disabled
[Thu Mar 28 19:25:38 2013] [warn] RSA server certificate CommonName (CN) `localhost' does NOT match server name!?
[Thu Mar 28 19:25:38 2013] [warn] Init: Name-based SSL virtual hosts only work for clients with TLS server name indication support (RFC 4366)
[Thu Mar 28 19:25:38 2013] [notice] Apache/2.2.24 (Unix) mod_ssl/2.2.24 OpenSSL/1.0.0-fips DAV/2 PHP/5.2.17 configured -- resuming normal operations

Little from PHP Info:

System  Linux xxx.xxxxx.com 2.6.32-358.0.1.el6.x86_64 #1 SMP Wed Feb 27 06:06:45 UTC 2013 x86_64
Build Date  Mar 17 2013 12:55:14
Configure Command   './configure' '--with-apxs2' '--with-curl=/usr/local/lib' '--with-gd' '--with-ttf' '--with-gettext' '--with-jpeg-dir=/usr/local/lib' '--with-freetype-dir=/usr/local/lib' '--with-kerberos' '--with-openssl' '--with-mcrypt' '--with-mhash' '--with-mysql=/usr' '--with-mysqli=/usr/bin/mysql_config' '--with-pdo-mysql=/usr' '--with-pcre-regex=/usr/local' '--with-pear' '--with-png-dir=/usr/local/lib' '--with-xsl' '--with-zlib' '--with-zlib-dir=/usr/local/lib' '--with-iconv=/usr/local' '--enable-bcmath' '--enable-calendar' '--enable-exif' '--enable-ftp' '--enable-gd-native-ttf' '--enable-magic-quotes' '--enable-safe-mode' '--enable-soap' '--enable-sockets' '--enable-mbstring' '--enable-zip' '--enable-wddx'
Server API  Apache 2.0 Handler
Virtual Directory Support   enabled
Configuration File (php.ini) Path   /usr/local/lib
Loaded Configuration File   /usr/local/lib/php.ini
Scan this dir for additional .ini files     (none)
additional .ini files parsed    (none)
PHP API     20041225
PHP Extension   20060613
Zend Extension  220060519
Debug Build     no
Thread Safety   enabled
Zend Memory Manager     enabled
IPv6 Support    enabled
Registered PHP Streams  https, ftps, compress.zlib, php, file, data, http, ftp, zip
Registered Stream Socket Transports     tcp, udp, unix, udg, ssl, sslv3, sslv2, tls
Registered Stream Filters   zlib.*, convert.iconv.*, string.rot13, string.toupper, string.tolower, string.strip_tags, convert.*, consumed

APC settings:

enabled
Version     3.1.13
APC Debugging   Disabled
MMAP Support    Enabled
MMAP File Mask  no value
Locking type    pthread read/write Locks
Serialization Support   php
Revision    $Revision: 327136 $
Build Date  Mar 28 2013 12:40:18

Php -v:

PHP 5.2.17 (cli) (built: Mar 17 2013 12:55:24) 
Copyright (c) 1997-2010 The PHP Group
Zend Engine v2.2.0, Copyright (c) 1998-2010 Zend Technologies

httpd -V:

Server version: Apache/2.2.24 (Unix)
Server built:   Mar 10 2013 09:18:00
Server's Module Magic Number: 20051115:31
Server loaded:  APR 1.4.6, APR-Util 1.4.1
Compiled using: APR 1.4.6, APR-Util 1.4.1
Architecture:   64-bit
Server MPM:     Worker
  threaded:     yes (fixed thread count)
    forked:     yes (variable process count)
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/worker"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=128
 -D HTTPD_ROOT="/etc/httpd"
 -D SUEXEC_BIN="/usr/sbin/suexec"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"

Any Idea?! I will reward 50 :)

Frymire answered 28/3, 2013 at 18:42 Comment(5)
I don't have an exact idea what is causing the problem but there are some things I would try. Move to fast cgi configuration of php to isolate it from apache (this might mean you need to move to php 5.3.x however). You could try a regular cgi configuration if you can't upgrade php. – Nieman
I was on CGI month ago. very bad performance. Very bad. And no way to use opcode caches! So I moved to mod_php. I dont like to go to same place I was :),but thanks any way. – Frymire
fcgi is way better than both mod_php and cgi. But its only supported without patches in php 5.3.3 and up. ut you can still find patches for earlier versions with under php-fpm. – Nieman
@Nieman Well, PHP 5.2 itself hasn't been supported for over 2 years now, so upgrading shouldn't be a big deal. Something more concrete than "way better" regarding fcgi would be nice, though... – Rimester
@Nieman Incidentally, the OP mentions mod_ruid2 which I hadn't heard of, but is a security module, and is incompatible with fcgi. – Rimester
Z
5

From the looks of your server status page, you are trying to run PHP as mod_php with a threaded Apache MPM. Many PHP extensions, likely including APC and Xcache, are not thread-safe. As such, they will crash frequently under load.

You will need to switch Apache to the prefork MPM, or run PHP as CGI or FastCGI.

Zahavi answered 30/3, 2013 at 23:1 Comment(7)
Out of interest, what was the clue that led you to that conclusion? I thought even pre-fork talked about "workers"... – Rimester
If you're right, though, it looks like it's incompatible with mod_ruid2 as well – Rimester
The clue is the way that the workers are displayed on the status page. Prefork workers show up all the way across the page, threaded MPMs only go partway across. – Zahavi
Server is in worker and it said threaded and forked both yes. I am not familiar with this forked/worker things. But in status page there was lot of . items that I ignored when copying here. – Frymire
How to switch Apache to the prefork MPM? Is there any article or info? – Frymire
Thank you. Although I was using Apache as prefork, I disabled XCache and this error disappeared. – Colwen
Been years since this, but in the last decade I have learned more about the issue and sure it does seem like the most probable reason. marked as the correct answer. – Frymire

© 2022 - 2024 — McMap. All rights reserved.