MySQL driver segfaulting under mod_perl - where to look for issue

Question

I have a webapp that segfaults when the database in restarted and it tries to use the old connections. Running it under gdb --args apache -X leads to the following output:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -1212868928 (LWP 16098)]
0xb7471c20 in mysql_send_query () from /usr/lib/libmysqlclient.so.15

I've checked that the drivers and database are all up to date (DBD::mysql 4.0008, MySQL 5.0.32-Debian_7etch6-log).

Annoyingly I can't reproduce this with a trivial script:

use DBI;
use Test::More tests => 2;

my $dbh = DBI->connect( "dbi:mysql:test", 'root' );

sub test_db {
    my ($number) = $dbh->selectrow_array("select 1 ");
    return $number;
}

is test_db, 1, "connected to db";

warn "restart db now";
getc;

is test_db, 1, "connected to db";

Which gives the following:

ok 1 - connected to db
restart db now at dbd-mysql-test.pl line 23.

DBD::mysql::db selectrow_array failed: MySQL server has gone away at dbd-mysql-test.pl line 17.
not ok 2 - connected to db
#   Failed test 'connected to db'
#   at dbd-mysql-test.pl line 26.
#          got: undef
#     expected: '1'

This behaves correctly, telling me why the request failed.

What stumps me is that it is segfaulting, which it shouldn't do. As it only appears to happen when the whole app is running (which uses DBIx::Class) it is hard to reduce it to a test case.

Where should I start to look to debug this? Has anyone else seen this?

UPDATE: further prodding showed that it being under mod_perl was a red herring. Having reduced it to a simple test script I've now posted to the DBI mailing list. Thanks for your answers.

Hux

Answer 1

This is a known problem in old DBD::mysql. Upgrade it (4.008 is not up to date).

There's a simple test script attached to https://rt.cpan.org/Public/Bug/Display.html?id=37027 that will trigger this bug.

Lemniscus

Answer 2

What this probably means is that there's a difference between your mod_perl environment and the one you were testing via your script. Some things to check:

• Was your mod_perl compiled with the same version of Perl

• Are the @INC's the same for both

• Are you using threads in your mod_perl setup? I don't believe DBD::mysql is completely thread-safe.

Scythia

Answer 3

I've seen this problem, but I'm not sure it had the same cause as yours. Are you by chance using a certain module for sending mails (forgot the name, sorry) from your application? When we had the problem in a project, after days of debugging we found that this mail module was doing strange things with open file descriptors, then forked off another process which called the console tool sendmail, which again did strange things with file descriptors. I guess one of the file descriptors it messed around with was the connection to the database, but I'm still not sure about that. The problem disappeared when we switched to another module for sending mails. Maybe it's worth a look for you too.

Elli

no - it appears to be a problem with either DBD::mysql or the mysql client binaries. Thanks though :) – Hux

Answer 4

If you're getting a segfault, do you have a core file greated? If not, check ulimit -c. If that returns 0, your system won't create core files and you'll have to change that. If you do have a core file, you can use gdb or similar tools to debug it. It's not particularly fun, but it's possible. The start of the command will look something like:

gbd /usr/bin/httpd core

There are plenty of tutorials for debugging core files scattered about the Web.

Update: Just found a reference for ensuring you get core dumps from mod_perl. That should help.

Unmeant

Thanks for the pointers - I ended up reducing it to a small test script and posting to the DBI mailing list (see question for link). – Hux