unable to create a C++ ruby extension

Question

I have problems creating a ruby extension to export a C++ library I wrote to ruby under OSX. This simple example:

#include <boost/regex.hpp>

extern "C" void Init_bayeux()
{
    boost::regex expression("^(([^:/?#]+):)?(//([^/?#]*))?([^?#]*)(\\?([^#]*))?(#(.*))?");
}

results in a bad_cast exception being thrown:

#0  0x00000001014663bd in __cxa_throw ()
#1  0x00000001014cf6b2 in __cxa_bad_cast ()
#2  0x00000001014986f9 in std::use_facet<std::collate<char> > ()
#3  0x0000000101135a4f in boost::re_detail::cpp_regex_traits_base<char>::imbue (this=0x7fff5fbfe4d0, l=@0x7fff5fbfe520) at cpp_regex_traits.hpp:218
#4  0x0000000101138d42 in cpp_regex_traits_base (this=0x7fff5fbfe4d0, l=@0x7fff5fbfe520) at cpp_regex_traits.hpp:173
#5  0x000000010113eda6 in boost::re_detail::create_cpp_regex_traits<char> (l=@0x7fff5fbfe520) at cpp_regex_traits.hpp:859
#6  0x0000000101149bee in cpp_regex_traits (this=0x101600200) at cpp_regex_traits.hpp:880
#7  0x0000000101142758 in regex_traits (this=0x101600200) at regex_traits.hpp:75
#8  0x000000010113d68c in regex_traits_wrapper (this=0x101600200) at regex_traits.hpp:169
#9  0x000000010113bae1 in regex_data (this=0x101600060) at basic_regex.hpp:166
#10 0x000000010113981e in basic_regex_implementation (this=0x101600060) at basic_regex.hpp:202
#11 0x0000000101136e1a in boost::basic_regex<char, boost::regex_traits<char, boost::cpp_regex_traits<char> > >::do_assign (this=0x7fff5fbfe710, p1=0x100540ae0 "^(([^:/?#]+):)?(//([^/?#]*))?([^?#]*)(\\?([^#]*))?(#(.*))?", p2=0x100540b19 "", f=0) at basic_regex.hpp:652
#12 0x0000000100540a66 in boost::basic_regex<char, boost::regex_traits<char, boost::cpp_regex_traits<char> > >::assign (this=0x7fff5fbfe710, p1=0x100540ae0 "^(([^:/?#]+):)?(//([^/?#]*))?([^?#]*)(\\?([^#]*))?(#(.*))?", p2=0x100540b19 "", f=0) at basic_regex.hpp:379
#13 0x0000000100540a13 in boost::basic_regex<char, boost::regex_traits<char, boost::cpp_regex_traits<char> > >::assign (this=0x7fff5fbfe710, p=0x100540ae0 "^(([^:/?#]+):)?(//([^/?#]*))?([^?#]*)(\\?([^#]*))?(#(.*))?", f=0) at basic_regex.hpp:364
#14 0x000000010054096e in basic_regex (this=0x7fff5fbfe710, p=0x100540ae0 "^(([^:/?#]+):)?(//([^/?#]*))?([^?#]*)(\\?([^#]*))?(#(.*))?", f=0) at basic_regex.hpp:333
#15 0x00000001005407e2 in Init_bayeux () at bayeux.cpp:10
#16 0x0000000100004593 in dln_load (file=0x1008bc000 "/Users/todi/sioux/lib/debug/rack/bayeux.bundle") at dln.c:1293

I compile the extension with:

g++ ./source/rack/bayeux.cpp -o /Users/todi/sioux/obj/debug/rack/bayeux.o -Wall -pedantic -Wno-parentheses -Wno-sign-compare -fno-common -c -pipe -I/Users/todi/sioux/source -ggdb -O0

And finally link the dynamic library with:

g++ -o /Users/todi/sioux/lib/debug/rack/bayeux.bundle -bundle -ggdb  /Users/todi/sioux/obj/debug/rack/bayeux.o -L/Users/todi/sioux/lib/debug -lrack -lboost_regex-mt-d -lruby

I've searched the web and tried all kind of link and compiler switches. If I build a executable there is no such problem. Does someone else had such a problem and found a solution?

I've further investigated this and found that the function causing the exception looks like this:

std::locale loc = std::locale("C");
std::use_facet< std::collate<char> >( loc );

In the source of std::collate<> I found the throw statment:

use_facet(const locale& __loc)
{
  const size_t __i = _Facet::id._M_id();
  const locale::facet** __facets = __loc._M_impl->_M_facets;
  if (__i >= __loc._M_impl->_M_facets_size || !__facets[__i])
    __throw_bad_cast();
#ifdef __GXX_RTTI
      return dynamic_cast<const _Facet&>(*__facets[__i]);
#else
      return static_cast<const _Facet&>(*__facets[__i]);
#endif
}

Does this makes any sense to you?

Update: I've tried Jan's suggestion:

Todis-MacBook-Pro:rack todi$ g++ -shared -fpic -o bayeux.bundle bayeux.cpp
Todis-MacBook-Pro:rack todi$ ruby -I. -rbayeux -e 'puts :ok'
terminate called after throwing an instance of 'std::bad_cast'
  what():  std::bad_cast
Abort trap

versions:

Todis-MacBook-Pro:rack todi$ ruby -v
ruby 1.9.2p136 (2010-12-25 revision 30365) [x86_64-darwin10.6.0]
Todis-MacBook-Pro:rack todi$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/opt/local/libexec/gcc/x86_64-apple-darwin10/4.5.2/lto-wrapper
Target: x86_64-apple-darwin10
Configured with: ../gcc-4.5.2/configure --prefix=/opt/local --build=x86_64-apple-darwin10 --enable-languages=c,c++,objc,obj-c++,fortran,java --libdir=/opt/local/lib/gcc45 --includedir=/opt/local/include/gcc45 --infodir=/opt/local/share/info --mandir=/opt/local/share/man --datarootdir=/opt/local/share/gcc-4.5 --with-local-prefix=/opt/local --with-system-zlib --disable-nls --program-suffix=-mp-4.5 --with-gxx-include-dir=/opt/local/include/gcc45/c++/ --with-gmp=/opt/local --with-mpfr=/opt/local --with-mpc=/opt/local --enable-stage1-checking --disable-multilib --enable-fully-dynamic-string
Thread model: posix
gcc version 4.5.2 (GCC) 

Update:

It's not the bound-check in use_facet() that throws, but the next line, that actually does a dynamic cast. This example boils it down to maybe something with RTTI:

#define private public
#include <locale>
#include <iostream>
#include <typeinfo>

extern "C" void Init_bayeux()
{
    std::locale loc = std::locale("C");
    printf( "size: %i\n", loc._M_impl->_M_facets_size );
    printf( "id: %i\n", std::collate< char >::id._M_id() );

    const std::locale::facet& fac = *loc._M_impl->_M_facets[ std::collate< char >::id._M_id() ];

    printf( "name: %s\n", typeid( fac ).name());
    printf( "name: %s\n", typeid( std::collate<char> ).name());

    const std::type_info& a = typeid( fac );
    const std::type_info& b = typeid( std::collate<char> );

    printf( "equal: %i\n", !a.before( b ) && !b.before( a ) );
    dynamic_cast< const std::collate< char >& >( fac );
}

I've used printf() because usage of cout also fails. The output of the code above is:

size: 28
id: 5
name: St7collateIcE
name: St7collateIcE
equal: 1
terminate called after throwing an instance of 'std::bad_cast'
  what():  std::bad_cast
Abort trap

Build with:

g++ -shared -fpic -o bayeux.bundle bayeux.cpp && ruby -I. -rbayeux -e 'puts :ok'

Update:

If I rename Init_bayeux to main() and link it to an executable, the output is the same, but no call to terminate.

Update:

When I write a little program to load the shared library and to execute Init_bayeux(), again, no exception is thrown:

#include <dlfcn.h>

int main()
{
    void* handle = dlopen("bayeux.bundle", RTLD_LAZY|RTLD_GLOBAL );
    void(*f)(void) = (void(*)(void)) dlsym( handle, "Init_bayeux" ) ;
    f();
}

So it looks to me, that it might be a problem with how the ruby.exe was build. Does that make sense?

Update: I had a look at the addresses containing the names of the two type_info objects. Same content, but different addresses. I added the -flat_namespace switch to the link command. Now the dynamic_cast works. The original Problem with the boost regex library still exists, but I think this might be solvable by linking boost statically into the shared library or by rebuilding the boost libraries with the -flat_namespace switch.

Update: Now I'm back to the very first example with the boost regex expression, build with this command:

g++ -shared -flat_namespace -fPIC -o bayeux.bundle /Users/todi/boost_1_49_0/stage/lib/libboost_regex.a bayeux.cpp

But when loading the extension into the ruby interpreter, initializing of static symbols fails:

ruby(59384,0x7fff712b8cc0) malloc: *** error for object 0x7fff70b19500: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug

Program received signal SIGABRT, Aborted.
0x00007fff8a6ab0b6 in __kill ()
(gdb) bt
#0  0x00007fff8a6ab0b6 in __kill ()
#1  0x00007fff8a74b9f6 in abort ()
#2  0x00007fff8a663195 in free ()
#3  0x0000000100541023 in boost::re_detail::cpp_regex_traits_char_layer<char>::init (this=0x10060be50) at basic_string.h:237
#4  0x0000000100543904 in boost::object_cache<boost::re_detail::cpp_regex_traits_base<char>, boost::re_detail::cpp_regex_traits_implementation<char> >::do_get (k=@0x7fff5fbfddd0) at cpp_regex_traits.hpp:366
#5  0x000000010056005b in create_cpp_regex_traits<char> (l=<value temporarily unavailable, due to optimizations>) at pending/object_cache.hpp:69
#6  0x0000000100544c33 in boost::basic_regex<char, boost::regex_traits<char, boost::cpp_regex_traits<char> > >::do_assign (this=0x7fff5fbfe090, p1=0x100567158 "^(([^:/?#]+):)?(//([^/?#]*))?([^?#]*)(\\?([^#]*))?(#(.*))?", p2=0x100567191 "", f=0) at cpp_regex_traits.hpp:880
#7  0x0000000100566280 in boost::basic_regex<char, boost::regex_traits<char, boost::cpp_regex_traits<char> > >::assign ()
#8  0x000000010056622d in boost::basic_regex<char, boost::regex_traits<char, boost::cpp_regex_traits<char> > >::assign ()
#9  0x0000000100566188 in boost::basic_regex<char, boost::regex_traits<char, boost::cpp_regex_traits<char> > >::basic_regex ()
#10 0x0000000100566025 in Init_bayeux ()
#11 0x0000000100003a23 in dln_load (file=0x10201a000 "/Users/todi/sioux/source/rack/bayeux.bundle") at dln.c:1293
#12 0x000000010016569d in vm_pop_frame [inlined] () at /Users/todi/.rvm/src/ruby-1.9.2-p320/vm_insnhelper.c:1465
#13 0x000000010016569d in rb_vm_call_cfunc (recv=4303980440, func=0x100042520 <load_ext>, arg=4303803000, blockptr=0x1, filename=<value temporarily unavailable, due to optimizations>, filepath=<value temporarily unavailable, due to optimizations>) at vm.c:1467
#14 0x0000000100043382 in rb_require_safe (fname=4303904640, safe=0) at load.c:602
#15 0x000000010017cbf3 in vm_call_cfunc [inlined] () at /Users/todi/.rvm/src/ruby-1.9.2-p320/vm_insnhelper.c:402
#16 0x000000010017cbf3 in vm_call_method (th=0x1003016b0, cfp=0x1004ffef8, num=1, blockptr=0x1, flag=8, id=<value temporarily unavailable, due to optimizations>, me=0x10182cfa0, recv=4303980440) at vm_insnhelper.c:528
...

Again, this doesn't fail, when I load the shared library by the little c program from above.

Update: Now I link the first example static:

g++ -shared -fPIC -flat_namespace -nodefaultlibs -o bayeux.bundle -static -lstdc++ -lpthread -lgcc_eh -lboost_regex-mt bayeux.cpp

With the same error:

ruby(15197,0x7fff708aecc0) malloc: *** error for object 0x7fff7027e500: pointer being freed was not allocated

otool -L confirmed that every library is linked static:

bayeux.bundle:
bayeux.bundle (compatibility version 0.0.0, current version 0.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.2.11)

debug:

If I link against the boost debug version, then it works like expected.

Hyperboloid

I cannot reproduce the behaviour you described. I compiled the example C++ file with g++ -shared -fpic -o bayeux.so -lboost_regex bayeux.cpp and ruby -I. -rbayeux -e 'puts :ok' loaded the extension without any issues. However, I had some problems when I omitted -lboost_regex. Besides, could you specify which versions of Ruby and Boost you're using and how can we reproduce this problem? – Millman

@Jan, Thanks for having a try. On OS X, I have to change the 'so' extension to 'bundle' (otherwise the require will not find the library) but the result is still the same. – Hyperboloid

@Jan, you can omit the boost library, it fails without it. It's the call to use_facet<> that fails. – Hyperboloid

What with the extern "C" modifier ? It seems awkward to build C++ code within a C section. – Subtonic

@Plexico: The extern "C" qualifier is needed to prevent C++ name mangling. The ruby Interpreter is looking for a function called Init_bayeux if the name of the dynamic library is bayeux. – Hyperboloid

Have you tried to statically link your lib with the crt ? – Inerney

@Inerney this will be the first for me to do tomorrow morning. Thanks. – Hyperboloid

This is so annoying that I'm now up to drop the ruby support at all. I've spend over a week now and not even a dirty workaround in sign. – Hyperboloid

You may examine rice. Ruby Interface for C++ Extensions. – Darren

@Selman I've tried mkmf already, with the same result. I think it might have something to do with the installation of the compiler (mac port), as I'm the only one with such a problem. – Hyperboloid

This is a bit of a stab in the dark, but I've had similarly inexplicable problems that turned out to be a mismatch between the LLVM ABI and the g++ ABI. Was ruby compiled with Apple's compilers (llvm-gcc or clang)? Have you tried compiling your extension with either Apple's g++ (it uses an LLVM backend) or clang++? – Onassis

@Onassis I've used RVM to install ruby and Mac Port to install the gcc compiler. I will try to figure out, what compiler RVM uses by default. Thank you. – Hyperboloid

You may want to just quickly try compiling with clang++ to see if the problem goes away. clang++ comes with XCode 4.x, so it's likely that you already have it (or can install it in the Downloads section of XCode's preferences). Depending on how you installed boost, you may have to reinstall it using either Apple's g++ (/usr/bin/g++) or clang++ for this to work. – Onassis

I've tried to reinstall ruby with the gcc version I'm using. This failed. After reinstalling ruby with RVM, I had a look at the build logs and they confirmed that the compiler used to build ruby was /usr/bin/gcc-4.2. reinstalling boost with a different compiler will take some time. – Hyperboloid

@Onassis that was the problem! Yeeeepeeeee!!! :-) 2 Weeks of stabbing in the dark. Thank you all very much for the help. I owe you a beer! – Hyperboloid

Why aren't you using extconf.rb? It generates a makefile with everything needed. – Rinarinaldi

@Rinarinaldi I'm using rake as build system and mkmf don't let me build easily 3 different extensions (debug, release, coverage). The makefile wasn't actually the problem, the problem occurred even when using mkmf. – Hyperboloid

Answer 1

For the records: I've now build boost and my application with the very same compiler (version 4.2.1 [official apple version]). No problems so far. Why it will not work as expected when the ruby extension links all libraries statically is a miracle to me. Thank to all who put time into this issue.

Kind regards Torsten

Hyperboloid