Native extensions fallback to pure Ruby if not supported on gem install

Question

I am developing a gem, which is currently pure Ruby, but I have also been developing a faster C variant for one of the features. The feature is usable, but sometimes slow, in pure Ruby. The slowness would only impact some of the potential users (depends which features they need, and how they use them), so it makes sense to have the gem available with graceful fallback to Ruby-only functions if it cannot compile on a target system.

I would like to maintain the Ruby and C variants of the feature in a single gem, and provide the best (i.e. fastest) experience from the gem on installation. That would allow me to support the widest set of potential users from a single project of mine. It would also allow other people's dependent gems and projects to use the best available dependency on a target system, as opposed to a lowest-common-denominator version for compatibility.

I would expect the require to fallback at runtime to appear in the main lib/foo.rb file simply like this:

begin
  require 'foo/foo_extended'
rescue LoadError
  require 'foo/ext_bits_as_pure_ruby'
end

However, I don't know how to get the gem installation to check (or try and fail) for native extension support so that the gem installs correctly whether or not it can build 'foo_extended'. When I researched how to do this, I mainly found discussions from a few years back e.g. http://permalink.gmane.org/gmane.comp.lang.ruby.gems.devel/1479 and http://rubyforge.org/pipermail/rubygems-developers/2007-November/003220.html that imply Ruby gems do not really support this feature. Nothing recent though, so I am hoping someone on SO has some more up-to-date knowledge?

My ideal solution would be a way to detect, prior to attempting a build of the extension, that the target Ruby did not support (or perhaps simply not want, at the project level) C native extensions. But also, a try/catch mechanism would be OK if not too dirty.

Is this possible, if so how? Or is the advice to have two gem variants published (e.g. foo and foo_ruby), that I am finding when I search, still current best practice?

Nucleolated

Two gems are fine, e.g. the json gem comes in two variants: json (with C extension) and json_pure (pure Ruby). – Wellturned

@Stefan: In the conversation I linked, the author/maintaner of json and json_pure would apparently prefer it otherwise. As well as the extra work publishing two variants of the gem, dependent projects that may themselves be dependencies for something else, end up having to use lowest-common-denominator or must also provide two variants just to manage dependencies that they don't code. I would not call that "fine", but if it is the best possible, then that's all I can do too – Nucleolated

@Stefan: Definitely Neil's design intention embodied in the OP is right. His question is great and very important, I'm interested in the answer myself, please upvote it. – Asmara

I'm interested in this too, the question is really important. – Acclimatize

Answer 1

This is my best result attempting to answer my own question to date. It appears to work for JRuby (tested in Travis and on my local installation under RVM), which was my main goal. However, I would be very interested in confirmations of it working in other environments, and for any input on how to make it more generic and/or robust:

---

The gem installation code expects a Makefile as output from extconf.rb, but has no opinion on what that should contain. Therefore extconf.rb can decide to create a do nothing Makefile, instead of calling create_makefile from mkmf. In practice that might look like this:

ext/foo/extconf.rb

can_compile_extensions = false
want_extensions = true

begin
  require 'mkmf'
  can_compile_extensions = true
rescue Exception
  # This will appear only in verbose mode.
  $stderr.puts "Could not require 'mkmf'. Not fatal, the extensions are optional."
end


if can_compile_extensions && want_extensions
  create_makefile( 'foo/foo' )

else
  # Create a dummy Makefile, to satisfy Gem::Installer#install
  mfile = open("Makefile", "wb")
  mfile.puts '.PHONY: install'
  mfile.puts 'install:'
  mfile.puts "\t" + '@echo "Extensions not installed, falling back to pure Ruby version."'
  mfile.close

end

As suggested in the question, this answer also requires the following logic to load the Ruby fallback code in the main library:

lib/foo.rb (excerpt)

begin
  # Extension target, might not exist on some installations
  require 'foo/foo'
rescue LoadError
  # Pure Ruby fallback, should cover all methods that are otherwise in extension
  require 'foo/foo_pure_ruby'
end

Following this route also requires some juggling of rake tasks, so that the default rake task doesn't attempt to compile on Rubies that we're testing on that don't have capability to compile extensions:

Rakefile (excerpts)

def can_compile_extensions
  return false if RUBY_DESCRIPTION =~ /jruby/
  return true
end 

if can_compile_extensions
  task :default => [:compile, :test]
else
  task :default => [:test]
end

Note the Rakefile part doesn't have to be completely generic, it just has to cover known environments we want to locally build and test the gem on (e.g. all the Travis targets).

I have noticed one annoyance. That is by default you will see Ruby Gems' message Building native extensions. This could take a while..., and no indication that the extension compilation was skipped. However, if you invoke the installer with gem install foo --verbose you do see the messages added to extconf.rb, so it's not too bad.

Nucleolated

Would be cool if you could put a compilation process (that uses Makefile) into begin rescue end block. In my case I don't want to make an extension to be a separate gem and the gem should fallback to use the Ruby implementation if compilation for some reason has failed, assuming that we can't check for sure before running the Makefile that it will success. – Anomalistic

Answer 2

https://stackoverflow.com/posts/50886432/edit

I tried the other answers and could not get them to build on recent Rubies.

This worked for me:

1. Use mkmf#have_* methods in extconf.rb to check for everything you need. Then call #create_makefile, no matter what.
2. Use the preprocessor constants generated by #have_* to skip things in your C file.
3. Check which methods/modules are defined in Ruby.
4. If you want to support JRuby et al, you'll need a more complex release setup.

---

A simple example where the whole C extension is skipped if something is missing:

1. ext/my_gem/extconf.rb

require 'mkmf'

have_struct_member('struct foo', 'bar')

create_makefile('my_gem/my_gem')

2. ext/my_gem/my_gem.c

#ifndef HAVE_STRUCT_FOO_BAR
  // C ext cant be compiled, ignore because it's optional
  void Init_my_gem() {}
#else
  #include "ruby.h"

  void Init_my_gem() {
    VALUE mod;
    mod = rb_define_module("MyGemExt");
    // attach methods to module
  }
#endif

3. lib/my_gem.rb

class MyGem
  begin
    require 'my_gem/my_gem'
    include MyGemExt
  rescue LoadError, NameError
    warn 'Running MyGem without C extension, using slower Ruby fallback'
    include MyGem::RubyFallback
  end
end

4. If you want to release the gem for JRuby, you need to adapt the gemspec before packaging. This will allow you to build and release multiple versions of the gem. The simplest solution I can think of:

Rakefile

require 'rubygems/package_task'

namespace :java do
  java_gemspec = eval File.read('./my_gem.gemspec')
  java_gemspec.platform = 'java'
  java_gemspec.extensions = [] # override to remove C extension

  Gem::PackageTask.new(java_gemspec) do |pkg|
    pkg.need_zip = true
    pkg.need_tar = true
    pkg.package_dir = 'pkg'
  end
end

task package: 'java:gem'

Then run $ rake package && gem push pkg/my_gem-0.1.0 && gem push pkg/my_gem-0.1.0-java to release a new version.

If you just want to run on JRuby, not distribute the gem for it, this will suffice (it will not work for releasing the gem, though, as it is evaluated before packaging):

my_gem.gemspec

if RUBY_PLATFORM !~ /java/i
  s.extensions = %w[ext/my_gem/extconf.rb]
end

---

This approach has two advantages:

• create_makefile should work in every environment
• a compile task can remain prepended to other tasks (except on JRuby)

Leomaleon

Answer 3

Here is a thought, based on info from http://guides.rubygems.org/c-extensions/ and http://yorickpeterse.com/articles/hacking-extconf-rb/.

Looks like you can put the logic in extconf.rb. For example, query the RUBY_DESCRIPTION constant and determine if you are in a Ruby that supports native extensions:

$ irb
jruby-1.6.8 :001 > RUBY_DESCRIPTION
=> "jruby 1.6.8 (ruby-1.8.7-p357) (2012-09-18 1772b40) (Java HotSpot(TM) 64-Bit Server VM       
    1.6.0_51) [darwin-x86_64-java]"

So you could try something like wrap the code in extconf.rb in a conditional (in extconf.rb):

unless RUBY_DESCRIPTION =~ /jruby/ do

  require 'mkmf'

  # stuff    
  create_makefile('my_extension/my_extension')

end

Obviously, you will want more sophisticated logic, grabbing parameters passed on "gem install", etc.

Synthesis

That code in extconf.rb did not work. But similar code in the gemspec around the dependency on rake-install and declaring the extension appears to work as required. I need to play with it a little more, but it's promising – Nucleolated

Editing gemspec allowed me to test the gem on JRuby again, and I had all my rake tasks working in all test rubies, but ultimately it didn't work either. The problem is that the .gemspec is processed too soon, so only works when the build of the gem is done in the same Ruby target. However extconf.rb is processed too late, if the gem contains extconf.rb the target system can reject it before any logic in it runs. – Nucleolated

Thanks for the pointers, they have led to what I think is a viable answer. However, there is more to it than detecting targets that do/don't compile, so have added my findings as a new answer. – Nucleolated