Can I incorporate system libraries (e.g. libxml2) I compile against into a gem (e.g. nokogiri) that I can deploy to Heroku?

Question

Nokogiri has a problem with translating to and from UTF-8 characters that turns out to come from libxml2, specifically version 2.7.6, which is the highest supported version on Ubuntu 10.04 LTS. The bug is fixed in version 2.7.7 and up, but since our app is hosted on Heroku (bamboo-ree-1.8.7 stack, based on Ubuntu 10.04), we have to use version 2.7.6, and continue to experience the bug, unless:

1. Someone can/has hacked nokogiri to get around the problem
2. Canonical bumps the supported libxml2 version for Ubuntu 10.04 (and/or Heroku updates libxml2 in their stack)
3. I can come up with a way for nokogiri to use a version of libxml2 which I can bundle with the app in such a way that it can be deployed to Heroku.

Happy to hear any feedback on 1 or 2 of course, but curious whether 3 is feasible. Here is what I know to be possible:

• Nokogiri can be linked against a library located in a non-standard place - http://nokogiri.org/tutorials/installing_nokogiri.html
• The gem created this way can be installed to vendor/gems, and thus take precedence over system gems

My question is, can I include the higher versioned libxml2 with the app in such a way that the compiled, vendor'ed gem uses it when I have committed it and pushed it to Heroku?

Sikes

Ended up being able to use the workaround suggested here - github.com/tenderlove/nokogiri/issues/458#issuecomment-3136620. – Sikes

Ben, what was the workaround. That link is broken now. – Velum

@NateCook the link is now github.com/sparklemotion/nokogiri/issues/… – Diffident

Answer 1

It's not the most user friendly solution, but here's what I did once to get a custom version of libpq. I did it on cedar, but it'll probably work for bamboo if you go into tmp first

1. heroku run bash
2. curl the source of what you want to build
3. build it
4. send the binaries out of the dyno
5. vendor the built binaries in vendor/whatever
6. heroku config:add LD_LIBRARY_PATH=/app/vendor/whatever

Sorry there isn't a better way to get custom binaires yet. Good luck.

Pyrites

Thanks Will, tried this and got all the steps to work, but it still didn't seem to use my library in preference to the system one. Put my library in /app/vendor/lib, set LD_LIBRARY_PATH to that, also tried setting LIBRARY_PATH, checked via bash that they were correctly set and that the libraries were present. Still no dice. Ended up using a workaround, but curious whether there was anything else you can think of that might have made this work for you. – Sikes

Answer 2

I'm not sure of the exact answer but I have found an issue raised for Nokogiri which sounds like the issue you've just described: https://github.com/tenderlove/nokogiri/issues/458

One of the responses talks about how they are hosted on Heroku and are locked to version 2.7.6 of libxml2 and have pleaded their case to upgrade to 2.7.8 to fix their issue: https://github.com/tenderlove/nokogiri/issues/458#issuecomment-2600583

Although unlikely, it might be worth contacting Heroku to see what they can do (if anything), or if they possibly might have any suggestions? I would hazard a guess they won't change the libxml2 version as it's on a locked down stack (Bamboo) but who knows?

Clubby

Answer 3

I have a slightly better solution. The more I play with bower the more I love it. Bower is simply a tool built on top of git and structures things in a fancy manner (usually targeted at frontend JS scripts / not a requirement). One other note as well, bower is great for installing private dependencies as well off github.

The example is python, but the overall goal is the same and can be adjusted for ruby.

So take this project for an example 3scale_python.

It requires libxml2 to be installed (via its requirements.txt). The easiest way for me was to include it in bower and then write a script to install it (shell script and procfile).

bower.json

{ "name": "someapp", "version": "0.0.1", "homepage": "http://github.com/yourusername/yourrepo", "authors": [nem], "description": "something that uses 3scale at a specific commit", "main": "./lib/clock.py", "private": true, "dependencies": { "3scale_ws_api_for_python": "http://github.com/3scale/3scale_ws_api_for_python/archive/82328aa8e7d43f7ef89e420921a4d63e025b527f.zip" } }

install script (manual_installs_python)

#!/bin/bash
set -e

oldPath=$(pwd)

installing(){
  echo "-- installing $1 dependencies --"
  echo '--------------------------------'
  echo
}

#check for brew, if brew then use it to install something
osx_brew() {
    #if brew is installed (no error)
    if hash brew 2 > /dev/null; then
      installing 'brew'
      brew "$@"
    fi
}

installing 'python easy_install'
easy_install figleaf

cd ./bower_components/some_other_python_app
python setup.py install

if [[ $IS_HEROKU = 1 ]]; then
  #installing on ubuntu/heroku only as, so far I have not gotten libxml2-python to install successfully on OSX
  cd ../3scale_ws_api_for_python
  #requirements.txt here actually installs and compiles libxml2
  pip install -r requirements.txt
  python setup.py install
fi

cd "$oldPath"

echo "-- done with dependencies -- "
exit 0

Procfile

web: pip install -r requirements.txt && ./scripts/manual_python_installs

Done

Too install libxml2 your might want to consult the scripts 3scale fires off via its requirements.txt.

Tangelatangelo

btw you can probably do this via npm as well – Tangelatangelo