Phusion Passenger memory consumption increase from 1.9.3 (system) to 2.1.2 (RVM) on Ubuntu

Question

I have an Ubuntu system with 512MB physical RAM and 512MB swap. Running my four Rails 3.2.12 apps on Passenger 4.0.45 server (Apache module) using single-user (my user) RVM Ruby 2.1.2, it uses about 30% more memory than when I configure Passenger to use the older system (root) Ruby 1.9.3. That is the only difference in the Passenger config.

This causes Passenger to move other applications to swap when one is accessed, which means each time a different application is accessed, it has to be loaded in from swap, resulting in a delay. This means that I almost run out of memory if multiple apps are being used concurrently, and they react slowly to users.

I can't understand why using RVM causes Passenger to use more memory. I doubt it is because of Ruby 2.1.2 unless there are memory parameters I need to tune.

Marmoset

Answer 1

The out of the box settings for ruby 2.1.2 do increase memory use compared to 2.0.0 (which I believe was on par with 1.9.3 if you don't consider the copy on write improvements in 2.0.0)

The reason is that Ruby 2.1 introduced a new garbage collection algorithm. In a nutshell the algorithm assumes that while some objects live for a long time (eg the objects representing your code) others are very short lived. Older rubies would spend a long time trying to see whether all objects were ready to be garbage collected, whereas ruby 2.1 switches between minor collections (only try to collect the short lived objects) and major collections (try to collect the long lived ones).

This increases performance (minor collections are much faster) at the expense of some memory usage.

You can tune this, using (among others) the RUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR environment variable. The default is 2, a setting of 0.9 turns off the the generational garbage collector and numbers in between will trade off memory for performance.

You can also use the jemalloc library (on any ruby version) to gain a little performance and reduce memory usage slightly.

Lastly part of the problem with rails apps is the many apps have what one might call medium lived objects that last for a whole request - Ruby's attempt to split objects into just 2 generations isn't quite sufficient. Ruby 2.2 is slated to improve on this.

Sam Saffron has a great post on this if you want to read more

In addition ruby 2.1.3 made some changes to gc timing that reduces memory uses in most cases compared to 2.1.2

Graphophone

Thank you. I can confirm that setting RUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR=0.9 removes this memory "bloat", which ironically results in higher performance given my limited memory (I have to limit Passenger to 1 process per application). Any idea how to inject jemalloc into my Apache/Passenger/Ruby process? I currently start the service using apachectl. – Marmoset

Actually, it does not remove it entirely but returns memory consumption to slightly above 1.9.3 and remains more stable when the different apps are accessed. This is a difference of about 200MB, about a quarter of my total virtual memory. It seems like Passenger aggressively swaps inactive application memory to the page file the fuller the page file gets. – Marmoset

I use jemalloc by setting LIBS=-ljemalloc when I compile ruby. You should be able to do it with the LD_PRELOAD environment variable but I haven't done that. – Graphophone