Setting memory consumption limits with Upstart

Question

I've recently become quite fond of Upstart. Previously I've been using God, Monit and Bluepill but I don't really like these solutions so I'm giving Upstart a try.

I've been using the Foreman gem to generate some basic Upstart configuration files for my processes in /etc/init. However, these generated files only handle the respawning of a crashed process. I was wondering whether it's possible to tell Upstart to restart a process that's consuming for example > 150mb of memory, as you would with Monit, God or Bluepill.

I read through the Upstart docs and this looks like the thing I'm looking for. Though I have no clue how to config something like this.

What I basically want is quite simple. I want to restart my web process if the memory usage is > 150mb ram. These are the files I have:

|-- myapp-web-1.conf
|-- myapp-web-2.conf
|-- myapp-web-3.conf
|-- myapp-web.conf
|-- myapp.conf

And their contents are:

myapp.conf

pre-start script

bash << "EOF"
  mkdir -p /var/log/myapp
  chown -R deployer /var/log/myapp
EOF

end script

myapp-web.conf

start on starting myapp
stop on stopping myapp

myapp-web-1.conf / myapp-web-2.conf / myapp-web-3.conf

start on starting myapp-web
stop on stopping myapp-web
respawn

exec su - deployer -c 'cd /var/applications/releases/20110607140607; cd myapp && bundle exec unicorn -p $PORT >> /var/log/myapp/web-1.log 2>&1'

Any help much appreciated!

Disparity

Answer 1

Appending this to the end of myapp-web-*.conf will cause any allocation calls trying to allocate more than 150mb of memory to return ENOMEM:

limit rss 157286400 157286400

The process might crash at this point, or it might not. That's up to the process!

Here's a test for this in the Upstart Source.

Phonometer

Hey, thanks for your answer. I'm not too familiar with all of this but if I understand correctly, my (Ruby) application will try to allocate memory after exceeding 150mb ram, at which point (when defining: limit rss 157286400 157286400) it would raise an exception (in Ruby's case after doing some research, I think this exception would be raised: Errno::ENOMEM) and would crash the process, usually. Probably not in the Rails framework though. I'll have to look in to that. Thanks! I think I'll be able to figure out the rest thanks to your answer! – Disparity

Did you find out how this would work with ruby/rails? I'm very interested in trying this out for our service arch. – Natica

@Phonometer any progress with the rails app? – Ramsay

I recently tried to implement this and discovered that the rss limit was not having any effect. From man 2 getrlimit: "RLIMIT_RSS Specifies the limit (in pages) of the process's resident set (the number of virtual pages resident in RAM). This limit only has effect in Linux 2.4.x, x < 30, and there only affects calls to madvise(2) specifying MADV_WILLNEED." I ended up using limit as <soft> <hard>, which caused my runaway process to receive an ENOMEM. – Orpington

@MichaelHale what does "as" mean, instead of using "rss"? I too am having problems getting it working but can't find any docs on upstart limit stanza terms... – Speedboat

Answer 2

From the Upstart docs, the limits come from the rlimit system call options. (http://upstart.ubuntu.com/cookbook/#limit)

Since Linux 2.4+ setting the rss (Resident Set Size) has no effect.

An alternative already suggested in other answers is as which sets the virtual memory Address Space size limits. This will have a very different effect of setting 'real' memory limits.

limit as <soft limit> <hard limit>

Excerpt from man pages for setrlimit:

RLIMIT_AS The maximum size of the process's virtual memory (address space) in bytes. This limit affects calls to brk(2), mmap(2), and mremap(2), which fail with the error ENOMEM upon exceeding this limit. Also automatic stack expansion will fail (and generate a SIGSEGV that kills the process if no alternate stack has been made available via sigaltstack(2)). Since the value is a long, on machines with a 32-bit long either this limit is at most 2 GiB, or this resource is unlimited.

Flagship