Check why ruby script hangs

Question

Sometimes my specs can just hang and I have to kill the corresponding ruby process. It's quite common when I run integration specs written with capybara and webkit driver.

Is it possible to inspect given ruby's process and see where it hangs? Which method, operation, file, line number etc.

Bharat

Can you run from a terminal and kill with control-c? This should give you a ruby trace. – Gittern

Sometimes I can't kill the process in this way. Especially when capybara-webkit hangs. – Bharat

can you attach gdb to it? that might help...another option might be to run it with "-rtracer" and output that to a file, it would show you the last method entered... – Desultory

Answer 1

tl;dr

Use gdb (e.g. Linux):

• echo 'call (void)rb_backtrace()' | gdb -p $(pgrep -f ruby)

or use lldb (e.g. OS X):

• echo 'call (void)rb_backtrace()' | lldb -p $(pgrep -f ruby)

---

You can debug Ruby script by using debug library.

If the script is executed from shell, this can be achieved by changing first line (shebang) of the script into:

#!/usr/bin/env ruby -rdebug

or running it as:

ruby -rdebug my_script.rb

Once the debugger is loaded, you can either set-up some breakpoints or just run the app by typing c to continue the execution.

Then the debugger automatically breaks on any Exceptions (such as Ctrl+C) or breakpoints (e.g. the lines which consist debugger).

Then every time when debugger console is shown, you can choose between:

• c for Continue (to the next Exception, breakpoint or line with: debugger),
• n for Next line,
• w/where to Display frame/call stack,
• l to Show the current code,
• cat to show catchpoints.
• h for more Help.

See also: Debugging with ruby-debug, Key shortcuts for ruby-debug gem.

The downside of this method is that there is no magic button to raise the debugger on demand, other than raising the Exception within the script as well which will show different block of code rather than hung one.

Here are few other ideas:

• Add debugger statement into your code, raise the debugger and go step by step.

• Use Pry debugger instead (see: GitHub).

  Install via: gem install pry, run as: pry or add as require 'pry'.

• Try lldb debugger (which aiming to replace gdb) which can attach to the currently running process.

  Example (replace PID with your process id):

  $ lldb -p PID
  (lldb) bt all
  * thread #1: tid = 0x11d68a, 0x00007fff86c71716 libsystem_kernel.dylib`__psynch_cvwait + 10
    * frame #0: 0x00007fff86c71716 libsystem_kernel.dylib`__psynch_cvwait + 10
      frame #1: 0x00007fff838a9c3b libsystem_pthread.dylib`_pthread_cond_wait + 727
      frame #2: 0x0000000100241aad libruby.2.0.0.dylib`native_cond_wait + 29
  

  Another example showing backtrace of live running ruby script (on its tty):

  echo 'call (void)rb_backtrace()' | lldb -p $(pgrep -f ruby)
  

• Alternatively use gdb (you can extend it by: gdb.rb which can show you ruby objects).

  1. Install via: sudo apt-get install gdb python-dev ncurses-dev && gem install gdb.rb
  2. On Unix/OS X press Ctrl+T on hanging process to check the PID and what's doing (or check via ps wuax | grep ruby).
  3. Attach to the process via: gdb -p PID.

  See also: using gdb to inspect a hung ruby process, GDB wrapper for Ruby and Inspecting a live Ruby process.

• Other libraries/tools which can help include: debugger, crash-watch, memprof, rack-perftools_profiler.

If nothing helps, you can simply try: strace (Linux) /dtruss (OS X) with the following syntax:

sudo strace -fp <PID>
sudo dtruss -fp <PID>

Or ltrace which can trace library calls as opposed to strace system calls.

If you think it's a network issue, use tcpdump.

See also:

• How do I debug Ruby scripts?
• How do I configure ruby to enter the debugger on Ctrl-C (SIGINT)?
• How do I use the keyboard to break into ruby-debug running a rails app?
• Is there a way to use a keystroke to invoke the pry ruby gem?
• Debugging Stuck Ruby Processes – What to do Before You Kill -9
• Tools for Debugging Running Ruby Processes and Debugging Ruby Tools
• Debugging Rails Applications
• The right way to deal with frozen processes on Unix
• Debugging Ruby-applications in Unix by Sergii Boiko
• Debugging using Ruby Debugger at JumpstartLab
• ruby-debug in 30 seconds (we don't need no stinkin' GUI!)
• How to debug stuck Ruby processes

Mays

This is a great answer. This should be accepted. Kudos and +1 for the exhaustive evaluation of different techniques. – Revolutionist

Answer 2

I had this problem as well and traced it to a ShareThis javascript widget on a particular page. You may or may not be using this, but the real problem is perhaps its hanging because something on the page is causing an external request that never finishes. Capybara-webkit will know about the original request, but if this code itself makes a request, capybara-webkit will never know about it, and if that last request hangs, say waiting for a response, so will capybara-webkit...

For you, run the test using webkit-debug and look at the very last request that is made. For me, i saw the following:

    1 requests remaining 
    Page finished with true 
    Received 200 from "http://w.sharethis.com/share4x/js/st.60709d5fdf0c137e879e64f41b8a6606.js" 
    0 requests remaining 
    Started request to "http://w.sharethis.com/share4x/css/share.470030190b6a6bdc89365fcc74d3bf55.css" 
    Received 200 from "http://w.sharethis.com/share4x/css/share.470030190b6a6bdc89365fcc74d3bf55.css" 
    0 requests remaining 

And that clued me to search my codebase for ShareThis. I put an if(Rails.env.test?) block around that code, and voila, i'm in business. Its a crappy workaround to have to put conditionals for test environments into your codebase...but it got me moving on from this silly problem...

Hope this helps.

Costrel