How to call shell commands from Ruby

Question

How do I call shell commands from inside of a Ruby program? How do I then get output from these commands back into Ruby?

Langur

While this question is useful, it's not asked well. Ruby has many ways to call sub-shells that are well documented and easily found by reading the Kernel and Open3 documentation and searching here on SO. – Truncate

Sadly this topic is quite complex. Open3 (docs) is the best choice for most situations, IMO, but on older versions of Ruby, it won't respect a modified PATH (bugs.ruby-lang.org/issues/8004), and depending on how you pass args (specifically, if you use opts hash with non-keywords), it can break. But, if you hit those situations, then you're doing something pretty advanced and you can figure out what to do by reading the implementation of Open3. – Linsang

I'm surprised no one mentioned Shellwords.escape (doc). You don't want to be inserting user-input directly into shell commands - escape it first! See also command injection. – Clapboard

Answer 1

This explanation is based on a commented Ruby script from a friend of mine. If you want to improve the script, feel free to update it at the link.

First, note that when Ruby calls out to a shell, it typically calls /bin/sh, not Bash. Some Bash syntax is not supported by /bin/sh on all systems.

Here are ways to execute a shell script:

cmd = "echo 'hi'" # Sample string that can be used

1. Kernel#` , commonly called backticks – `cmd`

   This is like many other languages, including Bash, PHP, and Perl.

   Returns the result (i.e. standard output) of the shell command.

   Docs: http://ruby-doc.org/core/Kernel.html#method-i-60

   value = `echo 'hi'`
   value = `#{cmd}`
   

2. Built-in syntax, %x( cmd )

   Following the x character is a delimiter, which can be any character. If the delimiter is one of the characters (, [, {, or <, the literal consists of the characters up to the matching closing delimiter, taking account of nested delimiter pairs. For all other delimiters, the literal comprises the characters up to the next occurrence of the delimiter character. String interpolation #{ ... } is allowed.

   Returns the result (i.e. standard output) of the shell command, just like the backticks.

   Docs: https://docs.ruby-lang.org/en/master/syntax/literals_rdoc.html#label-Percent+Strings

   value = %x( echo 'hi' )
   value = %x[ #{cmd} ]
   

3. Kernel#system

   Executes the given command in a subshell.

   Returns true if the command was found and run successfully, false otherwise.

   Docs: http://ruby-doc.org/core/Kernel.html#method-i-system

   wasGood = system( "echo 'hi'" )
   wasGood = system( cmd )
   

4. Kernel#exec

   Replaces the current process by running the given external command.

   Returns none, the current process is replaced and never continues.

   Docs: http://ruby-doc.org/core/Kernel.html#method-i-exec

   exec( "echo 'hi'" )
   exec( cmd ) # Note: this will never be reached because of the line above
   

Here's some extra advice: $?, which is the same as $CHILD_STATUS, accesses the status of the last system executed command if you use the backticks, system() or %x{}. You can then access the exitstatus and pid properties:

$?.exitstatus

For more reading see:

• http://www.elctech.com/blog/i-m-in-ur-commandline-executin-ma-commands
• http://blog.jayfields.com/2006/06/ruby-kernel-system-exec-and-x.html
• http://tech.natemurray.com/2007/03/ruby-shell-commands.html

Twiggy

I need to log the outputs of my executable on production server but found no way. I used puts #{cmd} and logger.info(#{cmd}). Is there any way to log their outputs on production ? – Alcoholize

And IO#popen() and Open3#popen3(). mentalized.net/journal/2010/03/08/… – Bautram

For the sake of completeness (as I first thought this would also be a Ruby command): Rake has sh which does "Run the system command cmd. If multiple arguments are given the command is not run with the shell (same semantics as Kernel::exec and Kernel::system)". – Freemason

Also, this blog post additionally covers spawn and open3. – Freemason

Backticks don't capture STDERR by default. Append ` 2>&1` to command if you want to capture – Jonahjonas

exec is nice because you can see the output on the console. – Szymanowski

If you're going to be shelling out via exec and system and passing along some options I suggest you checkout my library optout. – Kenogenesis

I think this answer would be slightly improved if it said that backticks and %x returned the "output", rather than the "result", of the given command. The latter could be mistaken for exit status. Or is that just me? – Perdu

Here's a (good?) example of how spawn can be used in conjunction with an embedded shell script within a ruby program: github.com/baitisj/spark-ping/blob/master/spark-ping Look at lines 76 - 94. – Trifoliate

Kernel#system also has a no-shell-involved mode (as of Ruby 1.9). It's in the documentation you linked to. AFAIK only Kernel#system and popen variants have that feature. It's highly desirable for security (no need to escape shell metacharacters) and efficiency (no extra fork). – Chelonian

@Perdu - I went ahead and made that edit, because it tripped me up too (as well as all the others that upvoted your comment). – Maressa

For some use cases you may want to use optout. It can validate Ruby hashes and transform them into shell-style arguments to pass to system, etc... – Kenogenesis

Something to note about $?. Like some other $ variables, there is one copy of it per thread. So it's ok to rely on it not being overwritten by another thread. – Laaspere

"First, note that when Ruby calls out to a shell, it typically calls /bin/sh, not Bash..." How do we know that? I would assume that ruby is going via exec(2) which, on a script, uses whatever interpreter is set in the #! line, and, only missing this line, defaults to /bin/sh – Gradygrae

Slight variation - What if I need to call a command-line program using ruby's system that ruby can't see? Is there a way to put such a command in whatever path the system command uses? – Ruder

Answer 2

Here's a flowchart based on "When to use each method of launching a subprocess in Ruby". See also, "Trick an application into thinking its stdout is a terminal, not a pipe".

Coquetry

Wow haha. Very useful even though the fact this has to exist is unfortunate – Synonymous

As a side note, I find the spawn() method found in many different places (e.g. Kernel and Process to be most versatile. It’s more or less same with PTY.spawn(), but more generic. – Marotta

IMO, backticks and %x are only good for scripting, because they don't return stderr to you, and you can't pass them arguments (note that you can use $? to get the exit status). So I almost always use Open3.capture3 instead of backticks or %x – Linsang

Answer 3

The way I like to do this is using the %x literal, which makes it easy (and readable!) to use quotes in a command, like so:

directorylist = %x[find . -name '*test.rb' | sort]

Which, in this case, will populate file list with all test files under the current directory, which you can process as expected:

directorylist.each do |filename|
  filename.chomp!
  # work with file
end

Pomfrey

Does %x[ cmd ] returns an array to you? – Barber

the above does not work for me. ``<main>': undefined method each' for :String (NoMethodError) how did it work for you? I am using ruby -v ruby 1.9.3p484 (2013-11-22 revision 43786) [i686-linux] Are you sure an array is returned from the command so that the loop will actually work? – Hypogynous

%x[ cmd ].split("\n") will return a list though :) – Statvolt

Answer 4

Here's the best article in my opinion about running shell scripts in Ruby: "6 Ways to Run Shell Commands in Ruby".

If you only need to get the output use backticks.

I needed more advanced stuff like STDOUT and STDERR so I used the Open4 gem. You have all the methods explained there.

Fuentes

The post described here doesn't discuss the %x syntax option. – Backlash

+1 for Open4. I had already started trying to implement my own version of its spawn method when I found this. – Assai

Answer 5

My favourite is Open3

  require "open3"

  Open3.popen3('nroff -man') { |stdin, stdout, stderr| ... }

Dodeca

I also like open3, especially Open3.capture3: ruby-doc.org/stdlib-1.9.3/libdoc/open3/rdoc/… -> stdout, stderr, status = Open3.capture3('nroff -man', :stdin_data => stdin) – Paintbrush

Is there any documentation on how to perform Spec and Unit testing with Open3, or other Open's in the Ruby std-lib? Its hard to test shell outs at my current level of understanding. – Aalii

What's the difference between open3 and open4? – Psi

Answer 6

Some things to think about when choosing between these mechanisms are:

1. Do you just want stdout or do you need stderr as well? Or even separated out?
2. How big is your output? Do you want to hold the entire result in memory?
3. Do you want to read some of your output while the subprocess is still running?
4. Do you need result codes?
5. Do you need a Ruby object that represents the process and lets you kill it on demand?

You may need anything from simple backticks (``), system(), and IO.popen to full-blown Kernel.fork/Kernel.exec with IO.pipe and IO.select.

You may also want to throw timeouts into the mix if a sub-process takes too long to execute.

Unfortunately, it very much depends.

Trishtrisha

Answer 7

I'm definitely not a Ruby expert, but I'll give it a shot:

$ irb 
system "echo Hi"
Hi
=> true

You should also be able to do things like:

cmd = 'ls'
system(cmd)

Twiggy

Answer 8

One more option:

When you:

• need stderr as well as stdout
• can't/won't use Open3/Open4 (they throw exceptions in NetBeans on my Mac, no idea why)

You can use shell redirection:

puts %x[cat bogus.txt].inspect
  => ""

puts %x[cat bogus.txt 2>&1].inspect
  => "cat: bogus.txt: No such file or directory\n"

The 2>&1 syntax works across Linux, Mac and Windows since the early days of MS-DOS.

Agamemnon

Answer 9

The answers above are already quite great, but I really want to share the following summary article: "6 Ways to Run Shell Commands in Ruby"

Basically, it tells us:

Kernel#exec:

exec 'echo "hello $HOSTNAME"'

system and $?:

system 'false' 
puts $?

Backticks (`):

today = `date`

IO#popen:

IO.popen("date") { |f| puts f.gets }

Open3#popen3 -- stdlib:

require "open3"
stdin, stdout, stderr = Open3.popen3('dc') 

Open4#popen4 -- a gem:

require "open4" 
pid, stdin, stdout, stderr = Open4::popen4 "false" # => [26327, #<IO:0x6dff24>, #<IO:0x6dfee8>, #<IO:0x6dfe84>]

Khalil

Answer 10

If you really need Bash, per the note in the "best" answer.

First, note that when Ruby calls out to a shell, it typically calls /bin/sh, not Bash. Some Bash syntax is not supported by /bin/sh on all systems.

If you need to use Bash, insert bash -c "your Bash-only command" inside of your desired calling method:

quick_output = system("ls -la")

quick_bash = system("bash -c 'ls -la'")

To test:

system("echo $SHELL")
system('bash -c "echo $SHELL"')

Or if you are running an existing script file like

script_output = system("./my_script.sh")

Ruby should honor the shebang, but you could always use

system("bash ./my_script.sh")

to make sure, though there may be a slight overhead from /bin/sh running /bin/bash, you probably won't notice.

Rhododendron

Answer 11

You can also use the backtick operators (`), similar to Perl:

directoryListing = `ls /`
puts directoryListing # prints the contents of the root directory

Handy if you need something simple.

Which method you want to use depends on exactly what you're trying to accomplish; check the docs for more details about the different methods.

Groningen

Answer 12

We can achieve it in multiple ways.

Using Kernel#exec, nothing after this command is executed:

exec('ls ~')

Using backticks or %x

`ls ~`
=> "Applications\nDesktop\nDocuments"
%x(ls ~)
=> "Applications\nDesktop\nDocuments"

Using Kernel#system command, returns true if successful, false if unsuccessful and returns nil if command execution fails:

system('ls ~')
=> true

Palmary

Answer 13

Using the answers here and linked in Mihai's answer, I put together a function that meets these requirements:

1. Neatly captures STDOUT and STDERR so they don't "leak" when my script is run from the console.
2. Allows arguments to be passed to the shell as an array, so there's no need to worry about escaping.
3. Captures the exit status of the command so it is clear when an error has occurred.

As a bonus, this one will also return STDOUT in cases where the shell command exits successfully (0) and puts anything on STDOUT. In this manner, it differs from system, which simply returns true in such cases.

Code follows. The specific function is system_quietly:

require 'open3'

class ShellError < StandardError; end

#actual function:
def system_quietly(*cmd)
  exit_status=nil
  err=nil
  out=nil
  Open3.popen3(*cmd) do |stdin, stdout, stderr, wait_thread|
    err = stderr.gets(nil)
    out = stdout.gets(nil)
    [stdin, stdout, stderr].each{|stream| stream.send('close')}
    exit_status = wait_thread.value
  end
  if exit_status.to_i > 0
    err = err.chomp if err
    raise ShellError, err
  elsif out
    return out.chomp
  else
    return true
  end
end

#calling it:
begin
  puts system_quietly('which', 'ruby')
rescue ShellError
  abort "Looks like you don't have the `ruby` command. Odd."
end

#output: => "/Users/me/.rvm/rubies/ruby-1.9.2-p136/bin/ruby"

Ratcliff

Answer 14

Don't forget the spawn command to create a background process to execute the specified command. You can even wait for its completion using the Process class and the returned pid:

pid = spawn("tar xf ruby-2.0.0-p195.tar.bz2")
Process.wait pid

pid = spawn(RbConfig.ruby, "-eputs'Hello, world!'")
Process.wait pid

The doc says: This method is similar to #system but it doesn't wait for the command to finish.

Jointless

Kernel.spawn() seems to be far more versatile than all other options. – Tambac

Answer 15

The easiest way is, for example:

reboot = `init 6`
puts reboot

Newer users please note that on linux this will reboot your system.

Dress

Answer 16

If you have a more complex case than the common case that can not be handled with ``, then check out Kernel.spawn(). This seems to be the most generic/full-featured provided by stock Ruby to execute external commands.

You can use it to:

• create process groups (Windows).
• redirect in, out, error to files/each-other.
• set env vars, umask.
• change the directory before executing a command.
• set resource limits for CPU/data/etc.
• Do everything that can be done with other options in other answers, but with more code.

The Ruby documentation has good enough examples:

env: hash
  name => val : set the environment variable
  name => nil : unset the environment variable
command...:
  commandline                 : command line string which is passed to the standard shell
  cmdname, arg1, ...          : command name and one or more arguments (no shell)
  [cmdname, argv0], arg1, ... : command name, argv[0] and zero or more arguments (no shell)
options: hash
  clearing environment variables:
    :unsetenv_others => true   : clear environment variables except specified by env
    :unsetenv_others => false  : dont clear (default)
  process group:
    :pgroup => true or 0 : make a new process group
    :pgroup => pgid      : join to specified process group
    :pgroup => nil       : dont change the process group (default)
  create new process group: Windows only
    :new_pgroup => true  : the new process is the root process of a new process group
    :new_pgroup => false : dont create a new process group (default)
  resource limit: resourcename is core, cpu, data, etc.  See Process.setrlimit.
    :rlimit_resourcename => limit
    :rlimit_resourcename => [cur_limit, max_limit]
  current directory:
    :chdir => str
  umask:
    :umask => int
  redirection:
    key:
      FD              : single file descriptor in child process
      [FD, FD, ...]   : multiple file descriptor in child process
    value:
      FD                        : redirect to the file descriptor in parent process
      string                    : redirect to file with open(string, "r" or "w")
      [string]                  : redirect to file with open(string, File::RDONLY)
      [string, open_mode]       : redirect to file with open(string, open_mode, 0644)
      [string, open_mode, perm] : redirect to file with open(string, open_mode, perm)
      [:child, FD]              : redirect to the redirected file descriptor
      :close                    : close the file descriptor in child process
    FD is one of follows
      :in     : the file descriptor 0 which is the standard input
      :out    : the file descriptor 1 which is the standard output
      :err    : the file descriptor 2 which is the standard error
      integer : the file descriptor of specified the integer
      io      : the file descriptor specified as io.fileno
  file descriptor inheritance: close non-redirected non-standard fds (3, 4, 5, ...) or not
    :close_others => false : inherit fds (default for system and exec)
    :close_others => true  : dont inherit (default for spawn and IO.popen)

Tambac

Answer 17

The backticks (`) method is the easiest one to call shell commands from Ruby. It returns the result of the shell command:

     url_request = 'http://google.com'
     result_of_shell_command = `curl #{url_request}`

Transitive

Answer 18

Given a command like attrib:

require 'open3'

a="attrib"
Open3.popen3(a) do |stdin, stdout, stderr|
  puts stdout.read
end

I've found that while this method isn't as memorable as

system("thecommand")

or

`thecommand`

in backticks, a good thing about this method compared to other methods is backticks don't seem to let me puts the command I run/store the command I want to run in a variable, and system("thecommand") doesn't seem to let me get the output whereas this method lets me do both of those things, and it lets me access stdin, stdout and stderr independently.

See "Executing commands in ruby" and Ruby's Open3 documentation.

Anesthetist

Answer 19

This is not really an answer but maybe someone will find it useful:

When using TK GUI on Windows, and you need to call shell commands from rubyw, you will always have an annoying CMD window popping up for less then a second.

To avoid this you can use:

WIN32OLE.new('Shell.Application').ShellExecute('ipconfig > log.txt','','','open',0)

or

WIN32OLE.new('WScript.Shell').Run('ipconfig > log.txt',0,0)

Both will store the ipconfig output inside log.txt, but no windows will come up.

You will need to require 'win32ole' inside your script.

system(), exec() and spawn() will all pop up that annoying window when using TK and rubyw.

Goldbrick

Answer 20

Not sure about shell commands. I used following for capturing system command's output into a variable val:

val = capture(:stdout) do
  system("pwd")
end

puts val

shortened version:

val = capture(:stdout) { system("pwd") }

capture method is provided by active_support/core_ext/kernel/reporting.rb

Simlarly we can also capture standard errors too with :stderr

Twinberry

Answer 21

Here's a cool one that I use in a ruby script on OS X (so that I can start a script and get an update even after toggling away from the window):

cmd = %Q|osascript -e 'display notification "Server was reset" with title "Posted Update"'|
system ( cmd )

Finicky

Answer 22

You can use format method as below to print some information:

puts format('%s', `ps`)
puts format('%d MB', (`ps -o rss= -p #{Process.pid}`.to_i / 1024))

Djambi