Getting output of system() calls in Ruby

Question

If I call a command using Kernel#system in Ruby, how do I get its output?

system("ls")

Irreverence

You may want to have a look at this thread in comp.lang.ruby – Nichy

This is a very hand thread, thanks. The class for running commands and getting feedback is great in the sample code. – Ianiana

For future googlers. If you want to learn about other system command calls and their differences, see this SO answer. – Munch

Answer 1

I'd like to expand & clarify chaos's answer a bit.

If you surround your command with backticks, then you don't need to (explicitly) call system() at all. The backticks execute the command and return the output as a string. You can then assign the value to a variable like so:

output = `ls`
p output

or

printf output # escapes newline chars

Rufinaruford

what if I need to give a variable as part of my command? That is, what would something like system("ls " + filename) translate into when backticks are to be used? – Topazolite

You can do expression evaluation just as you would with regular strings: ls #{filename}. – Rufinaruford

Hrm, makdown ate my formatting. There are back ticks ahead of the "l" in "ls" and after the "}" in "#{filename}". – Rufinaruford

Craig - in markdown you can quote with the backslash character: `ls #{filename}` – Indecorous

Does this capture all output? – Trinee

No, it only captures stdout, not stderr. – Rufinaruford

This answer isn't advisable: it introduces the new problem of unsanitized user input. – Nordic

@Dogweather: that may be true, but is it any different than any of the other methods? – Rufinaruford

if you want to capure stderr just put 2>&1 at the end of your command. e.g output = command 2>&1 – Geochemistry

I think the answer @martin-gross provides is the cleaner solution; while both work, this is what %x is for. – Impertinent

backticks is not equivalent to system, you can't pass environment, you can separate stdout and stderrr etc – Kingpin

@CraigWalker: I think micred's comment should be added to your answer. It's rather helpful. if you want to capure stderr just put 2>&1 at the end of your command. e.g output = command 2>&1 – Thoraco

It's often necessary to cut a trailing newline with output = `ls`.tr("\n","") – Sc

@Nordic It only introduces unsanitized user input if variables with user input that hasn't yet been sanitized are used. An experienced Ruby programmer would be careful to label relevant vars (e.g. filename_sane or filename_escd) and only use those in dangerous contexts such as this. – Minerva

Does anyone know whether the backtick operator is actually thread-safe? That is, if I pass different commands from different threads to it, and store the output in thread-local variables, are the variables guaranteed to only contain the output from the command I called in a particular thread, or might the output be manged with the output form command I called in another thread? – Pons

@CraigWalker. Your printf is not syntactically correct. – Indrawn

Backticks are perfectly fine in a program that does not accept user input, such as the one shown here. The only way to modify the system call is to modify the source, and if someone can modify the source, well... – Actuary

Be aware that using system and backticks, are similar, but not the same. When in doubt, try both. – Kamilah

This works only if you command is static. If you try to build dynamic command like "ls #{directory_path}" it wont work. – Patronage

Answer 2

Be aware that all the solutions where you pass a string containing user provided values to system, %x[] etc. are unsafe! Unsafe actually means: the user may trigger code to run in the context and with all permissions of the program.

As far as I can say only system and Open3.popen3 do provide a secure/escaping variant in Ruby 1.8. In Ruby 1.9 IO::popen also accepts an array.

Simply pass every option and argument as an array to one of these calls.

If you need not just the exit status but also the result you probably want to use Open3.popen3:

require 'open3'
stdin, stdout, stderr, wait_thr = Open3.popen3('usermod', '-p', @options['shadow'], @options['username'])
stdout.gets(nil)
stdout.close
stderr.gets(nil)
stderr.close
exit_code = wait_thr.value

Note that the block form will auto-close stdin, stdout and stderr- otherwise they'd have to be closed explicitly.

More information here: Forming sanitary shell commands or system calls in Ruby

Briant

This is the only answer that actually answers the question and solves the problem without introducing new ones (unsanitized input). – Nordic

Thanks! This is the sort of answer I was hoping for. One correction: the gets calls should pass the argument nil, as otherwise we just get the first line of the output. So e.g. stdout.gets(nil). – Hulky

stdin, stdout and stderr should be closed explicitly in non-block form. – Monophonic

Does anybody know if something changed in Ruby 2.0 or 2.1? Edits or comments would be appreciated;-) – Aduwa

I think that the discussion around Open3.popen3 is missing a major problem: If you have a subprocess that writes more data to stdout than a pipe can hold, subprocess gets suspend in stderr.write, and your program gets stuck in stdout.gets(nil). – Tun

@simon I am using the same code but I dont know for some reason in nginx+passenger the application is not waiting for the response from the script I dont know what is happening, please check the issue here #28697185 – Jounce

block form here (near bottom): makandracards.com/makandra/…. Note that the string marked '/some/command' should include all the arguments. I was trying stdin.puts "my arguments", but that's only needed if the executed command consumes user input – Deglutinate

Answer 3

Just for the record, if you want both (output and operation result) you can do:

output=`ls no_existing_file` ;  result=$?.success?

Pretypify

That only captures stdout, and stderr goes to the console. To get stderr, use: output=`ls no_existing_file 2>&1`; result=$?.success? – Isochronal

This answer is unsafe and shouldn't be used -- if the command is anything but a constant, then the backtick syntax is likely to cause a bug, possibly a security vulnerability. (And even if it is a constant, it will probably cause someone to use it for a non-constant later and cause a bug.) See Simon Hürlimann's answer for a correct solution. – Hulky

kudos to Greg Price for understanding about the need to escape user input, but it is not correct to say this answer as written is unsafe. The Open3 method mentioned is more complicated and introduces more dependencies, and the argument that someone will "use it for a non-constant later" is a strawman. True, you probably wouldn't use them in a Rails app, but for a simple system utility script with no possibility of untrusted user input, backticks are perfectly fine and nobody should be made to feel bad about using them. – Aceous

Two major issues with this solution are 1. user input is not escaped 2. $? is a global variable and thus I think it is not thread-safe – Teddi

Answer 4

The straightforward way to do this correctly and securely is to use Open3.capture2(), Open3.capture2e(), or Open3.capture3().

Using ruby's backticks and its %x alias are NOT SECURE UNDER ANY CIRCUMSTANCES if used with untrusted data. It is DANGEROUS, plain and simple:

untrusted = "; date; echo"
out = `echo #{untrusted}`                              # BAD

untrusted = '"; date; echo"'
out = `echo "#{untrusted}"`                            # BAD

untrusted = "'; date; echo'"
out = `echo '#{untrusted}'`                            # BAD

The system function, in contrast, escapes arguments properly if used correctly:

ret = system "echo #{untrusted}"                       # BAD
ret = system 'echo', untrusted                         # good

Trouble is, it returns the exit code instead of the output, and capturing the latter is convoluted and messy.

The best answer in this thread so far mentions Open3, but not the functions that are best suited for the task. Open3.capture2, capture2e and capture3 work like system, but returns two or three arguments:

out, err, st = Open3.capture3("echo #{untrusted}")     # BAD
out, err, st = Open3.capture3('echo', untrusted)       # good
out_err, st  = Open3.capture2e('echo', untrusted)      # good
out, st      = Open3.capture2('echo', untrusted)       # good
p st.exitstatus

Another mentions IO.popen(). The syntax can be clumsy in the sense that it wants an array as input, but it works too:

out = IO.popen(['echo', untrusted]).read               # good

---

For convenience, you can wrap Open3.capture3() in a function, e.g.:

#
# Returns stdout on success, false on failure, nil on error
#
def syscall(*cmd)
  begin
    stdout, stderr, status = Open3.capture3(*cmd)
    status.success? && stdout.slice!(0..-(1 + $/.size)) # strip trailing eol
  rescue
  end
end

Example:

p system('foo')
p syscall('foo')
p system('which', 'foo')
p syscall('which', 'foo')
p system('which', 'which')
p syscall('which', 'which')

Yields the following:

nil
nil
false
false
/usr/bin/which         <— stdout from system('which', 'which')
true                   <- p system('which', 'which')
"/usr/bin/which"       <- p syscall('which', 'which')

Klaxon

This is the correct answer. It is also the most informative. The only thing missing is a warning about closing the std*s. See this other comment: require 'open3'; output = Open3.popen3("ls") { |stdin, stdout, stderr, wait_thr| stdout.read } Note that the block form will auto-close stdin, stdout and stderr- otherwise they'd have to be closed explicitly. – Ceasefire

@PeterH.Boling: Best I'm aware, the capture2, capture2e and capture3 also close them std*s automatically. (At the very least, I never ran into the problem on my end.) – Klaxon

without using the block form there is no way for a codebase to know when something should be closed, so I highly doubt they are being closed. You probably never ran into a problem because not closing them won't cause problems in a short lived process, and if you restart a long-running process often enough, otto won't show up there either unless you are opening std*s in a loop. Linux has a high file descriptor limit, which you can hit, but until you hit it you won't see the "bug". – Ceasefire

@PeterH.Boling: No no, see the source code. The functions are just wrappers around Open3#popen2, popen2e and popen3 with a predefined block: ruby-doc.org/stdlib-2.1.1/libdoc/open3/rdoc/… – Klaxon

@Dennis de Barnardy Perhaps you missed that I linked to the same class documentation (albeit for Ruby 2.0.0, and a different method. ruby-doc.org/stdlib-2.1.1/libdoc/open3/rdoc/… From the example: ``` stdin, stdout, stderr, wait_thr = Open3.popen3([env,] cmd... [, opts]) pid = wait_thr[:pid] # pid of the started process ... stdin.close # stdin, stdout and stderr should be closed explicitly in this form. stdout.close stderr.close ``` I was just quoting the documentation. "# stdin, stdout and stderr should be closed explicitly in this form." – Ceasefire

echo "#{untrusted}" (with the double quotes) is not safe. Let untrusted be "; date; echo". – Tutelary

Good catch @AndreasKrey! – Klaxon

Answer 5

You can use system() or %x[] depending what kind of result you need.

system() returning true if the command was found and ran successfully, false otherwise.

>> s = system 'uptime'
10:56  up 3 days, 23:10, 2 users, load averages: 0.17 0.17 0.14
=> true
>> s.class
=> TrueClass
>> $?.class
=> Process::Status

%x[..] on the other hand saves the results of the command as a string:

>> result = %x[uptime]
=> "13:16  up 4 days,  1:30, 2 users, load averages: 0.39 0.29 0.23\n"
>> p result 
"13:16  up 4 days,  1:30, 2 users, load averages: 0.39 0.29 0.23\n"
>> result.class
=> String

Th blog post by Jay Fields explains in detail the differences between using system, exec and %x[..] .

Othelia

Thanks for the tip of using %x[]. It just solved a problem I had where I used back ticks in a ruby script in Mac OS X. When running the same script on a Windows machine with Cygwin, it failed because of the back ticks, but worked with %x[]. – Rhea

Thanks a lot! %x[..] it's the compact and working solution! – Unify

Answer 6

If you need to escape the arguments, in Ruby 1.9 IO.popen also accepts an array:

p IO.popen(["echo", "it's escaped"]).read

In earlier versions you can use Open3.popen3:

require "open3"

Open3.popen3("echo", "it's escaped") { |i, o| p o.read }

If you also need to pass stdin, this should work in both 1.9 and 1.8:

out = IO.popen("xxd -p", "r+") { |io|
    io.print "xyz"
    io.close_write
    io.read.chomp
}
p out # "78797a"

Brasserie

Answer 7

You use backticks:

`ls`

Insectarium

Backticks do not produce output at the terminal. – Suttles

It doesn't produce stderr but it gives stdout. – Thoraco

It doesn't write to stdout or stderr. Let's try this example ruby -e '%x{ls}' - note, no output. (fyi %x{} is equivalent to backticks.) – Newspaper

This worked great. Using sh would echo the output to the console (i.e. STDOUT) as well as return it. This doesn't. – Chilly

Answer 8

Another way is:

f = open("|ls")
foo = f.read()

Note that's the "pipe" character before "ls" in open. This can also be used to feed data into the programs standard input as well as reading its standard output.

Spearwort

Just used this to read standard output from an aws cli command in order to read the json and not the official return value of 'true' – Deglutinate

Answer 9

I found that the following is useful if you need the return value:

result = %x[ls]
puts result

I specifically wanted to list the pids of all the Java processes on my machine, and used this:

ids = %x[ps ax | grep java | awk '{ print $1 }' | xargs]

Amati

It's a great solution. – Bookstack

Answer 10

As Simon Hürlimann already explained, Open3 is safer than backticks etc.

require 'open3'
output = Open3.popen3("ls") { |stdin, stdout, stderr, wait_thr| stdout.read }

Note that the block form will auto-close stdin, stdout and stderr- otherwise they'd have to be closed explicitly.

Monophonic

Simple and beautiful – Igenia

Answer 11

While using backticks or popen is often what you really want, it doesn't actually answer the question asked. There may be valid reasons for capturing system output (maybe for automated testing). A little Googling turned up an answer I thought I would post here for the benefit of others.

Since I needed this for testing my example uses a block setup to capture the standard output since the actual system call is buried in the code being tested:

require 'tempfile'

def capture_stdout
  stdout = $stdout.dup
  Tempfile.open 'stdout-redirect' do |temp|
    $stdout.reopen temp.path, 'w+'
    yield if block_given?
    $stdout.reopen stdout
    temp.read
  end
end

This method captures any output in the given block using a tempfile to store the actual data. Example usage:

captured_content = capture_stdout do
  system 'echo foo'
end
puts captured_content

You can replace the system call with anything that internally calls system. You could also use a similar method to capture stderr if you wanted.

Allieallied

Answer 12

If you want the output redirected to a file using Kernel#system, you can do modify descriptors like this:

redirect stdout and stderr to a file(/tmp/log) in append mode:

system('ls -al', :out => ['/tmp/log', 'a'], :err => ['/tmp/log', 'a'])

For a long running command, this will store the output in real time. You can also, store the output using a IO.pipe and redirect it from Kernel#system.

Amputee

Answer 13

As a direct system(...) replacement you may use Open3.popen3(...)

Further discussion: http://tech.natemurray.com/2007/03/ruby-shell-commands.html

Petitioner

Answer 14

Simplest solution to capture standard output into a variable named val:

val = capture(:stdout) do
  system("pwd")
end

puts val

shortened version:

val = capture(:stdout) { system("ls") }

capture method is provided by active_support/core_ext/kernel/reporting.rb

Simlarly we can also capture standard errors too with :stderr

Stonebroke

capture was removed from Rails version 5 and above, for thread safety reasons. See PR github.com/rails/rails/pull/13392 – Wernsman

Answer 15

If you want to keep using system you can redirect the output to a pipe:

r, w = IO.pipe
system("ls", out: w)
w.close
output = r.read

Cormac

Answer 16

I didn't find this one here so adding it, I had some issues getting the full output.

You can redirect STDERR to STDOUT if you want to capture STDERR using backtick.

output = `grep hosts /private/etc/* 2>&1`

source: http://blog.bigbinary.com/2012/10/18/backtick-system-exec-in-ruby.html

Wivern

Answer 17

Short answer for most convenient way is:

    require 'open3'

    stdout_str, stderr_str, status = Open3.capture3(cmd)
    puts "exit status: #{status.exitstatus} stdout: #{stdout_str}"

Handspring

Answer 18

You can use a gem, called Frontkick

Frontkick.exec("echo *")

And here is how to check and read it:

result = Frontkick.exec("echo *")

puts result.successful? #=> true if exit_code is 0
puts result.success?    #=> alias to successful?, for compatibility with Process::Status
puts result.stdout      #=> stdout output of the command
puts result.stderr      #=> stderr output of the command
puts result.exit_code   #=> exit_code of the command
puts result.status      #=> alias to exit_code
puts result.exitstatus  #=> alias to exit_code, for compatibility with Process::Status
puts result.duration    #=> the time used to execute the command

Github https://github.com/sonots/frontkick#frontkick

Gem Page https://rubygems.org/gems/frontkick

Riggle

Answer 19

puts `date`
puts $?


Mon Mar  7 19:01:15 PST 2016
pid 13093 exit 0

Folacin