How to execute a shell script in Crystal while capturing output?

Question

I want to execute a shell script while handling stdout and stderr output. Currently I execute commands using Process.run, with shell=false and three pipes for stdin, stdout and stderr. I spawn fibers to read from stdout and stderr and log (or otherwise process) the output. This works pretty well for individual commands, but fails horribly for scripts.

I could simply set shell=true when calling Process.run, but looking at the Crystal source it seems that merely prepends "sh" to the commandline. I've tried prepending "bash" and it didn't help.

Things like redirection (>file) and pipes (e.g. curl something | bash) don't seem to work with Process.run

For example, to download a shell script and execute it, I tried:

cmd = %{bash -c "curl http://dist.crystal-lang.org/apt/setup.sh" | bash}

Process.run(cmd, ...)

The initial bash was added in the hope that it would enable the pipe operator. It doesn't seem to help. I also tried executing each command separately:

script.split("\n").reject(/^#/, "").each { Process.run(...) }

But of course, that still fails when a command uses redirection or pipes. For example, the command echo "deb http://dist.crystal-lang.org/apt crystal main" >/etc/apt/sources.list.d/crystal.list simply outputs:

"deb http://dist.crystal-lang.org/apt crystal main" >/etc/apt/sources.list.d/crystal.list`

It might work if I used the `` backticks method of execution instead; but then I wouldn't be able to capture the output in real time.

Citystate

you need to move the pipeline into the bash -c in the first case to get it to invoke. You're fighting against shell=false, though, which is the thing that would allow you to use shell syntax directly. – Denunciate

@Petesh Sadly, setting shell=true doesn't help any. – Citystate

@Petesh Moving the pipeline into the bash -c call seems to work, however. So...why doesn't shell=true work? Surely it's pretty much the same thing? – Citystate

Answer 1

I'm basing my understanding on reading the source code of the run.cr file. The behaviour is very similar to other languages in how it deals with commands and arguments.

Without shell=true, the default behaviour of Process.run is to use the command as the executable to run. This means that the string needs to be a program name, without any arguments, e.g. uname would be a valid name as there's a program on my system called uname in /usr/bin.

If you ever got behaviour of successfully using %{bash -c "echo hello world"} with shell=false, then something is wrong - the default behaviour should have been to try to run a program called bash -c "echo hello world", which is unlikely to exist on any system.

Once you pass in 'shell=true', then it does sh -c <command>, which will allow strings like echo hello world as a command to work; this will also allow redirections and pipelines to work.

The shell=true behaviour can generally be interpreted as doing the following:

cmd = "sh"
args = [] of String
args << "-c" << "curl http://dist.crystal-lang.org/apt/setup.sh | bash"
Process.run(cmd, args, …)

Note that I'm using an array of arguments here - without the array of arguments, you don't have any control over how the arguments are passed into the shell.

The reason why the first version, with or without shell=true doesn't work is because the pipeline is outside the -c, which is the command you're sending to bash.

Denunciate

I think the problem was due to the fact I was splitting the command string into cmd (a string) and args (an array) and then calling Process.new(cmd, args, ...). Even with shell=true, it didn't work - probably because I had already split the | into an argument of its own... – Citystate

Basically, I ended up calling Process.new("curl", ["http://dist.crystal-lang.org/apt/setup.sh", "|", "bash"], shell: true). And of course this didn't work. To support shell=true, I needed to modify my wrapper function to avoid splitting the command into words if shell is true. – Citystate

Answer 2

The problem is a UNIX problem. The parent process must be capable to access the STDOUT of the child process. Using a pipe you must start a shell process that will run the whole command, including the | bash and not just curl $URL. In Crystal this is:

command = "curl http://dist.crystal-lang.org/apt/setup.sh | bash"
io = MemoryIO.new
Process.run(command, shell: true, output: io)
output = io.to_s

Or if you want to duplicate what Crystal does for you:

Process.run("sh", {"-c", command}, output: io)

Hyozo

Answer 3

or if you want to call a shell script and get the output i just tried with crystal 0.23.1 and it's work !

def screen
    output = IO::Memory.new
     Process.run("bash", args: {"lib/bash_scripts/installation.sh"}, output: output)
     output.close
    output.to_s
end

Schuller