Why does Runtime.exec(String) work for some but not all commands?

Question

When I try to run Runtime.exec(String), certain commands work, while other commands are executed but fail or do different things than in my terminal. Here is a self-contained test case that demonstrates the effect:

public class ExecTest {
  static void exec(String cmd) throws Exception {
    Process p = Runtime.getRuntime().exec(cmd);

    int i;
    while( (i=p.getInputStream().read()) != -1) {
      System.out.write(i);
    }
    while( (i=p.getErrorStream().read()) != -1) {
      System.err.write(i);
    }
  }

  public static void main(String[] args) throws Exception {
    System.out.print("Runtime.exec: ");
    String cmd = new java.util.Scanner(System.in).nextLine();
    exec(cmd);
  }
}

The example works great if I replace the command with echo hello world, but for other commands -- especially those involving filenames with spaces like here -- I get errors even though the command is clearly being executed:

myshell$ javac ExecTest.java && java ExecTest
Runtime.exec: ls -l 'My File.txt'
ls: cannot access 'My: No such file or directory
ls: cannot access File.txt': No such file or directory

meanwhile, copy-pasting to my shell:

myshell$ ls -l 'My File.txt'
-rw-r--r-- 1 me me 4 Aug  2 11:44 My File.txt

Why is there a difference? When does it work and when does it fail? How do I make it work for all commands?

Outguess

Answer 1

Why do some commands fail?

This happens because the command passed to Runtime.exec(String) is not executed in a shell. The shell performs a lot of common support services for programs, and when the shell is not around to do them, the command will fail.

When do commands fail?

A command will fail whenever it depends on a shell features. The shell does a lot of common, useful things we don't normally think about:

1. The shell splits correctly on quotes and spaces

   This makes sure the filename in "My File.txt" remains a single argument.

   Runtime.exec(String) naively splits on spaces and would pass this as two separate filenames. This obviously fails.

2. The shell expands globs/wildcards

   When you run ls *.doc, the shell rewrites it into ls letter.doc notes.doc.

   Runtime.exec(String) doesn't, it just passes them as arguments.

   ls has no idea what * is, so the command fails.

3. The shell manages pipes and redirections.

   When you run ls mydir > output.txt, the shell opens "output.txt" for command output and removes it from the command line, giving ls mydir.

   Runtime.exec(String) doesn't. It just passes them as arguments.

   ls has no idea what > means, so the command fails.

4. The shell expands variables and commands

   When you run ls "$HOME" or ls "$(pwd)", the shell rewrites it into ls /home/myuser.

   Runtime.exec(String) doesn't, it just passes them as arguments.

   ls has no idea what $ means, so the command fails.

What can you do instead?

There are two ways to execute arbitrarily complex commands:

Simple and sloppy: delegate to a shell.

You can just use Runtime.exec(String[]) (note the array parameter) and pass your command directly to a shell that can do all the heavy lifting:

// Simple, sloppy fix. May have security and robustness implications
String myFile = "some filename.txt";
String myCommand = "cp -R '" + myFile + "' $HOME 2> errorlog";
Runtime.getRuntime().exec(new String[] { "bash", "-c", myCommand });

Secure and robust: take on the responsibilities of the shell.

This is not a fix that can be mechanically applied, but requires an understanding the Unix execution model, what shells do, and how you can do the same. However, you can get a solid, secure and robust solution by taking the shell out of the picture. This is facilitated by ProcessBuilder.

The command from the previous example that requires someone to handle 1. quotes, 2. variables, and 3. redirections, can be written as:

String myFile = "some filename.txt";
ProcessBuilder builder = new ProcessBuilder(
    "cp", "-R", myFile,        // We handle word splitting
       System.getenv("HOME")); // We handle variables
builder.redirectError(         // We set up redirections
    ProcessBuilder.Redirect.to(new File("errorlog")));
builder.start();

Outguess

Runtime.getRuntime().exec(new String[] { System.getenv("SHELL"), "-c", myCommand }); – Ayeaye

Prefer explicitly specifying a shell that you know can understand the command. Your program should not stop working just because a user picks fish or pwsh as their login shell. – Outguess

It seems like the "simple and sloppy" approach (which I think is adequate for what I am doing) only handles one command at a time. How can a sequence of commands be handled? – Astrea

Same way as in as shell: separating with ; or && or linefeeds – Outguess