Java: is there a way to run a system command and print the output during execution?

Question

I have a python script and it takes a long time to finish. I would like to run it from Java, but also output the script's output while it is executing, so that I can tell if it is properly running.

I've searched and only found examples where we output the output after the system command has finished, rather than during its execution.

Any way to do it while the script is running?

Here's what I have

public void doSomething() throws IOException {
    String[] callAndArgs = {"python", "/hi.py"};
    Process p = Runtime.getRuntime().exec(callAndArgs);
    BufferedReader stdInput = new BufferedReader(new InputStreamReader(p.getInputStream()));
    BufferedReader stdError = new BufferedReader(new InputStreamReader(p.getErrorStream()));

    String s;
    while ((s = stdInput.readLine()) != null) {
        System.out.println(s);
    }

    while ((s = stdError.readLine()) != null) {
        System.out.println(s);
    }
}

Separation

FWIW, I've replicated your results at my end. It's not just you. – Mclain

Also FWIW: The variable stdInput really should be called stdOutput. :-) – Mclain

Even reading individual chars from the raw InputStream doesn't give us anything until the end. I think it's a python thing. – Mclain

And finally: If I have it spawn something else (in my case, java running a Java program that has a pause), I see the output as it goes, not all at once at the end. It's a python thing. I don't know what, so it's not an answer, hence posting the comment instead. – Mclain

Answer 1

i managed to get it working like this (Note it requires java7):

package test;
import java.lang.ProcessBuilder.Redirect;

public class Test {

    public static void main(String... args) throws Exception {
        ProcessBuilder pb = new ProcessBuilder("python","/home/foobar/Programming/test/src/test/test.py");
        pb.redirectOutput(Redirect.INHERIT);
        Process p = pb.start();
        p.waitFor();
    }

}

python (note i flush on python to make it work using sys.stdout.flush())

import time,sys
c =0
while c<=50:
    time.sleep(1)
    print("----")
    c = c +1
    sys.stdout.flush()

Note if you don't want to flush in a loop you can use this:

ProcessBuilder pb = new ProcessBuilder("python","-u","/home/foobar/Programming/NetBeansProjects/test/src/test/test.py");

Redirect.INHERIT

Indicates that subprocess I/O source or destination will be the same as those of the current process. This is the normal behavior of most operating system command interpreters (shells).

Gloze

It's worth noting that your original code works just fine if you flush on the Python side as well, so MadProgrammer was correct that it's a Python-side fix that's required. – Mclain

Answer 2

I've searched and only found examples where we output the output after the system command has finished, rather than during its execution.

That's weird, because your example should be dumping the output as the command is executing.

Instead of using BufferedReader, you could try reading directly from the InputStream instead as the required conditions for readLine might not be being met until after the process exits.

I'd also recommend that you use a ProcessBuilder over Process directly, as, apart from anything else, it allows you to redirect the output from the error stream into the input stream, allowing you to read just one stream instead of two...

This might also be an issue with Python and how it flushes it output buffers...

For example, rather then waiting for the BufferedReader to decide when to return, try printing each character from the stream as it occurs/is reported

            ProcessBuilder pb = new ProcessBuilder("test.py");
            pb.redirectError();
            Process p = pb.start();

            InputStream is = null;
            try {
                is = p.getInputStream();
                int in = -1;
                while ((in = is.read()) != -1) {
                    System.out.print((char)in);
                }
            } finally {
                try {
                    is.close();
                } catch (Exception e) {
                }
            }

Update

Doing a little reading, Python seems to be buffering its out out before sending it to the stdout. I don't think you can fix this on the a Java side, but need to alter either the way Python is run or the script works.

See How to flush output of Python print? for more details

Hasa

I've already done that experiment. InputStream doesn't give any characters until the process terminates. I think it must be a python thing. – Mclain

+1 flush in the Python side does indeed sort out the problem (even with the OP's original code). – Mclain

@Hasa feel free to add the python code from my answer in your answer because it seems to be a better approach. also you can use python -u for flush – Gloze

Answer 3

I'm suspecting that you are writing to stderr, which you can't see because you are blocking on stdin. Use a ProcessBuilder instead of doing exec. This way, you can redirect stderr and stdin into a single stream.

Here is an example:

import java.io.*;

public class Test {
    public static void main(String... args) throws IOException {

        ProcessBuilder pb =
                new ProcessBuilder("test.py");

        pb.redirectErrorStream(true);
        Process proc = pb.start();

        Reader reader = new InputStreamReader(proc.getInputStream());
        BufferedReader bf = new BufferedReader(reader);
        String s;
        while ((s = bf.readLine()) != null) {
            System.out.println(s);
        }
    }
}

Alternatively you can spawn threads to read from stdin/stderr respectively.

Another thing to look for is output buffering by python. You can see if this is the cause by doing:

import sys
sys.stdout.flush()

after you write to stdout

Ample

What makes you think that Runtime.exec blocks until the program terminates? "...and thus you can read stdin while the script is running..." stdout (from the process), I think you mean. – Mclain

i think in order to block you have to use process.waitFor(); not sure – Gloze

i edit your post made a BufferedReader(reader) , then use bufferedReader.readLine() to compile – Gloze

i'm interested in the solution so i copied the code in ide to run it :) – Gloze

This doesn't work either. Same problem as the OP's. (And I'm writing to stdout, not stderr.) – Mclain

@T.J.Crowder: I don't really see how this won't work? I'm suspecting his python script is erroring and writing to stderr, which the code never reaches because it's blocking on stdin. – Ample

Answer 4

Don't use #readLine as the conditional in your while loop. Instead wrap your inputStream in a scanner and use #hasNextLine()

    Scanner in = new Scanner(p.getInputStream());
    while (in.hasNextLine()) {
        System.out.println(in.nextLine());
    }

Dogger

@Hasa Because he wants it to be updated AS it runs, not AFTER it runs. – Dogger

Okay, let's start with, there's no hasNextLine or nextLine methods in BufferedRead and move onto that readLine "Reads a line of text. A line is considered to be terminated by any one of a line feed ('\n'), a carriage return ('\r'), or a carriage return followed immediately by a linefeed." – Hasa

I still don't see how this is going to e any different fro using a BufferedReader? – Hasa

I can confirm that using Scanner instead doesn't change anything. – Mclain