Streaming commands output progress

Question

I'm writing a service that has to stream output of a executed command both to parent and to log. When there is a long process, the problem is that cmd.StdoutPipe gives me a final (string) result.

Is it possible to give partial output of what is going on, like in shell

func main() {
    cmd := exec.Command("sh", "-c", "some long runnig task")

    stdout, _ := cmd.StdoutPipe()
    cmd.Start()

    scanner := bufio.NewScanner(stdout)
    for scanner.Scan() {
        m := scanner.Text()
        fmt.Println(m)
        log.Printf(m)
    }

    cmd.Wait()
}

P.S. Just to output would be:

cmd.Stdout = os.Stdout

But in my case it is not enough.

Johnsonian

I am intersted in syntax, what is this language? – Hydrodynamic

@AliTorabi It's Go. – Seemaseeming

Not a direct answer (i.e. \n as @icza mentions) but you may find io.MultiWriter useful since you say you need to send the output two places. In particular, if you don't need the formatting that the log package gives you can just do something like: play.golang.org/p/4PG3qbiQx9 – Heliotropism

Answer 1

The code you posted works (with a reasonable command executed).

Here is a simple "some long running task" written in Go for you to call and test your code:

func main() {
    fmt.Println("Child started.")
    time.Sleep(time.Second*2)
    fmt.Println("Tick...")
    time.Sleep(time.Second*2)
    fmt.Println("Child ended.")
}

Compile it and call it as your command. You will see the different lines appear immediately as written by the child process, "streamed".

Reasons why it may not work for you

The Scanner returned by bufio.NewScanner() reads whole lines and only returns something if a newline character is encountered (as defined by the bufio.ScanLines() function).

If the command you execute doesn't print newline characters, its output won't be returned immediately (only when newline character is printed, internal buffer is filled or the process ends).

Possible workarounds

If you have no guarantee that the child process prints newline characters but you still want to stream the output, you can't read whole lines. One solution is to read by words, or even read by characters (runes). You can achieve this by setting a different split function using the Scanner.Split() method:

scanner := bufio.NewScanner(stdout)
scanner.Split(bufio.ScanRunes)

The bufio.ScanRunes function reads the input by runes so Scanner.Scan() will return whenever a new rune is available.

Or reading manually without a Scanner (in this example byte-by-byte):

oneByte := make([]byte, 1)
for {
    _, err := stdout.Read(oneByte)
    if err != nil {
        break
    }
    fmt.Printf("%c", oneByte[0])
}

Note that the above code would read runes that multiple bytes in UTF-8 encoding incorrectly. To read multi UTF-8-byte runes, we need a bigger buffer:

oneRune := make([]byte, utf8.UTFMax)
for {
    count, err := stdout.Read(oneRune)
    if err != nil {
        break
    }
    fmt.Printf("%s", oneRune[:count])
}

Things to keep in mind

Processes have default buffers for standard output and for standard error (usually the size of a few KB). If a process writes to the standard output or standard error, it goes into the respective buffer. If this buffer gets full, further writes will block (in the child process). If you don't read the standard output and standard error of a child process, your child process may hang if the buffer is full.

So it is recommended to always read both the standard output and error of a child process. Even if you know that the command don't normally write to its standard error, if some error occurs, it will probably start dumping error messages to its standard error.

Edit: As Dave C mentions by default the standard output and error streams of the child process are discarded and will not cause a block / hang if not read. But still, by not reading the error stream you might miss a thing or two from the process.

Blowup

Yes, as you mentioned there were a problem with line ending, thank you. – Johnsonian

@icza, reading/checking stderr is a good idea but not reading it when using os/exec isn't a problem since, unless you override it (via cmd.StderrPipe or setting cmd.Stderr for example), the default is to connect them to os.DevNull (i.e. output would be discarded, not buffered and blocking). – Heliotropism

@DaveC Oh, good to know. Thanks. In the Java world stderr is not discarded by default and I've seen others having (and myself experiencing) mysterious errors when not read properly. – Blowup