Reading Stdout from a subprocess

Question

I am attempting to spawn a subprocess from Golang. The goal is to read and process the input line-by-line. Here is what I am trying to get working:

func readStuff(scanner *bufio.Scanner) {
    for scanner.Scan() {
        fmt.Println("Performed Scan")
        fmt.Println(scanner.Text())
    }
    if err := scanner.Err(); err != nil {
        fmt.Fprintln(os.Stderr, "reading standard input:", err)
    }
}

func main() {
    cmd := exec.Command("/usr/local/bin/pocketsphinx_continuous", "-inmic", "yes")
    out, err := cmd.StdoutPipe()

    err = cmd.Start()
    checkError(err)

    scanner := bufio.NewScanner(out)
    fmt.Println("Scanner created")

    defer cmd.Wait()
    go readStuff(scanner)
}

In this example, "Scanner created" is printed, but nothing happens after that.

Running this command however does result in what I am expecting to be printed to :

/usr/local/bin/pocketsphinx_continuous -inmic yes 1>out.txt

And modifying the code to directly copy to stdout works as well:

cmd := exec.Command("/usr/local/bin/pocketsphinx_continuous", "-inmic", "yes")
cmd.Stdout = os.Stdout

What am I missing that is keeping me from reading the output?

Seligman

Does this answer your question? How do you get the output of a system command in Go? – Robbynrobe

Answer 1

There are multiple things you may want to check.

1. The error code returned by cmd.StdoutPipe() is not checked. It should be.

2. The pocketsphinx_continuous command requires the -hmm and -dict arguments to be provided. Otherwise, it will fail, and all the output is actually sent to stderr and not stdout. Here, you read only stdout, but there is nothing to read.

3. You should not call cmd.Wait() before being sure all the data have been read from stdout. The result is non deterministic (actually, it is a race condition). Check the documentation about the os/exec package. If you absolutely need the parsing to be done in a goroutine, you need to synchronize with the end of the goroutine before cmd.Wait() is called. For instance, you could write the function as:

   func readStuff(scanner *bufio.Scanner, stop chan bool) {
       // Scanning code
       // ...
       stop<-true
   }
   

   and the main code as:

   stop := make(chan bool)
   go readStuff(scanner,stop)
   <-stop
   cmd.Wait()
   

Kare

Thanks for the response. I updated my code to check that error, and not call wait before reading all the lines. I am still only seeing lines from StdErr. (PocketSphinx will still work without those arguments, it just has poor recognition) hastebin.com/ojetacoraw.go – Seligman

Answer 2

Seems you need not

go readStuff(scanner)

because of

cmd.Start()

do system fork itself so just

readStuff(scanner)

would be enough to my mind (not spawning gorouting for that)

Furlani

Answer 3

This seems to work fine, and it works with go readStuff(scanner) and also with just readStuff(scanner) - I don't think that this usage actually calls for a goroutine, but don't know the actual context.:

package main

import (
    "bufio"
    "fmt"
    "os"
    "os/exec"
)

func readStuff(scanner *bufio.Scanner) {
    for scanner.Scan() {
        fmt.Println("Performed Scan")
        fmt.Println(scanner.Text())
    }
    if err := scanner.Err(); err != nil {
        fmt.Fprintln(os.Stderr, "reading standard input:", err)
    }
}

func main() {
    cmd := exec.Command("/Users/rfay/bin/junk.sh")
    out, err := cmd.StdoutPipe()

    err = cmd.Start()
    if err != nil {
        fmt.Fprintf(os.Stderr, "Failed to start err=%v", err)
        os.Exit(1)
    }

    scanner := bufio.NewScanner(out)
    fmt.Println("Scanner created")

    defer cmd.Wait()

    go readStuff(scanner)
}

This is the junk.sh I used for testing.

#!/bin/bash
for i in `seq 1 10`;
do
  echo $i
  sleep 1
done    

Rotary

Answer 4

Ran into this issues today. Simplest solution i can up with is just setting the Stdout yourself:

cmd := exec.Command("program")
stdout := bytes.NewBuffer([]byte{})
cmd.Stdout = stdout

After execution of the subcommand you can just use the stdout to read the output.

// read stdout as string
stdout.String()
// or as bytes
stdout.Bytes()

Deportation