Zsh read output of command into array splitting on newline

Question

I have a command that outputs a bunch of stuff e.g. running mycmd will give:

foobar
derp derp
like so
etc

Some of these lines will have spaces in them.

How do I read these into an array in zsh such that ${arr[1]} gives foobar, ${arr[2]} gives derp derp etc.

I have tried something like but it seems to split the array on chars not newlines.

IFS=$'\n' read -d '' -r arr <<< "$(mycmd)"

i.e. ${arr[1]} gives f when it should give foobar

Expertize

Answer 1

Okay its actually very simple:

IFS=$'\n' arr=($(mycmd))

Expertize

This command will affect the IFS of the following commands. Is there a way to prevent it? – Remde

You can just set it back to its default afterward IFS=$' \t\n' – Expertize

Thanks. I got another approach, which is arr=(${(f)"$(mycmd)"}) or in two steps tmp=$(mycmd), arr=(${(f)tmp}) – Remde

@LiuSha That's not work for me. I want create a array for output of ps -aux | grep python, your suggestion failed. – Valerivaleria

@Valerivaleria I have no idea. did you try other simpler commands like ls? The double quote is essential. Maybe there is some zsh option controlling this. – Remde

@Remde the oneline for arr=(${(f)"$(mycmd)"}) works for me, thank you. One thing I found with that is that mycmd had to be the actual command and not the variable output stored as $mycmd. I am curious about the breakdown of what each part does. Thank you. – Cockrell

@keith-e-truesdell: Working from the inside out: $(mycmd) - Use a "command substitution", which expands to the output of mycmd. "..." - Double-quote the command substitution, which prevents its output from being split into words on whitespace, preserving any extra spaces. ${(f)...} - Apply the "parameter expansion flag" f to the command substitution, splitting its output into words using newline (\n) as the separator. arr=(...) - Set an array parameter arr, using the separated output lines as the elements of the array. See man zshexpn for more details. – Molal

Awesome little trick, @Remde ! Works in zsh, too. I use shell autocomplete a lot to reuse commands that do batch processing, and having to use IFS= in the beginning of such commands makes autocomplete more cumbersome (I have to use Ctrl+R instead of simply typing, e.g., bz and pressing arrow up. Your command helps with that, and it has the added benefit of not messing with IFS in case the command breaks or I opt to kill it. – Gazehound

Answer 2

I'm not sure exactly why the read usage in the original question didn't work. It's possibly related to mixing <<< and $(). Or maybe the user just had a messed up shell session. Or maybe it was a bug in an older version of Zsh.

In any case, it has nothing to do with the behavior of the read builtin, and the original proposal was very close to correct. The only problem was using <<< $(...) instead of a plain pipe, which should just be a stylistic goof (rather than an error).

The following works perfectly fine in Zsh 5.8.1 and 5.9:

function mycmd {
  print foobar
  print derp derp
  print like so
  print etc
}

typeset -a lines
mycmd | IFS=$'\n' read -r -d '' -A lines

echo ${(F)lines}

You should see:

foobar
derp derp
like so
etc

I prefer this style, instead of ( $(...) ). Not requiring a subshell is useful in many cases, and the quoting/escaping situation is a lot simpler.

Note that -d '' is required to prevent read from terminating at the first newline.

You can wrap this up in a function easily:

function read-lines {
    if (( $# != 1 )); then
        print -u2 'Exactly 1 argument is required.'
        return 2
    fi

    local array="${1:-}"
    read -r -d '' "$array"
}

Poetics

There is a gotcha, because now echo $#lines unexpectedly prints 5, and ${lines[5]-unset} expands to blank instead of unset. why does the array have a fifth blank element? it's because the final newline in the input, when there is one, is treated as a field *separator*, not a record *terminator*, when IFS=$'\n'. how you handle this is up to you, but it can be a real landmine for subsequent iteration. – Carboxylase