Convert multiline string to array [duplicate]

Question

I have this script:

nmapout=`sudo nmap -sP 10.0.0.0/24`
names=`echo "$nmapout" | grep "MAC" | grep -o '(.\+)'`
echo "$names"

now the $names variable contains strings delimited with newlines:

>_
 (Netgear)
 (Hon Hai Precision Ind. Co.)
 (Apple)

I tried to do the array conversion with the sub-string approach:

names=(${names//\\n/ })
echo "${names[@]}"

But the problem is that I can't access them by indexing (i.e., ${names[$i] etc.), if I run this loop

for (( i=0; i<${#names[@]}; i++ ))
do
     echo "$i: ${names[$i]"
     # do some processing with ${names[$i]}
done

I get this output:

>_
 0: (Netgear)
 1: (Hon
 2: Hai

but what I want is:

>_
 0: (Netgear)
 1: (Hon Hai Precision Ind. Co.)
 2: (Apple)

I could not figure out a good way to do this, please note that the second string has spaces in it.

Melleta

Any reason why do you want array? I would prefer to use read by line loop. – Perfumery

@Perfumery , actually this is a small portion of a large script, the original script uses the index for other purposes, that's why I want to keep the array. – Melleta

Related: how to convert a space-delimited string to a bash array: Reading a delimited string into an array in Bash – Faiyum

Answer 1

Set IFS (Internal Field Separator). Shell uses the IFS variable to determine what the field separators are. By default, IFS is set to the space character. Change it to the newline character, as demonstrated below:

#!/bin/bash
names="Netgear
Hon Hai Precision Ind. Co.
Apple"
    
SAVEIFS=$IFS   # Save current IFS (Internal Field Separator)
IFS=$'\n'      # Change IFS to newline char
names=($names) # split the `names` string into an array by the same name
IFS=$SAVEIFS   # Restore original IFS

for (( i=0; i<${#names[@]}; i++ ))
do
    echo "$i: ${names[$i]}"
done

Output

0: Netgear
1: Hon Hai Precision Ind. Co.
2: Apple

Groundsel

Because there could be special characters in the original $IFS, it's better to avoid trying to store it. Better to just wrap the whole thing in a subshell with parentheses. – Montespan

@Montespan , I am new in bash scripting, could you please elaborate more on "subshell with parantheses"? – Melleta

@Melleta This manual explains all the detail. I'm suggesting it here to make use of the variable scope mentioned on that page. Basically if you put a command or commands in (parentheses), they act a bit like a new instance of bash. They inherit variables and settings from the running script, but usually don't propagate changes back. – Montespan

If i execute that as non-root, I get Syntax error: "(" unexpected – Trawler

you can change IFS just for one line using IFS=$'\n' names=(${names}) on line 9. It's the same as joining line 8 and line 9. – Bloomfield

@Trawler For me, that error was because I had #!/bin/sh instead of #!/bin/bash. my_arr=(...) is bash-specific array initialization. I bet that your root account worked because it used bash as the default shell. – Ahron

If you initialize an array with an unquoted variable, then the variable value is not only subject to word splitting, but also to pathname expansion. So, for example, in x='foo *'; arr=($x) the array contains foo and all filenames of the current working directory. – Tamekia

Actually @andres I could be wrong, but for me, in GNU bash, version 4.4.19, that IFS assignment changes the IFS for the rest of the script and not just 1 line. The IFS did however only affect one line when I did this: IFS=$'\n' read -ra names <<< "$names" – Bromleigh

It's probably worth noting in your answer that IFS = Internal Field Separator. Source: unix.stackexchange.com/questions/184863/… – Faiyum

A more common for loop line would look like this: for name in ${names_array[@]}; do ... done. – Faiyum

note if you're using zsh (all new mac versions) you have to do setopt sh_word_split to enable splitting – Snail

What does the dollar sign ($) do in IFS=$'\n'? Where is this documented? – Faiyum

@GabrielStaples man bash section QUOTING: "Words of the form $'string' are treated specially. The word expands to string, with backslash-escaped characters replaced as specified by the ANSI C standard." – Apocryphal

Answer 2

Bash also has a readarray builtin command, easily searchable in the man page. It uses newline (\n) as the default delimiter, and MAPFILE as the default array, so one can do just like so:

    names="Netgear
    Hon Hai Precision Ind. Co.
    Apple"

    readarray -t <<<$names

    printf "0: ${MAPFILE[0]}\n1: ${MAPFILE[1]}\n2: ${MAPFILE[2]}\n"

The -t option removes the delimiter ('\n'), so that it can be explicitly added in printf. The output is:

    0: Netgear
    1: Hon Hai Precision Ind. Co.
    2: Apple

Yoon

This is the correct answer to the question that was asked. readarray is designed to do exactly this – Paschal

This indeed is the correct answer to the specific question. – Waldon

readarray was introduced in bash v4.0. Some systems like macOS <11.* are still on bash v3.2. In that case IFS-based solutions can used instead. – Kinlaw

Answer 3

Let me contribute to Sanket Parmar's answer. If you can extract string splitting and processing into a separate function, there is no need to save and restore $IFS — use local instead:

#!/bin/bash

function print_with_line_numbers {
    local IFS=$'\n'
    local lines=($1)
    local i
    for (( i=0; i<${#lines[@]}; i++ )) ; do
        echo "$i: ${lines[$i]}"
    done
}

names="Netgear
Hon Hai Precision Ind. Co.
Apple"

print_with_line_numbers "$names"

See also:

• Setting IFS for a single statement

Albuminate

Answer 4

How to read a multi-line string into a regular bash "indexed" array

The Bash shellcheck static code analyzer and checker tool recommends in SC2206 to use read -r or mapfile. Their mapfile example is complete, but their read example only covers the case of splitting a string by spaces, not newlines, so I learned the complete form of the read command for this purpose from @Toni Dietze's comment here.

So, here is how to use both to split a string by newlines. Note that <<< is called a "herestring". It is similar to << which is a "heredoc", and < which reads in a file:

# split the multiline string stored in variable `var` by newlines, and
# store it into array `myarray`

# Option 1
# - this technique will KEEP empty lines as elements in the array!
# ie: you may end up with some elements being **empty strings**!
mapfile -t myarray <<< "$multiline_string"

# OR: Option 2 [my preference]
# - this technique will NOT keep empty lines as elements in the array!
# ie: you will NOT end up with any elements which are empty strings!
IFS=$'\n' read -r -d '' -a myarray <<< "$multiline_string"

There is also a 3rd technique I use the most, which is not necessarily recommended by shellcheck, but which is fine if you use it correctly, and which is far more readable than either of the options above. I use it in many scripts in my eRCaGuy_dotfiles/useful_scripts directory here. Clone that repo and run grep -rn "IFS" in it to find all places where I use that technique.

See here for where I first learned this: Answer here by @Sanket Parmar: Convert multiline string to array.

Here it is:

# Option 3 [not necessarily recommended by shellcheck perhaps, since you must
# NOT use quotes around the right-hand variable, but it is **much
# easier to read**, and one I very commonly use!]
#
# Convert any multi-line string to an "indexed array" of elements:
#
# See:
# 1. "eRCaGuy_dotfiles/useful_scripts/find_and_replace.sh" for an example 
#    of this.
# 1. *****where I first learned it: https://mcmap.net/q/12399/-convert-multiline-string-to-array-duplicate
SAVEIFS=$IFS   # Save current IFS (Internal Field Separator).
IFS=$'\n'      # Change IFS (Internal Field Separator) to the newline char.
# Split a long string into a bash "indexed array" (via the parenthesis),
# separating by IFS (newline chars); notice that you must intentionally NOT use
# quotes around the parenthesis and variable here for this to work!
myarray=($multiline_string) 
IFS=$SAVEIFS   # Restore IFS

See also:

1. Where I learned my "Option 3" above: Answer here by @Sanket Parmar: Convert multiline string to array
2. Read file into array with empty lines
3. An example where I read a bash multi-line string into a bash array using the read cmd: Find all files in a directory that are not directories themselves

Faiyum

Answer 5

As others said, IFS will help you.IFS=$'\n' read -ra array <<< "$names" if your variable has string with spaces, put it between double quotes. Now you can easily take all values in a array by ${array[@]}

Rad

By default, read uses \n as delimiter, so you have to put -d '' in the read command, otherwise the array only contains the first line of $names. Corrected version: IFS=$'\n' read -r -d '' -a array <<< "$names". You also forgot to put a $ in front the {. – Tamekia

I am new to this, Could you elaborate more about -r and -a usage in this command – Rad

I am a bit confused. You already use -r and -a in your initial answer, just shortened to -ra. In my comment, I added -d ''. The bash man page nicely explains all these command line options (look for the read builtin command). – Tamekia

@ToniDietze, thanks for your corrections! I never would have figured out to add -d '' otherwise, and that part is essential. I added it to my answer here. – Faiyum

It's worth mentioning that the read builtin will return a non-zero exit status upon encountering an EOF, so if you have set -e somewhere in your shell script as the Bash Strict Mode document suggests, you better mask the exit code of read, e. g.: read -ra array -d '' <<< "${names}" || true. – Cross

Answer 6

^{Adding the needed null byte delimiter in @HariBharathi answer}

#!/bin/bash

IFS=$'\n' read -r -d '' -a array <<< "$names"

Remark: Unlike mapfile/readarray, this one is compatible with macOS bash 3.2

Schuler