Word splitting bash parameter on whitespace respecting and retaining quotes

Question

Given the bash parameter

foo='ab   "cd" "e f"  x="1 2" '

I wish to produce an array equivalent to

foo_transformed=( ab '"cd"' '"e f"' 'x="1 2"' )

in a portable way, meaning using either bash (v3+) builtins or programs available to most operating systems (Linux, Unix, Cygwin) by default. Simplicity and safety, given (almost) arbitrary input string, are desirable.

You may assume the input does not contain single quotes ' or backslashes \, but may contain an arbitrary number of whitespace characters, both where I wish them to delimit the string and where they do not (when inside double quotes).

If we try:

foo_transformed=( $foo )

then the internal quotes of foo are not respected (for k in "${foo_transformed[@]}"; do echo "- $k"; done):

- ab
- "cd"
- "e
- f"
- x="1
- 2"

If we try:

eval foo_transformed=( $foo )

then the quotes are lost:

- ab
- cd
- e f
- x=1 2

Shannashannah

You might look at the CSV option in GNU awk or Ruby or Python or Perl or mlr. – Ablation

@Ablation my OS requirements were vague, but I do not have GNU awk, Ruby or mlr available universally (AIX). Python may be problematic because some systems will have version 3 installed by default (macOS) and some version 2, but it could work. Although perhaps not the simplest solution. – Shannashannah

It's odd that you want the double quotes be interpreted as meta characters (affecting the significance of subsequent whitespace) but you also want to keep them as part of the actual data. Otherwise you could have just constructed and evaluated a simple array declaration, e.g. eval "declare -a foo_transformed=($foo)". – Dressel

@Dressel Indeed. But I am dealing with legacy systems that had hacky solutions, and these are my requirements, unfortunately. – Shannashannah

Answer 1

This might be what you want:

$ cat tst.sh
#!/usr/bin/env bash

foo='   ab   "cd"
"e f"  '\'' x="1 2"
   z="a b"7"c
 d"8   $HOME
          `date`  *  $(date)   '

fpat='(^[[:space:]]*)([^[:space:]]+|([^[:space:]"]*"([^"]|"")*"[^[:space:]"]*)+)'

foo_transformed=()
while [[ "$foo" =~ $fpat ]]; do
    foo_transformed+=( "${BASH_REMATCH[2]}" )
    foo="${foo:${#BASH_REMATCH[0]}}"
done

declare -p foo_transformed

$ ./tst.sh
declare -a foo_transformed=([0]="ab" [1]="\"cd\"" [2]="\"e f\"" [3]="'" [4]="x=\"1 2\"" [5]=$'z="a b"7"c\n d"8' [6]="\$HOME" [7]="\`date\`" [8]="*" [9]="\$(date)")

I gave foo some extra values including globbing chars, a single quote, potential variable references, old and new style command injections, newlines inside and outside of quoted strings, and strings containing multiple quoted substrings so it could test the script more fully.

The use of the fpat regexp above is inspired by GNU awk's FPAT which is used to identify fields in input, see for example how it's used in CSVs at What's the most robust way to efficiently parse CSV using awk?. It matches:

• (^[[:space:]]*) - an optional leading sequence of spaces (which are discarded) followed by (...) which matches the string we actually want to capture:
• [^[:space:]]+ - a series of non-spaces
• | - or
• ([^[:space:]"]*"([^"]|"")*"[^[:space:]"]*)+ - a repeated string of double quoted substrings containing any chars, or no chars, optionally surrounded by substrings that don't contain spaces or double quotes.

Vining

I think that looks like a complete answer to me. An unmatched double quote will be interpreted as forming its own word, but I'm happy to let that cause an error later on. So thank you very much! – Shannashannah

Answer 2

re=$'[ \t]*(([^ \t"]+|"[^"]*")+)'
s=$foo
foo_transformed=()

while [[ $s =~ $re ]]; do
    foo_transformed+=("${BASH_REMATCH[1]}")
    s=${s#"${BASH_REMATCH[0]}"}
done

The regex allows foo to contains embedded newlines between double-quotes. To correctly handle other newlines, replace both instances of \t with \t\n or perhaps [:space:].

(Bash regex doesn't accept \t so I use $'...\t...' syntax to convert into literal tabs/newlines.)

---

As noted in comments, that will hang on malformed input.

To accept malformed input, this modified regex could be used:

re=$'[ \t]*(([^ \t"]+|"[^"]*("|$))+)'

Alternatively, a pre-check could be done:

malformed='^([^"]+|"[^"]*")*"[^"]*$'
if [[ $foo =~ $malformed ]]; then
    : ...
fi

Methylnaphthalene

Perfect. Thank you very much. I had completely neglected =~ actually, but unless anyone has a simpler solution, I think that (or variations thereof) will do nicely. Thank you. – Shannashannah

Great solution! Just want to point out, that for the (somewhat pathological) input of s='"a"b"c', this seems to be stuck in the loop. Maybe it would make sense to break out of the loop after the ${#foo}-th iteration with the error message malformed input. – Isfahan

Answer 3

You could use a string replacement within a declare statement that does exactly what you want in one go:

#!/usr/bin/env bash

# Init
foo='ab   "cd" "e f"  x="1 2" * $(date) `date` $[1+1] $((1+1)) ((1+1)) foo=''"''$HOME''" bar="a '\'' b"'

# If the patsub_replacement shell option is enabled using shopt,
# any unquoted instances of & in string are replaced with the
# matching portion of pattern.
shopt -s patsub_replacement

# Escape arithmetic variables, commands and glob expansions
foo_escaped=${foo//[()\*\?\\\$\`\']/\\&}

# Escape and preserve double-quotes as string delimiters
foo_escaped=${foo_escaped//\"/\"\\\"}

# Transform
declare -a foo_transformed="($foo_escaped)"

# Debug declaration
declare -p foo_transformed

# Debug print
printf %s\\n "${foo_transformed[@]}"

Detailed explanation:

• ${foo//\"/\"\\\"} replaces each double-quote " in foo by "\", so it remains quoted while containing the quote itself to be retained.

• declare -a foo_transformed="(${foo//\"/\"\\\"})", uses the transformed string within an array declaration declare -a variable="(string)".

EDIT

Added escape of all expansions

Ouster

watch out for glob and other expansions: foo='* $(date)' – Methylnaphthalene

@Methylnaphthalene You are absolutely right. I then added an escape for all those expansions. – Ardellardella

also backticks, and possibly arithmetic contexts – Methylnaphthalene

Added escape for $ backtick and check everything is secured including arithmetic expansion and its legacy $[1+1] syntax as well. – Ardellardella

That's another idea that did not occur to me, thank you! But am I missing a shell option? When I try the above the first escapes are replaced by a literal ampersand – Shannashannah

@Shannashannah Possibly the Bash version involved is too old and does not have the pattern capture & feature. Using Bash 5.2.21 here – Ardellardella

GNU bash, version 5.0.17. Would you mind telling me the output of shopt | grep '\<on\>' | awk '{ print $1 }' and shopt -o | grep '\<on\>' | awk '{ print $1 }' and echo "$-" on your system please. – Shannashannah

@Shannashannah checkwinsize cmdhist complete_fullquote extquote force_fignore globasciiranges globskipdots hostcomplete interactive_comments patsub_replacement progcomp promptvars sourcepath and braceexpand hashall interactive-comments The option that makes it work is patsub_replacement – Ardellardella

Ah, that explains it then. That was added in 5.2 apparently (see also: unix.stackexchange.com/questions/733148/…). That rules this out for me unfortunately, but it was otherwise a very clean solution and without any explicit looping. Thank you anyway. – Shannashannah