Bash: How to persist and restore associative arrays with keys that contain square brackets or other special characters

Question

Problem

Sourcing the result of declare -p for a valid Bash associative array in which keys contain square brackets results in a bad array subscript error.

Testing Procedure

Do:

$ declare -A array
$ key='var[0]'
$ array["$key"]=37
$ echo ${array["$key"]}
37
$ declare -p array > def.sh
$ cat def.sh
declare -A array='(["var[0]"]="37" )'
$ . def.sh
bash: [var[0]]=37: bad array subscript

In the above code, note:

• I am able to specify a key that contains square brackets: var[0]
• The key is quoted for setters and getters
• I am able to do an assignment using this key
• I am able to get the value from the associative array using this key
• Using declare -p I am able to save this definition to a file: def.sh
• When sourcing the file def.sh an error is emitted.

My Environment

• The version of Bash I'm using is 4.2.46(1)-release (x86_64-redhat-linux-gnu).
• I am on a CentOS 7.3.1611 (Core) server

Workarounds

If instead of doing declare -p array > def.sh I do instead:

{
echo 'declare -A array'
for Key in "${!array[@]}"; do
   EscapedKey="$(sed 's|"|\\"|g' <<<"$Key")"
   echo "array[\"$EscapedKey\"]=${array["$Key"]}"
done
} > def.sh

then sourcing the def.sh file works. Note that in the above example, I'm also escaping quote characters that might be a part of the key. I do understand that what I have above is not exhaustive. Because of these complications, I would prefer a solution that doesn't involve such workarounds, if at all possible.

Question

Is there some shopt,set -o <option>, or something else I can do to enable me to persist an associative array whose keys may contain square brackets or other special characters to a file and to later be able to source that file successfully? I am needing a solution that works in my environment above.

Vegetarianism

An improvement on the above workaround might include the use of printf using %q. Again, I am hoping for a solution involving declare -p and perhaps some behavior-changing commands; however, if this turns out to be just a bug in Bash 4.2.26(1)-release, then my goal will turn towards a strong workaround solution, perhaps an improvement on what I've shown above. – Vegetarianism

I just tested it on an Arch Linux installation with bash 4.4 and there is no such problem - I can use the string just like you intend, no workaround required. I'm trying to find some reference for all changes between 4.2 and 4.4 to see if this was fixed or if later versions just changed behavior. – Butlery

Ok, I'm not sure this is related, but I have found a slight hint it might have been a bug. I got the sources from ftp.gnu.org/gnu/bash - in the archive for bash 4.4's source there's the complete changelog under CHANGES. Line 2370: "e. Fixed several bugs encountered when reading subscripts in associative array assignments and expansions. " - this may be it, but using shopt -s compat42 didn't seem to have any effect in my environment so I'm not sure how to test without compiling bash anew. – Butlery

This was under the bash 4.0 release changes, there's another one: 2559: "h. When displaying associative arrays, subscripts are now quoted.". What I also noticed is that under my environment, declare wrote the same string to the file except the right-hand value was not in single quotes. Everything else is the same. – Butlery

Ugh, nevermind the above comments, those changes are before 4.2. There seems to be an option called complete_fullquote but it has no effect on this for me. In the changes I found "q. The declare builtin no longer displays array variables using the compound assignment syntax with quotes; that will generate warnings when re-used as input, and isn't necessary. " under the bash 4.3 release notes though. – Butlery

@MechaLynx Thanks for checking. It is motivation to upgrade to Bash 4.4. It's a pity that the current CentOS 7.3.1611 distro is still on 4.2.26(1)-release. It's a bureaucratic issue for me to update outside of the updates repo. – Vegetarianism

I went ahead and started compiling the versions I found on gnu.org, so I'll be able to tell you a bit more in a few minutes. It looks like earlier bash was greedy in matching quotes, while later on it's not so greedy, hence the issue. – Butlery

In case you miss it, I added an alternative workaround you might prefer. – Butlery

As a final comment on this and a suggestion to anyone that has similar constraints and specifically needs to store key/value pairs en masse, see if your system has sqlite3, which would let you comfortably store data of this sort and fetch it as well. Depends on what kind of data you have and the scale of course, but for some it might be a far better alternative than working with bash. – Butlery

Answer 1

It's a bug

This is a bug in bash 4.2. It's fixed in 4.3.

I tested this by compiling bash 4.2, 4.2.53 and 4.3 from http://ftp.gnu.org/gnu/bash/ and replicated the steps above. 4.3 behaves like 4.4 - there is no such issue. In bash 4.3 however, declare will print

declare -A array='(["var[0]"]="37" )'

just as 4.2 does. 4.4 does not add the quotes around the right-hand side, instead printing this:

declare -A array=(["var[0]"]="37" )

This makes no difference from what the testing showed.

There a possibly related option in complete_fullquote but it was added in 4.4 so it can't be used as a workaround.

It seems that outside of using a version >=4.3 this needs to be worked around and the one you used is the most straightforward way of doing it.

A workaround

There is an alternative if you want to avoid the sed calls though (tested using bash 4.2):

function array2file {
  # local variable for the keys
  declare -a keys

  # check if the array exists, to protect against injection
  # by passing a crafted string
  declare -p "$1" >/dev/null || return 1;

  printf "declare -A %s\n" "$1"

  # create a string with all the keys so we can iterate
  # because we can't use eval at for's declaration
  # we do it this way to allow for spaces in the keys, since that's valid
  eval "keys=(\"\${!$1[@]}\")"

  for k in "${keys[@]}"
  do
    printf "%s[\"${k//\"/\\\\\"}\"]=" "$1"
    # the extra quoting here protects against spaces
    # within the element's value - injection doesn't work here
    # but we still need to make sure there's consistency
    eval "printf \"\\\"%s\\\"\n\" \"\${$1[\"${k//\"/\\\"}\"]}\""
  done
}

This will properly add quotes around the key and also escape all doublequotes within the key itself. You can place this in a file, which you source. Then use:

array2file array > ./def.sh

where array is whatever array name you've chosen. By redirecting the output you'll get properly quoted keys and you can define your associative array as you did before, then pass it to this function for storage.

Extra credit

If you change the variable provided to the first printf inside the for loop from $1 to ${2:-$1} and do the same at the printf at the top, then you can optionally create the definition of a new array with the 2nd argument as its name, allowing renaming of sorts. This will only happen if you provide 2 strings instead of one (quoted of course). The setup allows for this to be done easily, so I've added it here.

This would let you work around cases where interfacing with existing code can be difficult with a predefined function.

Butlery

Thank you very much for your thoughtful solution. I am going to incorporate it into the open-source Bash scripting platform I'm working on at eggsh.com. Upvoting with my thanks! – Vegetarianism

@SteveAmerige np, but can you explain the edit? I've tested both the previous version and this one using a test array under bash 4.2 and they seem to function identically - I'm curious about whether you encountered some issue. By the way, I accidentally turned a parenthesis into a bracket while writing the code in the post, it has been corrected. – Butlery

@SteveAmerige nevermind, saw the comment on your edit. However, the issue still remains, they function identically on test strings, including the one you had in the comment (which gets turned into array["0"]="["com.acme".var[0]]=73" that strangely works, but works with both). I omitted escaping all kinds of special characters because they can be treated in the same way quotes are, whereas the issues with spaces and injection were more complicated and subtle. – Butlery