Delaying wildcard expansion in bash, while quoting for special characters

Question

I've written a bash script that needs to do something later. It's called something like this:

later mv *.log /somewhere/else

However, when called like this *.log is expanded at call time, and the script is called as if I wrote

later mv 1.log 2.log 3.log /somewhere/else

I'm trying to get my script to expand wildcards later. I tried calling it like this

later mv '*.log' /somewhere/else

and also with \*, but these both result in the wildcard never getting expanded at all.

How do I expand the wildcards in the commandline? And is there a way to prevent expansion when the script is called, i.e. get the original parameters as they were typed?

This is the part of my scripts that prepares the call for later:

tmpfile=$(mktemp)

### Get a quoted commandline
line=
while (( "$#" > 0 )); do
        line="$line\"$1\" "
        shift
done

### Prepare a script to be run
echo '#!/bin/bash' > "$tmpfile"
echo "cd $(pwd)" >> "$tmpfile"
echo "trap 'rm \"$tmpfile\"' EXIT" >> "$tmpfile"
echo "$line" >> "$tmpfile"
chmod 777 "$tmpfile"

Note that I have to quote the commandline as some of my files and folders have spaces in their names; if I remove the quoting bit, even bits without wildcards stop working.

Conto

There is never a good reason to set 777 permissions. Also, you should probably use a here document rather than all those echo statements with complex quoting. – Cheeky

@DennisWilliamson: you make good points. The 777 was due to laziness when writing the script. Donno why I didn't use a heredoc though, it seems obvious now. – Conto

Answer 1

Interesting question. I suspect that a better answer than mine is possible. Nevertheless, given

A='foo.*'

this seems to do what you want:

echo $(eval echo "$A")

Obsession

The eval is entirely unnecessary, and in this instance basically just removes the also unnecessary quotes. It should just be echo $A. – Zagreb

What if there is a space? foo bar.* – Batho

@JariTurkia Yes, if you understand this and have some time, then please feel free to edit my answer. – Obsession

@Obsession I did. My edit was apparently rejected. – Batho

@JariTurkia Regrettably, I am not as active on StackExchange as I was years ago. I do not know how to retrieve a rejected edit to un-reject it. Your edit was not rejected by me: I never saw it. Unfortunately, I lack time to pursue the matter further, so thanks for the attention, and good luck. – Obsession

@Obsession Yes, I kinda concluded that. The thing is, your answer is getting lot of upvotes while being inaccurate. See above comment about using eval to mention one of the shortcomings. – Batho

Answer 2

The simplest way may be to simply pass the wildcard (glob) back to the shell for expansion,

for example:

 #!/bin/bash
 echo "For the wildcard $1"
 echo "I find these match(es): $(eval echo "$1")"

The $( … ) operator calls a sub-shell, and interpolates the results; the echo command simply returns them.

• As @thb put in their answer, the eval is neccesary within a shell script (but not when keying it in on an interactive shell, in which case $(echo $1) works.

Note that you'll still have to escape spacing and the like, but it seems you've taken that into consideration. Naturally, the glob will need to be quoted when calling your script — this is why most Unix commands only take one list in the first or last position(s), although examples like find do exist that require escaped fileglobs, themselves.

PS: As for a program forcing the shell to not perform expansion: no, there's no built-in way to do so. The shell expands globs before (well, logically “before” if not necessarily chronologically) even identifying what file your program lives in, and is pretty well universally agnostic to what programs it runs.

Jumada

The problem is that this works only if the strings are not escaped. Suppose I call my script later rm "Log folder/*.log" or later rm Log\ Folder/\*.log. The script can then call $(echo Log folder/*.log) or $(echo "Log Folder/*.log") which are both wrong. I honestly expected there to be a simpler way than to mess around with the parameters myself! – Conto

Why is $(echo "Log Folder/*.log") “wrong” in this context? That would seem to be what you were looking-for… all files in “Log Folder” whose name ends in “.log” … do I misunderstand the question? – Jumada

The quotes prevent expansion - which is exactly my original problem. – Conto

That's why you pass it into a sub-shell. Try the version I put above, e.g. in my ~ I can run ./tmp 'M*' and get For the wildcard M* / I find these match(es): Music Movies – Jumada

Try this: mkdir a\ a; echo > a\ a/1; echo > a\ a/2; ./tmp "a a/*". This will fail because the string isn't quotes; if you do quote it it won't expand the matches. The subshell doesn't actually help here, and I don't see a way around it other than double-escaping when calling the script (as in ./tmp 'a\ a/1'), which would be a very odd way to use the script. – Conto

Actually, I just checked, and even double-escaping by calling ./tmp 'a\ a/*' or even ./tmp '"a a"/*' doesn't work. – Conto

As for the subshell not helping, changing the script to echo "I find these match(es):" $1 results in the same output as far as I can tell. – Conto

d'oh. Non-interactive shell requires the eval as @Obsession pointed out — I had keyed these in directly, in which context it worked. :-( sorry – Jumada

Answer 3

Solution

In shell scripting there are bunch of tricky characters. Space and both quotes are the most difficult to handle. This suggested solution will address the problem of tricky characters by applying proper quoting.

later-script:

#!/bin/bash

if [ $# -lt 1 ]; then
        echo "Need argument!"
        exit 1
fi

### Get a quoted commandline
line=
files=($(echo "$1"))
for file in "${files[@]}"; do
        echo "Debug: Processing file [$file]"
        quoted_filename=$(printf '%q' "$file")
        if [ -n "$line" ]; then
                line="$line "
        fi
        line="$line$quoted_filename"
done
echo -e "Result:\n$line"

Testing the solution

Prep:

Preparing test environment with four matching entries having potential pitfalls:

mkdir 'foo.bar.*'
date > 'foo.bar test.dat'
touch 'foo."nickname".bar'
touch "foo.'single'.bar"

Run:

Using suggested wildcard from question.

./later.sh 'foo*'

Example output:

Debug: Processing file [foo.bar.*]
Debug: Processing file [foo.bar test.dat]
Debug: Processing file [foo."nickname".bar]
Debug: Processing file [foo.'single'.bar]
Result:
foo.bar.\* foo.bar\ test.dat foo.\"nickname\".bar foo.\'single\'.bar

Batho