Bash and filenames with spaces

Question

The following is a simple Bash command line:

grep -li 'regex' "filename with spaces" "filename"

No problems. Also the following works just fine:

grep -li 'regex' $(<listOfFiles.txt)

where listOfFiles.txt contains a list of filenames to be grepped, one filename per line.

The problem occurs when listOfFiles.txt contains filenames with embedded spaces. In all cases I've tried (see below), Bash splits the filenames at the spaces so, for example, a line in listOfFiles.txt containing a name like ./this is a file.xml ends up trying to run grep on each piece (./this, is, a and file.xml).

I thought I was a relatively advanced Bash user, but I cannot find a simple magic incantation to get this to work. Here are the things I've tried.

grep -li 'regex' `cat listOfFiles.txt`

Fails as described above (I didn't really expect this to work), so I thought I'd put quotes around each filename:

grep -li 'regex' `sed -e 's/.*/"&"/' listOfFiles.txt`

Bash interprets the quotes as part of the filename and gives "No such file or directory" for each file (and still splits the filenames with blanks)

for i in $(<listOfFiles.txt); do grep -li 'regex' "$i"; done

This fails as for the original attempt (that is, it behaves as if the quotes are ignored) and is very slow since it has to launch one 'grep' process per file instead of processing all files in one invocation.

The following works, but requires some careful double-escaping if the regular expression contains shell metacharacters:

eval grep -li 'regex' `sed -e 's/.*/"&"/' listOfFiles.txt`

Is this the only way to construct the command line so it will correctly handle filenames with spaces?

Polychaete

Answer 1

Try this:

(IFS=$'\n'; grep -li 'regex' $(<listOfFiles.txt))

IFS is the Internal Field Separator. Setting it to $'\n' tells Bash to use the newline character to delimit filenames. Its default value is $' \t\n' and can be printed using cat -etv <<<"$IFS".

Enclosing the script in parenthesis starts a subshell so that only commands within the parenthesis are affected by the custom IFS value.

Clance

Even if you don't export it, the value will persist if this wasn't run in a subshell. Just try running FOO=bar; echo $FOO on one line, then echo $FOO on another. Subshells are automatically started for commands in pipelines, but IFS=$'\n' is of course not part of a pipeline here. The best solution is to surround the whole statement with parentheses, which manually tell bash to run the command in a subshell. – Tintype

I still prefer running this in a subshell, but if you do want to save/restore IFS, you definitely want to quote the variable expansions. – Tintype

Another way to have a variable assignment only apply temporarily is to use a space between the variable assignment and the command it applies to: IFS=$'\n' grep -li 'regex' $(<listOfFiles.txt) – Unanswerable

@Dennis: I thought that as well, but for some reason that does not work on my machine. Perhaps that the scope of IFS=$'\n' is so limited in this case, that it doesn't even apply to $(<listOfFiles.txt)? – Clance

@Dennis: that doesn't work because the environment is only set after the processing of the argument list - so 'grep' sees the correct value of IFS, but the shell that is processing the argument list does not. – Kwangchow

Was the last edit, by "sweatybridge", correct? That is, the command line (with or without the $). – Odessaodetta

@PeterMortensen yes that change seems correct. What issue do you face with the latest change that didn't affect the penultimate revision? (I tested the expression like this: seq 0 9 | xargs -I{} sh -c 'echo {} > "{} {}.txt"; echo "{} {}.txt" >> listOfFiles.txt' && (IFS=$'\n'; grep -li '[0-5]' $(<listOfFiles.txt)). – Clance

Answer 2

cat listOfFiles.txt |tr '\n' '\0' |xargs -0 grep -li 'regex'

The -0 option on xargs tells xargs to use a null character rather than white space as a filename terminator. The tr command converts the incoming newlines to a null character.

This meets the OP's requirement that grep not be invoked multiple times. It has been my experience that for a large number of files avoiding the multiple invocations of grep improves performance considerably.

This scheme also avoids a bug in the OP's original method because his scheme will break where listOfFiles.txt contains a number of files that would exceed the buffer size for the commands. xargs knows about the maximum command size and will invoke grep multiple times to avoid that problem.

A related problem with using xargs and grep is that grep will prefix the output with the filename when invoked with multiple files. Because xargs invokes grep with multiple files one will receive output with the filename prefixed, but not for the case of one file in listOfFiles.txt or the case of multiple invocations where the last invocation contains one filename. To achieve consistent output add /dev/null to the grep command:

cat listOfFiles.txt |tr '\n' '\0' |xargs -0 grep -i 'regex' /dev/null

Note that was not an issue for the OP because he was using the -l option on grep; however it is likely to be an issue for others.

Mendive

Answer 3

This works:

while read file; do grep -li dtw "$file"; done < listOfFiles.txt

Unanswerable

This does not address the OP's concern that grep be invoked a minimal number of times to maintain performance. That is, multiple files on one command line. – Mendive

Answer 4

With Bash 4, you can also use the builtin mapfile function to set an array containing each line and iterate on this array:

$ tree
.
├── a
│   ├── a 1
│   └── a 2
├── b
│   ├── b 1
│   └── b 2
└── c
    ├── c 1
    └── c 2

3 directories, 6 files
$ mapfile -t files < <(find -type f)
$ for file in "${files[@]}"; do
> echo "file: $file"
> done
file: ./a/a 2
file: ./a/a 1
file: ./b/b 2
file: ./b/b 1
file: ./c/c 2
file: ./c/c 1

Vague

Answer 5

Though it may overmatch, this is my favorite solution:

grep -i 'regex' $(cat listOfFiles.txt | sed -e "s/ /?/g")

Smug

This is terrible! and, errr, useless use of cat spotted! – Taddeusz

Answer 6

Do note that if you somehow ended up with a list in a file which has Windows line endings, \r\n, NONE of the notes above about the input file separator $IFS (and quoting the argument) will work; so make sure that the line endings are correctly \n (I use scite to show the line endings, and easily change them from one to the other).

Also cat piped into while file read ... seems to work (apparently without need to set separators):

cat <(echo -e "AA AA\nBB BB") | while read file; do echo $file; done

... although for me it was more relevant for a "grep" through a directory with spaces in filenames:

grep -rlI 'search' "My Dir"/ | while read file; do echo $file; grep 'search\|else' "$ix"; done

Staffard