linux wildcard usage in cp and mv

Question

I am composing a script to process 20 files. All of them located in different directories. I have partial file name.

1. In log directory, File1_Date_time.err change to File1__Date_time_orig.err
2. cd ../scripts/
3. sh File.sh

File1 directory is /data/data1directory/Sample_File1/logs/File1_Data_time.err
File2 directory is /data/data2directory/Sample_File2/logs/File2_Data_time.err
.....

My script looks like this. (runrunrun.sh)

#!/bin/bash
INPUT=$1
mv /data/*/Sample_*/logs/*_Data_time.err /data/*/Sample_*/logs/*_Data_time_orig.err
cp /data/*/Sample_*/scripts/*.sh /data/*/Sample_*/scripts/*_orig.sh
sh /data/*/Sample_*/scripts/*_orig.sh

When running it, I tried.
./runrunrun.sh File1
. runrunrun.sh File1
sh runrunrun.sh File1

mv: cannot move /data/data1directory/Sample_File1/logs/File1_Data_time.err /data/*/Sample_*/logs/*_Data_time_orig.err: No such file or directory cp also got similar feedback

Am I doing it correct?

Thanks!

Spagyric

mv, cp, etc. aren't passed the wildcards you're writing, if those wildcards have any valid expansions. Instead, they're passed literal filenames created by running those expansions. Thus, since they don't know the literal wildcards used in the input name, they can't possibly map a * in the input name to a * in the output name. – Priscian

If you want to write renaming tools, a good place to start is BashFAQ #30: mywiki.wooledge.org/BashFAQ/030 – Priscian

Related: #20629802 – Priscian

BTW, sh foo.sh is very dangerous, unless you're absolutely certain that that script starts with #!/bin/sh rather than #!/bin/bash, #!/bin/ksh, #!/bin/zsh, etc; otherwise, it may use the wrong interpreter. – Priscian

Answer 1

Let's talk about how wildcards work for a minute.

cp *.txt foo

doesn't actually invoke cp with an argument *.txt, if any files matching that glob exist. Instead, it runs something like this:

cp a.txt b.txt c.txt foo

Similarly, something like

mv *.txt *.old

...can't possibly know what to do, because when it's invoked, what it sees is:

mv a.txt b.txt c.txt *.old

or, worse, if you already have a file named z.old, it'll see:

mv a.txt b.txt c.txt z.old

---

Thus, you need to use different tools. Consider:

# REPLACES: mv /data/*/Sample_*/logs/*_Data_time.err /data/*/Sample_*/logs/*_Data_time_orig.err
for f in /data/*/Sample_*/logs/*_Data_time.err; do
  mv "$f" "${f%_Data_time.err}_Data_time_orig.err"
done

# REPLACES: cp /data/*/Sample_*/scripts/*.sh /data/*/Sample_*/scripts/*_orig.sh
for f in /data/*/Sample_*/scripts/*.sh; do
  cp "$f" "${f%.sh}_orig.sh"
done

# REPLACES: sh /data/*/Sample_*/scripts/*_orig.sh
for f in /data/*/Sample_*/scripts/*_orig.sh; do
  if [[ -e "$f" ]]; then
    # honor the script's shebang and let it choose an interpreter to use
    "$f"
  else
    # script is not executable, assume POSIX sh (not bash, ksh, etc)
    sh "$f"
  fi
done

This uses a parameter expansion to strip off the tail end of the old name before adding the new name.

Priscian

This is not the case anymore as of 2020 with GNU bash 5.0 and above – Analisaanalise

@user164863, please provide a contrary demonstration. mv and cp are not part of bash itself, they're provided by your OS vendor, and globs are already expanded before they're invoked; none of that has changed. – Priscian

Answer 2

The find command can be used quite concisely in simple cases where you want to perform operations on wildcard (or more complex) filename matches. The technique below can be committed to memory ... almost !

This works by letting the find command run another command on each filename it finds. You can dry-run this example using echo instead of/in front of mv .

If we wanted to move all files in the current directory with name beginning 'report', to another parallel directory called 'reports' :

find . -name "report*.*" -exec mv '{}' ../reports/ \;

The wildcard string must be in quotes, the {} marking the filename that was 'found' must be in quotes, and the final semicolon must be escaped - all due to Bash/shell treatment of those characters.

Look at the man page for find for more uses: https://linux.die.net/man/1/find

Inundate

if the file names of multiple files are the same, this solution seems to overwrite the files in the destination. How can the entire folder structure be copied over? – Workbook

1st answer covers my case. I also used cp --parents ============================ for f in ./*/*20220202.csv; do cp -rfv --parents "$f" "/projects/drprojects/evmc/tempo4/" done – Workbook

The code was just an example. The original question also had that problem. Ask a separate question, or work it out ! – Inundate

Answer 3

This is what I use, very fast to write and remember

For copying:

ls -1 *.txt | xargs -L1 -I{} cp {} {}.old 

For moving:

ls -1 *.txt | xargs -L1 -I{} mv {} {}.old 

the explanation:

xargs(1) -L1 -I{}  

build and execute command lines from standard input

-L max-lines
       Use  at most max-lines nonblank input lines per command line.  Trailing blanks cause an input line
       to be logically continued on the next input line.  Implies -x.

-I replace-str
       Replace occurrences of replace-str in the initial-arguments with names read from  standard  input.
       Also,  unquoted  blanks  do  not  terminate  input  items;  instead  the  separator is the newline
       character.  Implies -x and -L 1.

Basically with -L1 it sends the arguments on by one, and -I{} makes {} the placeholder

Darg

Buggy with interesting filenames. In general, xargs is only safe with NUL-delimited input and the -0 argument; default behavior splits input on spaces, not just newlines (though newlines aren't safe either, as they're allowed in filenames); treats quote characters as syntax rather than literal data; and otherwise generally munges input. – Priscian

Also, see Why you shouldn't parse the output of ls – Priscian

Answer 4

using awk and sed

Filter using ls, use awk to build your move command for each record and the sed 'e' in the end will execute the command.

ls  *.avro  | awk ' { print "mv "$1" aa_"$1 }' | sed 'e'

First try without the sed -- once you know your command is showing correctly add the sed.

Gastropod

Answer 5

Try this:

Change these:

zc_cd_delim_01.csv
zc_cd_delim_04.csv

into these:

zc_cd_delim_113_01.csv
zc_cd_delim_113_04.csv

command line:

for f in zc_cd_delim_??.csv; do var1=${f%%??.*}; 
    mv $f "${var1}113_${f#$var1}"; 
done

Gon

mv "$f", rather -- if the ?? matches a filename with whitespace, or IFS contains a character found in the names (ie. _), behavior could otherwise be surprising. – Priscian

BTW, I'm a little unclear on how this is applicable to the question at hand -- c_cd_delim_??.csv doesn't look to me like one of the patterns the OP was using as an example of what they wanted to match. Would you mind editing to make applicability to the question more clear? – Priscian