Bash: read a file line-by-line and process each segment as parameters to other prog

Question

I have some dirty work to do, so a Bash script seems to be a good choice. I'm new to Bash, and the experience makes me kind of frustrated.

The file mapfiles.txt consists of lines as follow. Each line has four segments separated by a white space. Each segment represents a input parameter to an external program name 'prog'. For example, "cm19_1.png" is the filename, "0001" the index, "121422481" the longitude, and "31035995" the latitude.

File: mapfiles.txt

cm19_1.png 0001 121422481 31035995
cm19_2.png 0002 121423224 31035995
cm19_3.png 0003 121423967 31035995
…

I want to execute similar commands to each line. As show below, the prog's input parameter order is slightly different. So it makes sense to write a bash script to handle the repeated work.

[Usage] prog <index> <longitude> <latitude> <filename>
example: prog 0001 121422481 31035995 cm19_1.png

Generally, the bash script will operate in this way:

1. Read one line from mapfiles.txt
2. Split the segments
3. Call the prog with a correct parameter order

Here comes run.sh.

#!/bin/sh

input=mapfiles.txt
cmd=prog

while read line
do
        file=$(echo $line | cut -d' ' -f1)
        key=$(echo $line | cut -d' ' -f2)
        log=$(echo $line | cut -d' ' -f3)
        lat=$(echo $line | cut -d' ' -f4)
        echo $cmd $key $log $lat $file
done < "$input"

What I expected is

prog 0001 121422481 31035995 cm19_1.png
prog 0002 121423224 31035995 cm19_2.png
prog 0003 121423967 31035995 cm19_3.png
… 

The ACTUAL result I got is

 cm19_1.png21422481 31035995
 cm19_2.png21423224 31035995
 cm19_3.png21423967 31035995

Problems that confused me

1. Where is 'prog'?
2. Where is the white space?
3. What's wrong with the parameter order?

Hmm… I wrote this script on my Mac using vim and copy it to a Scientific Linux box and a gentoo box. These three guys get the same ridiculous outputs.

Grimaldi

Not reproducible here. Look for typos. Also, instead of all the cuts use just read file key log lat. UPD see the answer. – Janinejanis

When combined with ssh in the loop, take a look at this question: #9393538 – Travistravus

Answer 1

You can simplify this a lot:

while read file key log lat
do
  echo "$cmd" "$key" "$log" "$lat" "$file"
done < "$input"

Revile

You can simplify it even further. You don't need individual variables for each argument: while read line; do echo $cmd $line; done < "$input" – Crosspollination

Answer 2

Using GNU Parallel you can do it in a single line + you get it done in parallel for free:

cat mapfile.txt | parallel --colsep '\s' prog {2} {3} {4} {1}

Watch the intro videos to learn more: http://www.youtube.com/watch?v=OpaiGYxkSuQ

Hallucinogen

Answer 3

prog could have disappeared because $cmd is not exported. Your version of /bin/sh might execute the while statement in a separate shell. This should not be the case, and this is not the case for my installation of bash, but perhaps yours behaves in interesting ways in this department.

UPD I see that you have several boxes that give the same results. This makes the subshell theory unlikely. Perhaps you have some funny characters in your script and/or source file.

I have copied and pasted your script and your source file to my gentoo box and it gives the expected results. Perhaps you should do the same, and compare the files with your original ones.

Janinejanis

Thank you so much. Your comment help me find out what REALLY goes wrong. It's not the Bash Script -- I test it on three different boxes -- it's the mapfiles.txt that goes wrong. The file is generated in Windows, ending with ^M at each line. After removing the ^M, everything goes well. – Grimaldi