How to expand shell variables in a text file?

Question

Consider a ASCII text file (lets say it contains code of a non-shell scripting language):

Text_File.msh:

spool on to '$LOG_FILE_PATH/logfile.log';
login 'username' 'password';
....

Now if this were a shell script I could run it as $ sh Text_File.msh and the shell would automatically expand the variables. What I want to do is have shell expand these variables and then create a new file as Text_File_expanded.msh as follows:

Text_File_expanded.msh:

spool on to '/expanded/path/of/the/log/file/../logfile.log';
login 'username' 'password';
....

---

Consider:

$ a=123
$ echo "$a"
123

So technically this should do the trick:

$ echo "`cat Text_File.msh`" > Text_File_expanded.msh

...but it doesn't work as expected and the output-file while is identical to the source.

So I am unsure how to achieve this.. My goal is make it easier to maintain the directory paths embedded within my non-shell scripts. These scripts cannot contain any UNIX code as it is not compiled by the UNIX shell.

Libeler

Will your variables be within single quotes as shown above? – Engrossment

@Engrossment - Currently, it is in single quotes or no surrounding quotes at all in the text file. – Libeler

related question: superuser.com/q/235738/126693 – Submicroscopic

Answer 1

This solution is not elegant, but it works. Create a script call shell_expansion.sh:

echo 'cat <<END_OF_TEXT' >  temp.sh
cat "$1"                 >> temp.sh
echo 'END_OF_TEXT'       >> temp.sh
bash temp.sh >> "$2"
rm temp.sh

You can then invoke this script as followed:

bash shell_expansion.sh Text_File.msh Text_File_expanded.msh

Nahamas

Thanks @Hai Vu. I first set the variable $ LOG_FILE_PATH=/some/path then ran $ ./shell_expansion.sh Text_File.msh Text_File_expanded.msh (and also $ bash shell_expansion.sh Text_File.msh Text_File_expanded.msh) - but in the Text_File_expanded.msh the $LOG_FILE_PATH entries are replaced with blanks spaces. I may be doing this incorrectly.. – Libeler

I had not exported the variables.. After I tried $ export LOG_FILE_PATH=/sone/path/ it worked fine. Thanks! :) – Libeler

There is no need to use temporary files, just use eval - see @ThirteenThirtySeven's answer. Also the method of using temporary files is somewhat problematic (multiple instances of shell_expansion.sh will write all to the same file, permission issues, error checking etc - temporary file will create many potential complications you need to be prepared to handle or know to avoid with certainty). – Multifarious

eval is more dangerous than using temp files. – Cahilly

I find that echo -e "cat <<END_OF_TEXT\n$(cat [filename])\nEND_OF_TEXT" | bash works well without a temp file. – Lianeliang

Thanks, @Lianeliang – Nahamas

Nice one @JacobTDC. Works perfectly. Watch out for usage of $$ with procedural code, and also don't include the text END_OF_TEXT ;-) – Slipsheet

@Cahilly How is eval more dangerous that using a temp file? I disagree with the assertion. If the tempfile is kept it makes it easier to do a post-mortem to determine what went wrong, but it provides no additional safety over eval. – Anthracoid

@Lianeliang is not the same thing. This short syntax interpret all escape characters inside the document and not only the two linefeeds, producing potentially unwanted results. – Requisite

Eval is fine but try file text with secret=pa\$\$ata – Lowder

Answer 2

This question has been asked in another thread, and this is the best answer IMO:

export LOG_FILE_PATH=/expanded/path/of/the/log/file/../logfile.log
cat Text_File.msh | envsubst > Text_File_expanded.msh

if on Mac, install gettext first: brew install gettext

see: Forcing bash to expand variables in a string loaded from a file

Novocaine

2 things to remember with envsubst is that environment variables must be exported for it to work and it isn't preinstalled on a lot of systems. If on Debian/Ubuntu: sudo apt-get install gettext – Separator

Answer 3

If you want it in one line (I'm not a bash expert so there may be caveats to this but it works everywhere I've tried it):

when test.txt contains

${line1}
${line2}

then:

>line1=fark
>line2=fork
>value=$(eval "echo \"$(cat test.txt)\"")
>echo "$value"
line1 says fark
line2 says fork

Obviously if you just want to print it you can take out the extra value=$() and echo "$value".

Sulphathiazole

If test.txt contains, for example, "; touch /etc/your_system_is_pwned; echo " you will see Bad Things happen. – Xenophobe

Thank you this is awesome – Trometer

Answer 4

If a Perl solution is ok for you:

Sample file:

$ cat file.sh
spool on to '$HOME/logfile.log';
login 'username' 'password';

Solution:

$ perl -pe 's/\$(\w+)/$ENV{$1}/g' file.sh
spool on to '/home/user/logfile.log';
login 'username' 'password';

Engrossment

Thanks @Engrossment - I tried it and it works fine.. As I need to avoid a dependency on Perl I cannot apply this solution. I am not a Perl expert but are we sure that \w will always capture the shell variable name correctly..? – Libeler

Ok I goggled and got my answer - 1. PERL reference: \w will Match "word" character (alphanumeric plus ""). 2. UNIX shell reference: The name of a variable can contain only alphanumeric characters and "". – Libeler

Hi All - Typos in my above comment: the "_" was taken as not marked as code and so was used to make the text italics.. Sorry. – Libeler

I tried your script . it works almost fine. It will replace indeed all env variables regardless they are defined or not. I think it would be useful to avoid replacement of variables which are not defined. – Samala

Answer 5

One limitation of the above answers is that they both require the variables to be exported to the environment. Here's what i came up with that would allow the variables to be local to the current shell script:

#!/bin/sh
FOO=bar;
FILE=`mktemp`; # Let the shell create a temporary file
trap 'rm -f $FILE' 0 1 2 3 15;   # Clean up the temporary file 

(
  echo 'cat <<END_OF_TEXT'
  cat "$@"
  echo 'END_OF_TEXT'
) > $FILE
. $FILE

The above example allows the variable $FOO to be substituted in the files named on the command line. I'm sure it can be improved, but this works for me so far.

Thanks to both previous answers for their ideas!

Kendall

Thanks @PaulGear! Welcome to StackOverflow. +1 For using trap to clean up temporary files. I have suggested some edits to make it easier for newbies like myself to understand the code; kindly review the edits and revert them if required. thanks – Libeler

I didn't get which signal the 0 refers to. – Libeler

0 signifies normal exit rather than a signal. – Kendall

I'm not sure why you think semis are necessary on variable assignments and traps but not echos and cats, but the comments are useful so i don't plan to revert. – Kendall

Thanks Paul. About the semis- just added them while adding the comments; not that its necessary, just a force of habit :) – Libeler

this solution will replace also variables which are not defined. Some times you would like to avoid that and keep variables placeholders. – Samala

Answer 6

If the variables you want to translate are known and limited in number, you can always do the translation yourself:

sed "s/\$LOG_FILE_PATH/$LOG_FILE_PATH/g" input > output

And also assuming the variable itself is already known

Andre

It won't work assuming $LOG_FILE_PATH contains slashes. It will be simply expanded inside the pattern causing an error like "bad flag in substitute command". Just tried it with $PATH. – Kobayashi

Answer 7

This solution allows you to keep the same formatting in the ouput file

Copy and paste the following lines in your script

cat $1 | while read line
do
  eval $line
  echo $line
  eval echo $line
done | uniq | grep -v '\$'

this will read the file passed as argument line by line, and then process to try and print each line twice: - once without substitution - once with substitution of the variables. then remove the duplicate lines then remove the lines containing visible variables ($)

Swop

Answer 8

Yes eval should be used carefully, but it provided me this simple oneliner for my problem. Below is an example using your filename:

eval "echo \"$(<Text_File.msh)\""

I use printf instead of echo for my own purposes, but that should do the trick. Thank you abyss.7 providing the link that solve my problem. Hope it helps.

Transpierce

Answer 9

Create an ascii file test.txt with the following content:

Try to replace this ${myTestVariable1} 
bla bla
....

Now create a file “sub.sed” containing variable names, eg

's,${myTestVariable1},'"${myTestVariable1}"',g;
s,${myTestVariable2},'"${myTestVariable2}"',g;
s,${myTestVariable3},'"${myTestVariable3}"',g;
s,${myTestVariable4},'"${myTestVariable4}"',g'

Open a terminal move to the folder containing test.txt and sub.sed.
Define the value of the varible to be replaced

myTestVariable1=SomeNewText

Now call sed to replace that variable

sed "$(eval echo $(cat sub.sed))" test.txt > test2.txt

The output will be

$cat test2.txt

Try to replace this SomeNewText 
bla bla
....

Thapsus

Answer 10

#logfiles.list:

$EAMSROOT/var/log/LinuxOSAgent.log
$EAMSROOT/var/log/PanacesServer.log
$EAMSROOT/var/log/PanacesStrutsGUI.log

#My Program:

cat logfiles.list | while read line
    do
        eval Eline=$line
        echo $Eline
    done

Octarchy