Bash Templating: How to build configuration files from templates with Bash?

Question

I'm writing a script to automate creating configuration files for Apache and PHP for my own webserver. I don't want to use any GUIs like CPanel or ISPConfig.

I have some templates of Apache and PHP configuration files. Bash script needs to read templates, make variable substitution and output parsed templates into some folder. What is the best way to do that? I can think of several ways. Which one is the best or may be there are some better ways to do that? I want to do that in pure Bash (it's easy in PHP for example)

1. How to replace ${} placeholders in a text file?

template.txt:

The number is ${i}
The word is ${word}

script.sh:

#!/bin/sh

#set variables
i=1
word="dog"
#read in template one line at the time, and replace variables
#(more natural (and efficient) way, thanks to Jonathan Leffler)
while read line
do
    eval echo "$line"
done < "./template.txt"

BTW, how do I redirect output to external file here? Do I need to escape something if variables contain, say, quotes?

2. Using cat & sed for replacing each variable with its value:

Given template.txt (see above)

Command:

cat template.txt | sed -e "s/\${i}/1/" | sed -e "s/\${word}/dog/"

Seems bad to me because of the need to escape many different symbols and with many variables the line will be tooooo long.

Can you think of some other elegant and safe solution?

Bespread

Does this answer your question? How to replace ${} placeholders in a text file? – Derringdo

The "pure bash" requirement seems unnecessary if you do have PHP, a robust templating language, available. – Visitant

Answer 1

You can use this:

perl -p -i -e 's/\$\{([^}]+)\}/defined $ENV{$1} ? $ENV{$1} : $&/eg' < template.txt

to replace all ${...} strings with corresponding enviroment variables (do not forget to export them before running this script).

For pure bash this should work (assuming that variables do not contain ${...} strings):

#!/bin/bash
while read -r line ; do
    while [[ "$line" =~ (\$\{[a-zA-Z_][a-zA-Z_0-9]*\}) ]] ; do
        LHS=${BASH_REMATCH[1]}
        RHS="$(eval echo "\"$LHS\"")"
        line=${line//$LHS/$RHS}
    done
    echo "$line"
done

. Solution that does not hang if RHS references some variable that references itself:

#!/bin/bash
line="$(cat; echo -n a)"
end_offset=${#line}
while [[ "${line:0:$end_offset}" =~ (.*)(\$\{([a-zA-Z_][a-zA-Z_0-9]*)\})(.*) ]] ; do
    PRE="${BASH_REMATCH[1]}"
    POST="${BASH_REMATCH[4]}${line:$end_offset:${#line}}"
    VARNAME="${BASH_REMATCH[3]}"
    eval 'VARVAL="$'$VARNAME'"'
    line="$PRE$VARVAL$POST"
    end_offset=${#PRE}
done
echo -n "${line:0:-1}"

WARNING: I do not know a way to correctly handle input with NULs in bash or preserve the amount of trailing newlines. Last variant is presented as it is because shells “love” binary input:

1. read will interpret backslashes.
2. read -r will not interpret backslashes, but still will drop the last line if it does not end with a newline.
3. "$(…)" will strip as many trailing newlines as there are present, so I end … with ; echo -n a and use echo -n "${line:0:-1}": this drops the last character (which is a) and preserves as many trailing newlines as there was in the input (including no).

Unequaled

I would change [^}] to [A-Za-Z_][A-Za-z0-9_] in the bash version to prevent the shell from going beyond strict substitution (e.g. if it tried to process ${some_unused_var-$(rm -rf $HOME)}). – Mucin

Is it safe if environment variable contains some quotes or backslashes or something? I mean perl version. – Bespread

@FractalizeR yes, it is safe. It will just fail, because perl does not do bash substitution. It only tries to find variable name in %ENV hash. – Unequaled

@FractalizeR you may want to change $& in the perl solution to "": first leaves ${...} untouched if it failes to substitute, second replaces it with empty string. – Unequaled

NOTE: Apparently a there was a change from bash 3.1 to 3.2 (and up) in which the single quotes around the regex - treat the contents of the regex as a string literal. So the regex above should be... (\$\{[a-zA-Z_][a-zA-Z_0-9]*\}) #305364 – Goebel

@Rindeal Thanks, updated the answer. Missed Anthony Bouch’s comment for some reason. – Unequaled

I just found out that the BASH variant can go endless easily when $RHS contains $LHS - the quick&dirty solution is to check the line word by word like this – Syncarpous

For the Perl solution to perform in-place updating as implied by -i, the file must be passed directly as a filename operand, not via stdin; i.e.:template.txt, not < template.txt – Huesman

To make the while loop read the last line even if it's not terminated by a newline, use while read -r line || [[ -n $line ]]; do. Additionally, your read command strips leading and trailing whitespace from each line; to avoid that, use while IFS= read -r line || [[ -n $line ]]; do – Huesman

To read the entire input including trailing newlines, use IFS= read -d '' -r line instead of line="$(cat; echo -n a)". Since variable $line receives all input lines, please consider renaming it to $lines to avoid confusion. – Huesman

POST="${BASH_REMATCH[4]}${line:$end_offset:${#line}}" can be simplified to POST="${BASH_REMATCH[4]}${line:$end_offset}" (not specifying a length defaults to the remainder of the string). The adjacent lines eval 'VARVAL="$'$VARNAME'"' and line="$PRE$VARVAL$POST" can be replaced with single line line="$PRE${!VARNAME}$POST", thanks to bash's indirect expansion feature. – Huesman

Just to note a constraint for those looking for a comprehensive solution: These otherwise handy solutions do not allow you to selectively protect variable references from expansion (such as by \ -escaping them). – Huesman

Option -i to perl does not make sense due to using input redirection thus perl lacking name of file to adjust. – Austria

what is $& in perl? - entire matched string – Torbart

@Torbart This is a string matched by last successful match. You can find relevant documentation in perldoc perlvar. – Unequaled

Answer 2

Try envsubst

$ cat envsubst-template.txt
Variable FOO is (${FOO}).
Variable BAR is (${BAR}).

$ FOO=myfoo

$ BAR=mybar

$ export FOO BAR

$ cat envsubst-template.txt | envsubst
Variable FOO is (myfoo).
Variable BAR is (mybar).

Bibliophage

Just for reference, envsubst isn't required when using a heredoc since bash treats the heredoc as a literal double-quoted string and interpolates variables in it already. It's a great choice when you want to read the template from another file though. A good replacement for the much more cumbersome m4. – Consort

I was very pleasantly surprised to learn about this command. I was trying to cobble envsubst's functionality manually with zero success. Thanks yottatsa! – Warrington

Note: envsubst is a GNU gettext utility, and is actually not all that robust (since gettext is meant for localizing human messages). Most importantly, it doesn't recognize backslash-escaped ${VAR} substitutions (so you can't have a template that uses $VAR substitutions at runtime, like a shell script or Nginx conf file). See my answer for a solution that handles backslash escapes. – Kawai

@Consort In this case, if you wanted to pass this template to envsubst for some reason, you'd want to use <<"EOF", which doesn't interpolate variables (quoted terminators are like the single-quotes of heredocs). – Kawai

I used it like: cat template.txt | envsubst – Midyear

Sadly, its not possible to get envsubst to fail on missing environment variables, which makes it unsuitable for lots of use cases. – Sliver

Note that as explained here I had to apt install gettext-base in my Dockerfile in order to be able to use envsubst – Tope

Large replacement strings will cause Argument list too long. – Marsipobranch

Answer 3

A heredoc is a builtin way to template a conf file.

STATUS_URI="/hows-it-goin";  MONITOR_IP="10.10.2.15";

cat >/etc/apache2/conf.d/mod_status.conf <<EOF
<Location ${STATUS_URI}>
    SetHandler server-status
    Order deny,allow
    Deny from all
    Allow from ${MONITOR_IP}
</Location>
EOF

Regarding yottsa's answer: envsubst was new to me. Fantastic.

Fiester

i prefer this better than envsubst coz it saved my from the additional apt-get install gettext-base in my Dockerfile – Midyear

The shell as a Template-like script however without any external library installation nor stress from coping with tricky expressions. – Palatinate

My preferred solution ! – Pedagogy

Answer 4

I agree with using sed: it is the best tool for search/replace. Here is my approach:

$ cat template.txt
the number is ${i}
the dog's name is ${name}

$ cat replace.sed
s/${i}/5/g
s/${name}/Fido/g

$ sed -f replace.sed template.txt > out.txt

$ cat out.txt
the number is 5
the dog's name is Fido

Gingery

This requires temporary file for substitution string, right? Is there a way to do that without temporary files? – Bespread

@FractalizeR: Some versions of sed have a -i option (edit files in place) that is similar to the perl option. Check the manpage for your sed. – Mucin

@FractalizeR Yes, sed -i will replace inline. If you are comfortable with Tcl (another scripting language), then check out this thread: #2818630 – Gingery

I created the replace.sed from a propertyfiles whit the following sed command: sed -e 's/^/s\/${/g' -e 's/=/}\//g' -e 's/$/\//g' the.properties > replace.sed – Nagual

@hai vu’s code creates a sed program and passes that program in using sed’s -f flag. If you wanted, you could instead pass in each line of the sed program into sed using the -e flags. FWIW I like the idea of using sed for templating. – Lasley

Answer 5

I have a bash solution like mogsie but with heredoc instead of herestring to allow you to avoid escaping double quotes

eval "cat <<EOF
$(<template.txt)
EOF
" 2> /dev/null

Rump

This solution supports Bash parameter expansion in the template. My favorites are required parameters with ${param:?} and nesting text around optional parameters. Example: ${DELAY:+<delay>$DELAY</delay>} expands to nothing when DELAY is undefined and <delay>17</delay> when DELAY=17. – Footpoundsecond

Oh! And the EOF delimiter can use a dynamic string, like the PID _EOF_$$. – Footpoundsecond

@Huesman A workaround for trailing newlines is to use some expansion like e.g. an empty variable $trailing_newline, or use $NL5 and make sure it gets expanded as 5 newlines. – Womanish

@xebeche: Yes, placing what you suggest at the very end inside template.txt would work in order to preserve trailing newlines. – Huesman

An elegant solution, but note that the command substitution will strip any trailing newlines from the input file, although that will typically not be a problem. Another edge case: due to use of eval, if template.txt contains EOF on a line of its own, it'll prematurely terminate the here-doc and thus break the command. (Tip of the hat to @xebeche). – Huesman

if template contain single `, this solution will has error。I have encountered this kind of problems when rendering zabbix_agentd.conf – Barye

Is this solution really any different from eval "$(<template.txt)" 2> /dev/null? – Reword

To answer my own question: Yes this is different than doing eval "$(<template.txt)" 2> /dev/null because in the case where cat is used, the template will be printed, but in the case I have above, eval will evaluate the template string and basically try to run it as code – Reword

Answer 6

Try eval

I think eval works really well. It handles templates with linebreaks, whitespace, and all sorts of bash stuff. If you have full control over the templates themselves of course:

$ cat template.txt
variable1 = ${variable1}
variable2 = $variable2
my-ip = \"$(curl -s ifconfig.me)\"

$ echo $variable1
AAA
$ echo $variable2
BBB
$ eval "echo \"$(<template.txt)\"" 2> /dev/null
variable1 = AAA
variable2 = BBB
my-ip = "11.22.33.44"

This method should be used with care, of course, since eval can execute arbitrary code. Running this as root is pretty much out of the question. Quotes in the template need to be escaped, otherwise they will be eaten by eval.

You can also use here documents if you prefer cat to echo

$ eval "cat <<< \"$(<template.txt)\"" 2> /dev/null

@plockc provoded a solution that avoids the bash quote escaping issue:

$ eval "cat <<EOF
$(<template.txt)
EOF
" 2> /dev/null

Edit: Removed part about running this as root using sudo...

Edit: Added comment about how quotes need to be escaped, added plockc's solution to the mix!

Hague

This strips quotes you have in your template, and won't substitute inside single quotes, so depending on your template format, may lead to subtle bugs. This is probably applicable to any Bash-based templating method, though. – Slide

IMHO Bash-based templates are madness, since you need to be a bash programmer in order to understand what your template is doing! But thanks for the comment! – Hague

@AlexB: This approach will substitute between single quotes, as they're just literal characters inside the enclosing double-quoted string rather than string delimiters when the evaled echo / cat commands processes them; try eval "echo \"'\$HOME'\"". – Huesman

Answer 7

Edit Jan 6, 2017

I needed to keep double quotes in my configuration file so double escaping double quotes with sed helps:

render_template() {
  eval "echo \"$(sed 's/\"/\\\\"/g' $1)\""
}

I can't think of keeping trailing new lines, but empty lines in between are kept.

---

Although it is an old topic, IMO I found out more elegant solution here: http://pempek.net/articles/2013/07/08/bash-sh-as-template-engine/

#!/bin/sh

# render a template configuration file
# expand variables + preserve formatting
render_template() {
  eval "echo \"$(cat $1)\""
}

user="Gregory"
render_template /path/to/template.txt > path/to/configuration_file

All credits to Grégory Pakosz.

Wynny

This removes double quotes from the input and, if there are multiple trailing newlines in the input file, replaces them with a single one. – Huesman

I needed two fewer backslashes to make it work, i.e., eval "echo \"$(sed 's/\"/\\"/g' $1)\"" – Hearth

Unfortunately, this approach does not allow you to template php files (they contains $variables). – Quinby

Answer 8

Instead of reinventing the wheel go with envsubst Can be used in almost any scenario, for instance building configuration files from environment variables in docker containers.

If on mac make sure you have homebrew then link it from gettext:

brew install gettext
brew link --force gettext

./template.cfg

# We put env variables into placeholders here
this_variable_1 = ${SOME_VARIABLE_1}
this_variable_2 = ${SOME_VARIABLE_2}

./.env:

SOME_VARIABLE_1=value_1
SOME_VARIABLE_2=value_2

./configure.sh

#!/bin/bash
cat template.cfg | envsubst > whatever.cfg

Now just use it:

# make script executable
chmod +x ./configure.sh
# source your variables
. .env
# export your variables
# In practice you may not have to manually export variables 
# if your solution depends on tools that utilise .env file 
# automatically like pipenv etc. 
export SOME_VARIABLE_1 SOME_VARIABLE_2
# Create your config file
./configure.sh

Stickler

this invocation sequence of envsubst actually works. – Cymophane

For anyone else looking, envsubst does not work on MacOS, you'd need to install it using homebrew: brew install gettext. – Euler

I dig the cat template.cfg | envsubst > whatever.cfg method; nice and clean. Well played. – Bloem

Answer 9

I'd have done it this way, probably less efficient, but easier to read/maintain.

TEMPLATE='/path/to/template.file'
OUTPUT='/path/to/output.file'

while read LINE; do
  echo $LINE |
  sed 's/VARONE/NEWVALA/g' |
  sed 's/VARTWO/NEWVALB/g' |
  sed 's/VARTHR/NEWVALC/g' >> $OUTPUT
done < $TEMPLATE

Chokeberry

You can do this without reading line-by-line and with only one sed invocation: sed -e 's/VARONE/NEWVALA/g' -e 's/VARTWO/NEWVALB/g' -e 's/VARTHR/NEWVALC/g' < $TEMPLATE > $OUTPUT – Kerikeriann

Answer 10

If you want to use Jinja2 templates, see this project: j2cli.

It supports:

• Templates from JSON, INI, YAML files and input streams
• Templating from environment variables

Turro

Answer 11

A longer but more robust version of the accepted answer:

perl -pe 's;(\\*)(\$([a-zA-Z_][a-zA-Z_0-9]*)|\$\{([a-zA-Z_][a-zA-Z_0-9]*)\})?;substr($1,0,int(length($1)/2)).($2&&length($1)%2?$2:$ENV{$3||$4});eg' template.txt

This expands all instances of $VAR or ${VAR} to their environment values (or, if they're undefined, the empty string).

It properly escapes backslashes, and accepts a backslash-escaped $ to inhibit substitution (unlike envsubst, which, it turns out, doesn't do this).

So, if your environment is:

FOO=bar
BAZ=kenny
TARGET=backslashes
NOPE=engi

and your template is:

Two ${TARGET} walk into a \\$FOO. \\\\
\\\$FOO says, "Delete C:\\Windows\\System32, it's a virus."
$BAZ replies, "\${NOPE}s."

the result would be:

Two backslashes walk into a \bar. \\
\$FOO says, "Delete C:\Windows\System32, it's a virus."
kenny replies, "${NOPE}s."

If you only want to escape backslashes before $ (you could write "C:\Windows\System32" in a template unchanged), use this slightly-modified version:

perl -pe 's;(\\*)(\$([a-zA-Z_][a-zA-Z_0-9]*)|\$\{([a-zA-Z_][a-zA-Z_0-9]*)\});substr($1,0,int(length($1)/2)).(length($1)%2?$2:$ENV{$3||$4});eg' template.txt

Kawai

Answer 12

Here's another pure bash solution:

• it's using heredoc, so:
  • complexity doesn't increase because of additionaly required syntax
  • template can include bash code
    • that also allows you to indent stuff properly. See below.
• it doesn't use eval, so:
  • no problems with the rendering of trailing empty lines
  • no problems with quotes in the template

$ cat code

#!/bin/bash
LISTING=$( ls )

cat_template() {
  echo "cat << EOT"
  cat "$1"
  echo EOT
}

cat_template template | LISTING="$LISTING" bash

Input:
$ cat template (with trailing newlines and double quotes)

<html>
  <head>
  </head>
  <body> 
    <p>"directory listing"
      <pre>
$( echo "$LISTING" | sed 's/^/        /' )
      <pre>
    </p>
  </body>
</html>

Output:

<html>
  <head>
  </head>
  <body> 
    <p>"directory listing"
      <pre>
        code
        template
      <pre>
    </p>
  </body>
</html>

Blayze

Answer 13

Here is another solution: generate a bash script with all the variables and the contents of the template file, that script would look like this:

word=dog           
i=1                
cat << EOF         
the number is ${i} 
the word is ${word}

EOF                

If we feed this script into bash it would produce the desired output:

the number is 1
the word is dog

Here is how to generate that script and feed that script into bash:

(
    # Variables
    echo word=dog
    echo i=1

    # add the template
    echo "cat << EOF"
    cat template.txt
    echo EOF
) | bash

Discussion

• The parentheses opens a sub shell, its purpose is to group together all the output generated
• Within the sub shell, we generate all the variable declarations
• Also in the sub shell, we generate the cat command with HEREDOC
• Finally, we feed the sub shell output to bash and produce the desired output

• If you want to redirect this output into a file, replace the last line with:

  ) | bash > output.txt
  

Gingery

Answer 14

Taking the answer from ZyX using pure bash but with new style regex matching and indirect parameter substitution it becomes:

#!/bin/bash
regex='\$\{([a-zA-Z_][a-zA-Z_0-9]*)\}'
while read line; do
    while [[ "$line" =~ $regex ]]; do
        param="${BASH_REMATCH[1]}"
        line=${line//${BASH_REMATCH[0]}/${!param}}
    done
    echo $line
done

Fictive

Answer 15

If using Perl is an option and you're content with basing expansions on environment variables only (as opposed to all shell variables), consider Stuart P. Bentley's robust answer.

This answer aims to provide a bash-only solution that - despite use of eval - should be safe to use.

The goals are:

• Support expansion of both ${name} and $name variable references.
• Prevent all other expansions:
  • command substitutions ($(...) and legacy syntax `...`)
  • arithmetic substitutions ($((...)) and legacy syntax $[...]).
• Allow selective suppression of variable expansion by prefixing with \ (\${name}).
• Preserve special chars. in the input, notably " and \ instances.
• Allow input either via arguments or via stdin.

Function expandVars():

expandVars() {
  local txtToEval=$* txtToEvalEscaped
  # If no arguments were passed, process stdin input.
  (( $# == 0 )) && IFS= read -r -d '' txtToEval
  # Disable command substitutions and arithmetic expansions to prevent execution
  # of arbitrary commands.
  # Note that selectively allowing $((...)) or $[...] to enable arithmetic
  # expressions is NOT safe, because command substitutions could be embedded in them.
  # If you fully trust or control the input, you can remove the `tr` calls below
  IFS= read -r -d '' txtToEvalEscaped < <(printf %s "$txtToEval" | tr '`([' '\1\2\3')
  # Pass the string to `eval`, escaping embedded double quotes first.
  # `printf %s` ensures that the string is printed without interpretation
  # (after processing by by bash).
  # The `tr` command reconverts the previously escaped chars. back to their
  # literal original.
  eval printf %s "\"${txtToEvalEscaped//\"/\\\"}\"" | tr '\1\2\3' '`(['
}

Examples:

$ expandVars '\$HOME="$HOME"; `date` and $(ls)'
$HOME="/home/jdoe"; `date` and $(ls)  # only $HOME was expanded

$ printf '\$SHELL=${SHELL}, but "$(( 1 \ 2 ))" will not expand' | expandVars
$SHELL=/bin/bash, but "$(( 1 \ 2 ))" will not expand # only ${SHELL} was expanded

• For performance reasons, the function reads stdin input all at once into memory, but it's easy to adapt the function to a line-by-line approach.
• Also supports non-basic variable expansions such as ${HOME:0:10}, as long as they contain no embedded command or arithmetic substitutions, such as ${HOME:0:$(echo 10)}
  • Such embedded substitutions actually BREAK the function (because all $( and ` instances are blindly escaped).
  • Similarly, malformed variable references such as ${HOME (missing closing }) BREAK the function.
• Due to bash's handling of double-quoted strings, backslashes are handled as follows:
  • \$name prevents expansion.
  • A single \ not followed by $ is preserved as is.
  • If you want to represent multiple adjacent \ instances, you must double them; e.g.:
    • \\ -> \ - the same as just \
    • \\\\ -> \\
  • The input mustn't contain the following (rarely used) characters, which are used for internal purposes: 0x1, 0x2, 0x3.
• There's a largely hypothetical concern that if bash should introduce new expansion syntax, this function might not prevent such expansions - see below for a solution that doesn't use eval.

---

If you're looking for a more restrictive solution that only supports ${name} expansions - i.e., with mandatory curly braces, ignoring $name references - see this answer of mine.

---

Here is an improved version of the bash-only, eval-free solution from the accepted answer:

The improvements are:

• Support for expansion of both ${name} and $name variable references.
• Support for \-escaping variable references that shouldn't be expanded.
• Unlike the eval-based solution above,
  • non-basic expansions are ignored
  • malformed variable references are ignored (they don't break the script)

 IFS= read -d '' -r lines # read all input from stdin at once
 end_offset=${#lines}
 while [[ "${lines:0:end_offset}" =~ (.*)\$(\{([a-zA-Z_][a-zA-Z_0-9]*)\}|([a-zA-Z_][a-zA-Z_0-9]*))(.*) ]] ; do
      pre=${BASH_REMATCH[1]} # everything before the var. reference
      post=${BASH_REMATCH[5]}${lines:end_offset} # everything after
      # extract the var. name; it's in the 3rd capture group, if the name is enclosed in {...}, and the 4th otherwise
      [[ -n ${BASH_REMATCH[3]} ]] && varName=${BASH_REMATCH[3]} || varName=${BASH_REMATCH[4]}
      # Is the var ref. escaped, i.e., prefixed with an odd number of backslashes?
      if [[ $pre =~ \\+$ ]] && (( ${#BASH_REMATCH} % 2 )); then
           : # no change to $lines, leave escaped var. ref. untouched
      else # replace the variable reference with the variable's value using indirect expansion
           lines=${pre}${!varName}${post}
      fi
      end_offset=${#pre}
 done
 printf %s "$lines"

Huesman

Answer 16

Try shtpl

Perfect case for shtpl. (project of mine, so it is not widely in use and lacks in documentation. But here is the solution it offers anyhow. May you want to test it.)

Just execute:

$ i=1 word=dog sh -c "$( shtpl template.txt )"

Result is:

the number is 1
the word is dog

Have fun.

Tallula

If it's crap, it's downvoted anyway. And i'm ok with that. But ok, point taken, that it is not clearly visible, that it is actually my project. Going to make it more visible in the future. Thank you anyhow for your comment and your time. – Tallula

I want to add, that i really searched for usecases yesterday, where shtpl would be a perfect solution. Yeah, i was bored... – Tallula

Answer 17

To follow up on plockc's answer on this page, here is a dash-suitable version, for those of you looking to avoid bashisms.

eval "cat <<EOF >outputfile
$( cat template.in )
EOF
" 2> /dev/null

Supplementary

Answer 18

This page describes an answer with awk

awk '{while(match($0,"[$]{[^}]*}")) {var=substr($0,RSTART+2,RLENGTH -3);gsub("[$]{"var"}",ENVIRON[var])}}1' < input.txt > output.txt

Bluepoint

This keeps all the quotes intact. Great! – Cassowary

Answer 19

# Usage: template your_file.conf.template > your_file.conf
template() {
        local IFS line
        while IFS=$'\n\r' read -r line ; do
                line=${line//\\/\\\\}         # escape backslashes
                line=${line//\"/\\\"}         # escape "
                line=${line//\`/\\\`}         # escape `
                line=${line//\$/\\\$}         # escape $
                line=${line//\\\${/\${}       # de-escape ${         - allows variable substitution: ${var} ${var:-default_value} etc
                # to allow arithmetic expansion or command substitution uncomment one of following lines:
#               line=${line//\\\$\(/\$\(}     # de-escape $( and $(( - allows $(( 1 + 2 )) or $( command ) - UNSECURE
#               line=${line//\\\$\(\(/\$\(\(} # de-escape $((        - allows $(( 1 + 2 ))
                eval "echo \"${line}\"";
        done < "$1"
}

This is the pure bash function adjustable to your liking, used in production and should not break on any input. If it breaks - let me know.

Hogg

Answer 20

You can also use bashible (which internally uses the evaluating approach described above/below).

There is an example, how to generate a HTML from multiple parts:

https://github.com/mig1984/bashible/tree/master/examples/templates

Pleuropneumonia

Answer 21

Look at simple variables substitution python script here: https://github.com/jeckep/vsubst

It is very simple to use:

python subst.py --props secure.properties --src_path ./templates --dst_path ./dist

Hymettus

Answer 22

You can use a template file with environment variables and run

cat template-file.yaml |envsubst

Untangle

Answer 23

Here's a bash function that preserves whitespace:

# Render a file in bash, i.e. expand environment variables. Preserves whitespace.
function render_file () {
    while IFS='' read line; do
        eval echo \""${line}"\"
    done < "${1}"
}

Frederic

Answer 24

Here's a modified perl script based on a few of the other answers:

perl -pe 's/([^\\]|^)\$\{([a-zA-Z_][a-zA-Z_0-9]*)\}/$1.$ENV{$2}/eg' -i template

Features (based on my needs, but should be easy to modify):

• Skips escaped parameter expansions (e.g. \${VAR}).
• Supports parameter expansions of the form ${VAR}, but not $VAR.
• Replaces ${VAR} with a blank string if there is no VAR envar.
• Only supports a-z, A-Z, 0-9 and underscore characters in the name (excluding digits in the first position).

Stringfellow

Answer 25

You can also use printf to fill a template.

#!/bin/bash

IFS='' read -rd '' TEMPL <<-'EOB'
The number is %d
The word is "%s"
Birds of Massachusetts:
    %s




EOB

N=12
WORD="Bird"
MULTILINE="Eastern Bluebirds
Common Grackles"

echo "START"
printf "${TEMPL}" ${N} ${WORD} "${MULTILINE}"
echo "END"

Here's the output, with quotes and whitespace intact:

START
The number is 12
The word is "Bird"
Birds of Massachusetts:
    Eastern Bluebirds
Common Grackles




END

Overland

Answer 26

I was looking for a standard-utils-only solution and also I was not comfortable with evaling an arbitrary template – this can go wrong only too easily. Eventually I came up with the following approach.

Template file:

$ cat addusers.template.sh 
# Available variables:
# $USER_NAME
# $USER_ID
# $USER_GROUP_ID

adduser --uid $USER_ID --gid $USER_GROUP_ID --gecos '' --disabled-password $USER_NAME
cp -v /tmp/bash.rc /home/$USER_NAME/.bashrc
chown -v $USER_ID:$USER_GROUP_ID /home/$USER_NAME/.bashrc
chmod 444 /home/$USER_NAME/.bashrc

Bash script:

$ cat cook-addusers.sh
# Read template file skipping comments and empty lines
TEMPLATE="$(grep -vE -e '^[[:blank:]]*#' -e '^[[:blank:]]*$' addusers.template.sh)"

# Cook regex template with the list of substituted variables
for VAR in USER_NAME USER_ID USER_GROUP_ID; do
    RE="${RE}"'s/\$'$VAR'/$'$VAR'/g\; '
done

# Fill in the template for a list of users
for USER_NAME in admin superuser supervisor; do
    USER_ID=$(id -u "$USER_NAME")
    USER_GROUP_ID=$(id -g "$USER_NAME")
    sed "$(eval echo "$RE")" <<<"${TEMPLATE}"
done >generated.sh

And the result of running it:

$ ./cook-addusers.sh
$ cat generated.sh 
adduser --uid 1001 --gid 101 --gecos '' --disabled-password admin
cp -v /tmp/bash.rc /home/admin/.bashrc
chown -v 1001:101 /home/admin/.bashrc
chmod 444 /home/admin/.bashrc
adduser --uid 1010 --gid 101 --gecos '' --disabled-password superuser
cp -v /tmp/bash.rc /home/superuser/.bashrc
chown -v 1010:101 /home/superuser/.bashrc
chmod 444 /home/superuser/.bashrc
adduser --uid 1100 --gid 101 --gecos '' --disabled-password supervisor
cp -v /tmp/bash.rc /home/supervisor/.bashrc
chown -v 1100:101 /home/supervisor/.bashrc
chmod 444 /home/supervisor/.bashrc

It's rather crude though seems to be working as intended both on typical Linux and BSD… sorry, I mean macOS ;-)

Aruspex