Can envsubst not do in-place substitution?

Question

I have a config file which contains some ENV_VARIABLE styled variables.

This is my file.
It might contain $EXAMPLES of text.

Now I want that variable replaced with a value which is saved in my actual environment variables. So I'm trying this:

export EXAMPLES=lots
envsubst < file.txt > file.txt

But it doesn't work when the input file and output file are identical. The result is an empty file of size 0.

There must be a good reason for this, some bash basics that I'm not aware of? How do I achieve what I want to do, ideally without first outputting to a different file and then replacing the original file with it?

I know that I can do it easily enough with sed, but when I discovered the envsubst command I thought that it should be perfect for my use case, so I'd like to use that.

Hoi

You never accepted the best answer. I think this is https://mcmap.net/q/350880/-can-envsubst-not-do-in-place-substitution – Bolero

Answer 1

Here is the solution that I use:

originalfile="file.txt"
tmpfile=$(mktemp)
cp --attributes-only --preserve $originalfile $tmpfile
cat $originalfile | envsubst > $tmpfile && mv $tmpfile $originalfile

Be careful with other solutions that do not use a temporary file. Pipes are asynchronous, so the file will occasionally be read after it has already been truncated.

Amii

This may change the permissions of file.txt, since the temporary file may not have the same permissions. – Iodoform

Thanks for your comment, @maiermic. I have updated the code to preserve file permissions. – Amii

What's the point of tee here? That dumps the output into stdout, presumably the console, which seems a bit unnecessary. – Stat

@rici, I think you are right. I edited the answer by replacing tee with >. Thanks for the feedback! – Amii

Upvoted for using good names and --long-form options. Readability is incredibly important. – Hoi

Answer 2

To avoid creating a temporary file, use sponge not tee:

envsubst < file.txt | sponge file.txt

From https://linux.die.net/man/1/sponge:

sponge reads standard input and writes it out to the specified file. Unlike a shell redirect, sponge soaks up all its input before opening the output file. This allows constricting pipelines that read from and write to the same file.

Dulci

This is the most elegant way – Bolero

Beautiful, I've never heard of this before. Is this specific to any shell or distro or would this also work in GNU and on Mac? – Hoi

@Hoi sponge binary is a part of moreutils package – Dulci

Answer 3

Redirects are handled by the shell, not the program being executed, and they are set up before the program is invoked.

The redirect >output.file has the effect of creating output.file if it doesn't exist and emptying it if it does. Either way, you end up with an empty file, and that is what the program's output is redirected to.

Programs like sed which are capable of "in-place" modification must take the filename as a command-line argument, not as a redirect.

In your case, I would suggest using a temporary file and then renaming it if all goes OK.

Stat

Answer 4

I found another shortcut to put into temp file and then rename it to original file.

envsubst < in.txt > out.txt && mv out.txt in.txt

Personage

Answer 5

envsubst < file.txt | tee file.txt

Caution: @SoftwareFactor rightly pointed out in the comments that this answer suffers from a race condition and can cause the resulting file to be empty at random.

Hammett

This actually works! You can glue it with find if you want to do recursive substitution. find . -type f -exec bash -c 'envsubst < $1 | tee' -- {} \; – Pratique

Be careful with this answer. It has a race condition. Pipes are asynchronous, so envsubst will occasionally read the file after tee has already truncated it. Try running the following experiment - you will see that your file will become empty: for i in {0..1000}; do envsubst < file.txt | tee file.txt; done – Amii

@Pratique I think it is really has some race condition. isn't it better to do find . -type f -exec /bin/sh -c 'envsubst < $1 > $1.tmp && mv $1.tmp $1' -- {} \; – Decoration

BE CAREFUL !!! Thanks to @SoftwareFactor. Sometimes in our CI we faced an undefined behavior. It required long time to realize the issue was in race condition from the command as in answer ! – Obtrude

See the correct answer: https://mcmap.net/q/350880/-can-envsubst-not-do-in-place-substitution – Bolero

Answer 6

You can achieve in-place substitution by calling envsubst from gnu sed with the "e" command:

EXAMPLES=lots sed -i 's/.*/echo & | envsubst/e' file.txt

Swaggering

This doesn't work on mac: sed -i 's/.*/echo & | envsubst/e' infra/codedeploy/scripts/install_dependencies sed: 1: "infra/codedeploy/script ...": command i expects \ followed by text – Fayola

@ChristianBongiorno it works if you install gnu sed, which is not the default on macos. But I wouldn't recommend this solution anyways, because & is unquoted and will lead to errors (best case) or unwanted command executions (worst case). – Nowak

Answer 7

It's worth noting that the mv solution won't maintain file permissions. Using cp -pf would be preferable in the case that you're modifying an executable file.

tmpfile=$(mktemp)
cat file.txt | envsubst > "$tmpfile" && cp -pf "$tmpfile" file.txt
rm -f "$tmpfile"

Aleksandrovsk

This may change the permissions of file.txt, since the temporary file may not have the same permissions. – Iodoform

That’s what -p does by preserving permissions, no? – Aleksandrovsk

-p preserves the permissions of the source file, which is the temporary file and not the original file (file.txt). See ss64.com/osx/cp.html – Iodoform

Answer 8

Updated 20221011 - Using 1 sed command

sed -i -r 's/["`]|\$\(/\\&/g; s/.*/echo "&"/ e' ./input.txt

Updated 20221007 - Using 2 sed commands

sed -i -r 's/["`]|\$\(/\\&/g' input.txt
sed -i -r 's/.*/echo "&"/ e' input.txt

Do it without envsubst

envsubst_file () {
    local original_file=$1
    local temp_file=$(mktemp)
    trap "rm -f ${temp_file}" 0 2 3 15
    cp -p ${original_file} ${temp_file}
    cat ${original_file} | sed -r 's/["`]|\$\(/\\&/g' | sed -r 's/.*/echo "&"/g' | sh > ${temp_file}
    mv ${temp_file} ${original_file}
}

envsubst_file 'input.txt'

First using sed to escapes double quotes("), backtick(`) and command $( by prefixing with backslash(\),then using sed again replace with

echo "&"

Finally executing the shell script and redirecting to ${temp_file}

Overthecounter

Answer 9

This answer was framed from two other answers. I guess this is the best solution.

originalFile=file.txt
tmpfile=$(mktemp)
cat $originalFile | envsubst > "$tmpfile" && cp -pf "$tmpfile" $originalFile
rm -f "$tmpfile"

Bergmans

Answer 10

If you use bash, check this:

a=`<file.txt` && envsubst <<<"$a" >file.txt

Tested on 500mb file, works as expected.

Karyolymph

Answer 11

This works just fine without the need for temp file. Expression is evaluated before it flushes the content to the file.

echo "$(envsubst < file.txt)" > file.txt

Oliana

Answer 12

In the end I found that using envsubst was too dangerous after all. My files might contain dollar signs in places where I don't want any substitution to happen, and envsubst will just replace them with empty strings if no corresponding environment variable is defined. Not cool.

Hoi

In that case you need to explicitly specify which env vars to interpolate. It will not touch those that are not listed. ie. envsubst "$MYVARS" < source > destination – Absorptance

Thanks for this tip, @lisak! I had to use single quotes, e.g. envsubst '${PARAMETER}' < source > destination... – Saxtuba