How to assign a heredoc value to a variable in Bash?

Question

I have this multi-line string (quotes included):

abc'asdf"
$(dont-execute-this)
foo"bar"''

How would I assign it to a variable using a heredoc in Bash?

I need to preserve newlines.

I don't want to escape the characters in the string, that would be annoying...

Bimonthly

@JohnM - I have just tried a heredoc assignment with single-quoted 'EOF', with escaped linebreaks with ` in the content: if the second line has cd` command, I get back: ".sh: line X: cd: command not found"; but if I double-quote "EOF"; then bash variables ${A} do not get preserved as strings (they get expanded); but then, line-breaks are preserved - and, I don't have a problem running a command with cd in second line (and both 'EOF' and "EOF" seem to play well also with eval, for running a set of commands stored in a string variable). Cheers! – Mcmurry

... and to add to my previous comment: bash comments "#" in double-qouted "EOF" variable, if called via eval $VAR, will cause all of the rest of the script to be commented, as here $VAR will be seen as a single line; to be able to use bash # comments in multiline script, double-quote also variable in the eval call: eval "$VAR"`. – Mcmurry

@sdaau: I had problems with eval ith this methods, but did not track it down since it was part of some package which evals some variables defined in it's config file. Error message was: /usr/lib/network/network: eval: line 153: syntax error: unexpected end of file. I just switched to another solution. – Byerly

There are situations when you really genuinely want a here document, but if you are simply looking for how to put a newline in a static string, probably read #3006463 instead. – Maser

Answer 1

You can avoid a useless use of cat and handle mismatched quotes better with this:

$ read -r -d '' VAR <<'EOF'
abc'asdf"
$(dont-execute-this)
foo"bar"''
EOF

If you don't quote the variable when you echo it, newlines are lost. Quoting it preserves them:

$ echo "$VAR"
abc'asdf"
$(dont-execute-this)
foo"bar"''

If you want to use indentation for readability in the source code, use a dash after the less-thans. The indentation must be done using only tabs (no spaces).

$ read -r -d '' VAR <<-'EOF'
    abc'asdf"
    $(dont-execute-this)
    foo"bar"''
    EOF
$ echo "$VAR"
abc'asdf"
$(dont-execute-this)
foo"bar"''

If, instead, you want to preserve the tabs in the contents of the resulting variable, you need to remove tab from IFS. The terminal marker for the here doc (EOF) must not be indented.

$ IFS='' read -r -d '' VAR <<'EOF'
    abc'asdf"
    $(dont-execute-this)
    foo"bar"''
EOF
$ echo "$VAR"
    abc'asdf"
    $(dont-execute-this)
    foo"bar"''

Tabs can be inserted at the command line by pressing Ctrl-V Tab. If you are using an editor, depending on which one, that may also work or you may have to turn off the feature that automatically converts tabs to spaces.

Dreary

This is useful to embed multi-line perl programs in bash variables when combined with the -r option to preserve backslashes as well. – Cornerwise

@TrinitronX: True, read should almost always have -r. I forgot to include it (fixed). – Dreary

I think it's worth mentioning that if you have set -o errexit (a.k.a set -e) in your script and you use this then it will terminate your script because read returns a non-zero return code when it reaches EOF. – Doorsill

@MarkByers: That's one of the reasons I never use set -e and always recommend against its use. It's better to use proper error handling instead. trap is your friend. Other friends: else and || among others. – Dreary

@Dennis How do I do this while including the tab character for indention? I know I can just use space, but is this possible? – Noelyn

@DennisWilliamson is there an alternative to using the -d parameter on read? I'm stuck using an environment that doesn't have the -d option, but I can't figure out any other way to get read to load multiple lines without it. – Didymium

Is the avoidance of cat really worth it in such a case? Assigning a heredoc to a variable with cat is a well known idiom. Somehow using read obfuscates things for little benefits imho. – Lanita

Meh. A warning from someone who's been trying to be too smart (me): read -rd'' VAR << EOF won't work, because VAR will be parsed as the delimiter (and the heredoc will end up in $REPLY). – Direct

@Direct That's because you don't have a space between d and the empty string; bash collapses -rd'' to simply -rd before read ever sees its arguments, so VAR is treated as the argument to -d. – Dejong

In this format, read will return with a non-zero exit code. This makes this method less than ideal in a script with error checking enabled (eg set -e). – Unconventional

@Swiss: set -e should never be used - do proper error checking instead – Dreary

@DennisWilliamson set -e was just an example. My point is that the read command reports failure when this is run. – Unconventional

Is it possible to use this and get variable substitution? I would like to use $variables in my multiline string. – Kilt

@LandonKuhn: Remove the quotes from around the name of the heredoc. This will also allow command substitutions to be evaluated and will affect white space in the manner i indicated in my answer. – Dreary

It is too slow because shell calls read(2) for each 1 byte in the here document. – Mosra

Why does is $? == 1 after executing IFS='' read -r -d '' VAR <<'EOF' even when it consumes an "EOF" at the beginning of the line? Any way to work around this? – Luminance

Answering my own question, see this answer (thanks @ffledgling). – Luminance

While I love traping errors too, when you are working on a shared code base, it's not polite to change an agreed upon standard (like #!/bin/bash -eu). So, I added a note to the end of this otherwise excellent answer. – Perspicuous

@BrunoBronosky: set -e should never be used. – Dreary

what do these arguments mean -r -d ? I couldn't find in man read. Example 3 worked well in my case. – Monody

@lacostenycoder: man read is about the operating system function call so it doesn't directly apply. In this context, read is a Bash builtin and you can get information about it using help read or man bash. The option -r is to accept backslashes literally instead of using them to escape any characters. The -d option sets the first character of the following argument as the delimiter between records (lines) instead of the default which is newline. In the example, I'm setting the delimiter to a null string. – Dreary

A tiny note to this otherwise wonderful answer: Leading/trailing blank lines are chomped by read unless IFS=. – Malawi

You could use sed to keep your indentation, for example.... – Peggie

Thanks to @dennis-williamson and @mark-byers I am still astonished why no one suggested: set -o errexit eof=$'\4' read -r -d "$eof" VAR <<EOF here-document ${eof} EOF – Singularize

@166_MMX: I repeat: don't use errexit. – Dreary

Its not useless use of cat. With read, you will always get exit code 1. With cat you don't. Do you use bash unofficial strict mode? In this case, I consider cat the winner. – Swedenborgian

?ote that: zsh allows this syntax to work: var=$(<<EOF … – Repossess

@Pauseduntilfurthernotice. At the bottom of the document that you keep linking, there is a conclusions section. Two of the three conclusions suggest not using errexit, but one suggests using it while being aware of the consequences. In short, it's debatable. – Milkman

Regarding getting non-zero exit code one could use a || true right after the start of heredoc, i.e read -r -d '' VAR <<'EOF' || true for the first example. – Northeast

Absolutely do use -e / -o errexit. The bash FAQ was written decades ago and is outdated. You cannot predict all possible failure scenarios. Not using -e is equivalent to ON ERROR RESUME NEXT in Visual Basic. Do not do this. As for this particular instance, the error is due to -d '', which tells read to stop on a NUL character (i.e. a zero byte). Because there is none in the input, the command correctly reports an error, so masking it (e.g. with `|| true) is the correct fix. (You also clearly can't use this for data containing NUL bytes, but you can't put those in variables anyway.) – Mythify

IFS='' is needed to preserve trailing whitespace. – Mythify

Amendment to my previous comment: -e / -o errexit and/or an ERR trap. – Mythify

Pro: read is a bash builtin, and so will invoke quicker than cat. Con: -d is not apart of POSIX, and may not be portable to other shells. – Reannareap

is there utility to simplification? by the looks of this comment thread, this is a useless use of read. – Disorderly

Answer 2

Use $() to assign the output of cat to your variable like this:

VAR=$(
cat <<'END_HEREDOC'
abc'asdf"
$(dont-execute-this)
foo"bar"''
END_HEREDOC
)

# this will echo variable with new lines intact
echo "$VAR"
# this will echo variable without new lines (changed to space character)
echo $VAR

Making sure to delimit starting END_HEREDOC with single-quotes. This will prevent the content of the heredoc from being expanded, so dont-execute-this will not be executed.

Note that ending heredoc delimiter END_HEREDOC must be alone on the line (hence the ending parenthesis ) is on the next line).

Thanks to @ephemient for the answer.

Bimonthly

Actually, this one lets through quotes in some circumstances. I managed to do this in Perl easily... – Bimonthly

+1. This is the most readable solution, at least for my eyes. It leaves the name of the variable at the far left of the page, instead of embedding it in the read command. – Erlineerlinna

When using this construct, I see Newlines are being converted to $ (dollar character) ?? – Mussulman

There's a nitch: you musn't leave blank beside the '=' sign. – Crux

PSA: remember that the variable must be quoted to preserve newlines. echo "$VAR" instead of echo $VAR. – Urticaria

This is nice with ash and OpenWRT where read doesn't support -d. – Perren

For reasons I cannot fathom, this fails with an "unexpected EOF" error if you have an unpaired backtick in the heredoc. – Imtiaz

Drop the heredoc, and just use $(cat) and press Ctrl-D to send an EOF. This has the advantage of allowing "END_HEREDOC" to possibly occur in the text. – Luminance

What is the advantage of quoting 'END_HEREDOC'? – Snowclad

@HubertGrzeskowiak it prevents variable expansion. – Profluent

This is the shell-agnostic solution (if you care about portability) as read is a Bash built-in and not necessarily compatible with Zsh. – Cowen

$(cat <<-'}' -- terminate with indented }, followed by ) on the following line. – Surra

Here is the best solution for all – Garnishment

@RadonRosborough Getting "unexpected EOF" if there are UN-paired backticks happens in Bash v3.x. The newer versions 4.x and 5.0 don't have this issue. Unfortunately this matters because many Apple users are stuck with Bash 3.2.x now and in the future. (Apple just announce their new OS versions will NOT get Bash 4.x; instead zsh will be the default shell in 10.15+) – Atwitter

The other answer seems to provide a mostly compatible bash3 / macOS portable solution for this. – Tierney

Warning: the command substitution used in this solution strips trailing whitespace/newlines from the heredoc. Using read on the other hand does not. – Bandstand

You shouldn't use echo $VAR or echo "$VAR" because those may result in undefined behavior if the $VAR starts with a dash. Use printf "%s" "$VAR" or printf "%s\n" "$VAR" instead. – Conidium

@ResignedJune2023 Because paired backticks are (almost) equivalent to $( ). – Laurentium

Answer 3

this is variation of Dennis method, looks more elegant in the scripts.

function definition (bash version):

define(){ IFS=$'\n' read -r -d '' ${1} || true; }

usage:

define VAR <<'EOF'
abc 'asdf " \$ $ \n $'\n'
$(dont-exec ute-this)
foo " bar " ' ' 

`bad bad ``
EOF

echo "$VAR"

Updated ksh 'read loop' version (also works on bash):

function define { typeset a o=; while IFS= read -r a; do o+="$a"$'\n'; done; o=${o%?}; eval "$1=\$o"; }

Tests (also includes versions that does not remove the last newline): https://gist.github.com/ootada/e80b6d7fb86acdcda75d77eb7ade364c

enjoy

Revareval

This seems to work only superficially. The define function will return a status of 1, and I'm not quite sure what needs to be corrected. – Exactly

This is also superior to the accepted answer, because it can be modified to support POSIX sh in addition to bash (a read loop in the function, to avoid the -d '' bashism necessary to preserve newlines). – Venezuela

Unlike the cat-in-a-subshell option, this works with unpaired backticks in the heredoc. Thank you! – Imtiaz

This solution works with set -e set, whereas the selected answer does not. It seems to be because of http://unix.stackexchange.com/a/265151/20650 – Sheepshead

@Venezuela how would the read loop actually look like to avoid the -d ''? Since the beautiful answer of the OP certainly does not work on AIX. – Perdita

@Moreaki added 'read loop' version, but have no access to AIX for testing, so ymmv. – Revareval

@Exactly p.s. return status has long been fixed – Revareval

ShellCheck SC2141 says it should be define(){ IFS=$'\n' ... (added $) – Moeller

IFS='\n' doesn't what you want (it sets the IFS to a literal n after removing the backslash). You can fix that with a literal newline between the quotes, or in Bash with IFS=$'\n' as suggested in previous comments. – Maser

In addition to $'\n', the eval is a bit shady. especially since it's missing quotes. eval "$1=\"\$o\"" perhaps better to use printf -v instead of eval... as it is, this would be a security hole, and could $(insert executed code). – Christoforo

Answer 4

VAR=<<END
abc
END

doesn't work because you are redirecting stdin to something that doesn't care about it, namely the assignment

export A=`cat <<END
sdfsdf
sdfsdf
sdfsfds
END
` ; echo $A

works, but there's a back-tic in there that may stop you from using this. Also, you should really avoid using backticks, it's better to use the command substitution notation $(..).

export A=$(cat <<END
sdfsdf
sdfsdf
sdfsfds
END
) ; echo $A

Duralumin

I've updated my question to include $(executable). Also, how do you preserve newlines? – Bimonthly

@l0st3d: So close... Use $(cat <<'END' instead. @Neil: The very last newline will not be part of the variable, but the rest will be preserved. – Antibaryon

It doesn't seem like any newlines are preserved. Running the above example I see: "sdfsdf sdfsdf sdfsfds"... ah! But writing echo "$A" (i.e. putting $A in double quotes) and you do see the newlines! – Ventriloquy

@Darren: aha! I had noticed the newlines issue, and using the quotes around the output variable does fix the issue. thx! – Mussulman

@Duralumin I would just leave out mention of backticks. – Pony

You don't need to use cat inside the $(...) since << already sends the here document to stdout. So export A=$(<<END ... END) works just as well, with one less process running. – Herder

Interestingly, due to the quirk of the first example, in a pinch you can use it for makeshift comment blocks like this: REM=<< 'REM' ... comment block goes here ... REM. Or more compactly, : << 'REM' .... Where "REM" could be something like "NOTES" or "SCRATCHPAD", etc. – Georgiana

Answer 5

There is still no solution that preserves newlines.

This is not true - you're probably just being misled by the behaviour of echo:

echo $VAR # strips newlines

echo "$VAR" # preserves newlines

Automate

Really this is the behavior of how quoting a variable works. Without quotes, it will insert them as different parameters, space deliminated, while with quotes the entire variable contents will be treated as one argument – Lansing

Answer 6

Branching off Neil's answer, you often don't need a var at all, you can use a function in much the same way as a variable and it's much easier to read than the inline or read-based solutions.

$ complex_message() {
  cat <<'EOF'
abc'asdf"
$(dont-execute-this)
foo"bar"''
EOF
}

$ echo "This is a $(complex_message)"
This is a abc'asdf"
$(dont-execute-this)
foo"bar"''

Timer

This solution is truly awesome. By far the most elegant IMHO. – Crissum

Lots of options here. – Leilani

Answer 7

An array is a variable, so in that case mapfile will work

mapfile y <<'z'
abc'asdf"
$(dont-execute-this)
foo"bar"''
z

Then you can print like this

printf %s "${y[@]}"

Venetian

Use cases here for certain. – Leilani

Better version: https://mcmap.net/q/36790/-how-to-assign-a-heredoc-value-to-a-variable-in-bash – Magalymagan

Answer 8

I can't believe I'm the first to post this.

@Erman and @Zombo are close, but mapfile doesn't just read arrays...

Consider this:

#!/bin/bash
mapfile -d '' EXAMPLE << 'EOF'
Hello
こんにちは
今晩は
小夜なら
EOF
echo -n "$EXAMPLE"

Yielding:

Hello
こんにちは
今晩は
小夜なら

$'' is the delimiter given to mapfile, it will never occur, it means "not delimited".

So there's no need for a useless use of cat and no need to incur the penalty of recombining arrays.

Furthermore, you get this benefit:

$ echo $EXAMPLE
Hello こんにちは 今晩は 小夜なら

Which you do not receive with @Zombo's method:

mapfile y <<'z'
abc'asdf"
$(dont-execute-this)
foo"bar"''
z
echo $y

abc'asdf"

Bonus

If you run it through head -c -1 you can also get rid of that last newline in a way that won't be non-performant:

unset EXAMPLE
mapfile -d '' EXAMPLE < <(head -c -1 << EOF
Hello
こんにちは
今晩は
小夜なら
EOF
)
printf "%q" "$EXAMPLE"

$'Hello\nこんにちは\n今晩は\n小夜なら'

Magalymagan

mapfile -d requires bash 4.4 or higher. – Tetracaine

Oh, true. If the shell is older than bash 4.4 (Sat, 17 Sep 2016) this won't work, indeed. – Magalymagan

Apple only ships bash 3.2 from 2006 :) – Tetracaine

Good to know for those who use Apple computers. ;-) – Magalymagan

Answer 9

assign a heredoc value to a variable

VAR="$(cat <<'VAREOF'
abc'asdf"
$(dont-execute-this)
foo"bar"''
VAREOF
)"

used as an argument of a command

echo "$(cat <<'SQLEOF'
xxx''xxx'xxx'xx  123123    123123
abc'asdf"
$(dont-execute-this)
foo"bar"''
SQLEOF
)"

Reclinate

When I tried the first method, there seems to be no line terminators between the lines. Must be some kind of configuration on my linux machine? – Rampart

This probably means when you were echoing your variable, you didn't put quotes around it... Try it like so: echo "$VAR" – Preselector

Answer 10

Thanks to dimo414's answer, this shows how his great solution works, and shows that you can have quotes and variables in the text easily as well:

example output

$ ./test.sh

The text from the example function is:
  Welcome dev: Would you "like" to know how many 'files' there are in /tmp?

  There are "      38" files in /tmp, according to the "wc" command

test.sh

#!/bin/bash

function text1()
{
  COUNT=$(\ls /tmp | wc -l)
cat <<EOF

  $1 Would you "like" to know how many 'files' there are in /tmp?

  There are "$COUNT" files in /tmp, according to the "wc" command

EOF
}

function main()
{
  OUT=$(text1 "Welcome dev:")
  echo "The text from the example function is: $OUT"
}

main

Preselector

It would be interesting to see an unmatched quote in the text to see how it handles that. Maybe ` Don't freak out, there are "$COUNT" files` so the apostrophe/single quote can make things interesting. – Brufsky

Answer 11

I found myself having to read a string with NULL in it, so here is a solution that will read anything you throw at it. Although if you actually are dealing with NULL, you will need to deal with that at the hex level.

$ cat > read.dd.sh

read.dd() {
     buf= 
     while read; do
        buf+=$REPLY
     done < <( dd bs=1 2>/dev/null | xxd -p )

     printf -v REPLY '%b' $( sed 's/../ \\\x&/g' <<< $buf )
}

Proof:

$ . read.dd.sh
$ read.dd < read.dd.sh
$ echo -n "$REPLY" > read.dd.sh.copy
$ diff read.dd.sh read.dd.sh.copy || echo "File are different"
$ 

HEREDOC example (with ^J, ^M, ^I):

$ read.dd <<'HEREDOC'
>       (TAB)
>       (SPACES)
(^J)^M(^M)
> DONE
>
> HEREDOC

$ declare -p REPLY
declare -- REPLY="  (TAB)
      (SPACES)
(^M)
DONE

"

$ declare -p REPLY | xxd
0000000: 6465 636c 6172 6520 2d2d 2052 4550 4c59  declare -- REPLY
0000010: 3d22 0928 5441 4229 0a20 2020 2020 2028  =".(TAB).      (
0000020: 5350 4143 4553 290a 285e 4a29 0d28 5e4d  SPACES).(^J).(^M
0000030: 290a 444f 4e45 0a0a 220a                 ).DONE

Chianti

Answer 12

In many cases, this page has over-complex answers:

GNUV3="
    MyProg Copyright (C) 2024 NVRM 
    This program comes with ABSOLUTELY NO WARRANTY; for details type show w.
    This is free software, and you are welcome to redistribute it
    under certain conditions; type show c for details.
"

echo "$GNUV3"

Canzone

This will add an empty line before the string starts. To prevent this, add a backslash right after the opening quote: `GNUV3="\` – Etherege

Answer 13

This will work in any POSIX shell, avoids the issue of the non-zero exit status of read, and looks cleaner than an inline cat.

def() {
    eval "$1=\$(cat)"
}

def variable << '>>'
abc'asdf"
$(dont-execute-this)
foo"bar"''
>>

Caveat: This will strip newlines at the end of the here-doc. Newlines that are not at the end will be preserved.

Ellipsoid

Answer 14

Here's a way to do it that is (imho) quite elegant and avoids a UUOC:

  VAR=$(sed -e 's/[ ]*\| //g' -e '1d;$d' <<'--------------------'
      | 
      | <!DOCTYPE html>
      | <html>
      |   <head>
      |     <script src='script.js'></script>
      |   </head>
      |   <body>
      |     <span id='hello-world'></span>
      |   </body>
      | </html>
      | 
--------------------
    )

The '|' characters define the margin, and only the whitespace to the right of the margin is respected in the printed string. The '1d;$d' strips the first and last line, which are just added as a top and bottom margin around the content. Everything can be indented to whatever level you like, except the HEREDOC delimiter, which in this case is just a bunch of hyphens.

echo "$VAR"

# prints

<!DOCTYPE html>
<html>
  <head>
    <script src='script.js'></script>
  </head>
  <body>
    <span id='hello-world'></span>
  </body>
</html>

Josiah

Um, dude, good attempt but clearly compiling a regular expression and running a global replace is far more resource-hungry of an operation than a UUOC. The mapfile approach is best per appearance and performance, this one is way out there (why the margin??) – Magalymagan

Answer 15

$TEST="ok"
read MYTEXT <<EOT
this bash trick
should preserve
newlines $TEST
long live perl
EOT
echo -e $MYTEXT

Ostensorium

Even with the various errors fixed, this only places the first line of the here document in MYTEXT – Maser

Easy to fix, the idea is ok! I used read -d $'\000' MYTEXT <<EOT and worked well... – Izzo