How does "cat << EOF" work in bash?

Question

I needed to write a script to enter multi-line input to a program (psql).

After a bit of googling, I found the following syntax works:

cat << EOF | psql ---params
BEGIN;

`pg_dump ----something`

update table .... statement ...;

END;
EOF

This correctly constructs the multi-line string (from BEGIN; to END;, inclusive) and pipes it as an input to psql.

But I have no idea how/why it works, can some one please explain?

I'm referring mainly to cat << EOF, I know > outputs to a file, >> appends to a file, < reads input from file.

What does << exactly do?

And is there a man page for it?

Phenothiazine

That's probably a useless use of cat. Try psql ... << EOF ... See also "here strings". mywiki.wooledge.org/BashGuide/InputAndOutput?#Here_Strings – Eros

I'm surprised it works with cat but not with echo. cat should expect a file name as stdin, not a char string. psql << EOF sounds logical, but not othewise. Works with cat but not with echo. Strange behaviour. Any clue about that? – Jobie

Answering to myself: cat without parameters executes and replicates to the output whatever send via input (stdin), hence using its output to fill the file via >. In fact a file name read as a parameter is not a stdin stream. – Jobie

@Jobie echo just prints it's command line arguments while cat reads stding(when piped to it) or reads a file that corresponds to it's command line args – Thrombus

@DennisWilliamson the link you posted has changed to mywiki.wooledge.org/BashGuide/… – Dupleix

Answer 1

This is called heredoc format to provide a string into stdin. See https://en.wikipedia.org/wiki/Here_document#Unix_shells for more details.

---

From man bash:

Here Documents

This type of redirection instructs the shell to read input from the current source until a line containing only word (with no trailing blanks) is seen.

All of the lines read up to that point are then used as the standard input for a command.

The format of here-documents is:

          <<[-]word
                  here-document
          delimiter

No parameter expansion, command substitution, arithmetic expansion, or pathname expansion is performed on word. If any characters in word are quoted, the delimiter is the result of quote removal on word, and the lines in the here-document are not expanded. If word is unquoted, all lines of the here-document are subjected to parameter expansion, command substitution, and arithmetic expansion. In the latter case, the character sequence \<newline> is ignored, and \ must be used to quote the characters \, $, and `.

If the redirection operator is <<-, then all leading tab characters are stripped from input lines and the line containing delimiter. This allows here-documents within shell scripts to be indented in a natural fashion.

Pet

I was having the hardest time disabling variable/parameter expansion. All I needed to do was use "double-quotes" and that fixed it! Thanks for the info! – Amboceptor

Concerning <<- please note that only leading tab characters are stripped -- not soft tab characters. This is one of those rare case when you actually need the tab character. If the rest of your document uses soft tabs, make sure to show invisible characters and (e.g.) copy and paste a tab character. If you do it right, your syntax highlighting should correctly catch the ending delimiter. – Paleobiology

I don't see how this answer is more helpful than the ones below. It merely repeats information that can be found in other places (that have likely already been checked) – Unreeve

@BrDaHa, maybe it is not. Why the question? because of upvotes? it was the only one for several years. it's seen by comparing dates. – Depressor

This excerpt from a man page has taught me how to write beautiful yet easy to understand docs. Amazing! – Microgroove

@Xeoncross, what do you mean use "double-quotes"? – Donetsk

Here is answered how to disable the expansion, #17839815 – Donetsk

Answer 2

The cat <<EOF syntax is very useful when working with multi-line text in Bash, eg. when assigning multi-line string to a shell variable, file or a pipe.

Examples of cat <<EOF syntax usage in Bash:

1. Assign multi-line string to a shell variable

$ sql=$(cat <<EOF
SELECT foo, bar FROM db
WHERE foo='baz'
EOF
)

The $sql variable now holds the new-line characters too. You can verify with echo -e "$sql".

2. Pass multi-line string to a file in Bash

$ cat <<EOF > print.sh
#!/bin/bash
echo \$PWD
echo $PWD
EOF

The print.sh file now contains:

#!/bin/bash
echo $PWD
echo /home/user

3. Pass multi-line string to a pipe in Bash

$ cat <<EOF | grep 'b' | tee b.txt
foo
bar
baz
EOF

The b.txt file contains bar and baz lines. The same output is printed to stdout.

Hake

1. 1 and 3 can be done without cat; 2. Example 1 can be done with a simple multi-line string – Flu

Instead of creating another process with 'cat', why not use IFS='' read -r -d instead? – Pianola

worth noting that when using tee instead of cat, sudo can be used to write to a file at a restricted location. Like e.g. sudo tee /etc/somepath/file > /dev/null <<EOF ... – Mir

I think cat is only for complete bash command with here-document, if without cat or other commands, the here-document started with symbol << can not echo to the stdout, and the rest of the first line "grep 'b' | tee b.txt" could not get input. – Betancourt

Why does this answer have any upvotes at all? You just completely ignored the question, which was "how/why it works". Nobody asked for a list of random examples, and yet somehow it has more upvotes than the (correctly) accepted answer. – Biotite

@Biotite There was already an accepted answer when I added these extra examples. It turns out they were quite useful based on the number of upvotes. Peace. – Hake

Answer 3

In your case, "EOF" is known as a "Here Tag". Basically <<Here tells the shell that you are going to enter a multiline string until the "tag" Here. You can name this tag as you want, it's often EOF or STOP.

Some rules about the Here tags:

1. The tag can be any string, uppercase or lowercase, though most people use uppercase by convention.
2. The tag will not be considered as a Here tag if there are other words in that line. In this case, it will merely be considered part of the string. The tag should be by itself on a separate line, to be considered a tag.
3. The tag should have no leading or trailing spaces in that line to be considered a tag. Otherwise it will be considered as part of the string.

example:

$ cat >> test <<HERE
> Hello world HERE <-- Not by itself on a separate line -> not considered end of string
> This is a test
>  HERE <-- Leading space, so not considered end of string
> and a new line
> HERE <-- Now we have the end of the string

Truman

this is the best actual answer ... you define both and clearly state the primary purpose of the use instead of related theory ... which is important but not necessary ... thanks - super helpful – Anneal

@Truman you must add that when <<- is used leading tab will not prevent the tag from being recognized – Thrombus

your answer clicked me on "you are going to enter a multiline string" – Contingence

Answer 4

Quotes prevent parameter expansion

Without quotes:

a=0
cat <<EOF
$a
EOF

Output:

0

With quotes:

a=0
cat <<'EOF'
$a
EOF

or (ugly but valid):

a=0
cat <<E"O"F
$a
EOF

Outputs:

$a

Hyphen removes leading tabs

Without hyphen:

cat <<EOF
<tab>a
EOF

where <tab> is a literal tab, and can be inserted with Ctrl + V <tab>

Output:

<tab>a

With hyphen:

cat <<-EOF
<tab>a
<tab>EOF

Output:

a

This exists of course so that you can indent your cat like the surrounding code, which is easier to read and maintain. E.g.:

if true; then
    cat <<-EOF
    a
    EOF
fi

Unfortunately, this does not work for space characters: POSIX favored tab indentation here. Yikes.

POSIX 7

kennytm quoted man bash, but most of that is also POSIX 7: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_07_04 :

The redirection operators "<<" and "<<-" both allow redirection of lines contained in a shell input file, known as a "here-document", to the input of a command.

The here-document shall be treated as a single word that begins after the next <newline> and continues until there is a line containing only the delimiter and a <newline>, with no <blank> characters in between. Then the next here-document starts, if there is one. The format is as follows:

[n]<<word
    here-document
delimiter

where the optional n represents the file descriptor number. If the number is omitted, the here-document refers to standard input (file descriptor 0).

If any character in word is quoted, the delimiter shall be formed by performing quote removal on word, and the here-document lines shall not be expanded. Otherwise, the delimiter shall be the word itself.

If no characters in word are quoted, all lines of the here-document shall be expanded for parameter expansion, command substitution, and arithmetic expansion. In this case, the <backslash> in the input behaves as the <backslash> inside double-quotes (see Double-Quotes). However, the double-quote character ( '"' ) shall not be treated specially within a here-document, except when the double-quote appears within "$()", "``", or "${}".

If the redirection symbol is "<<-", all leading <tab> characters shall be stripped from input lines and the line containing the trailing delimiter. If more than one "<<" or "<<-" operator is specified on a line, the here-document associated with the first operator shall be supplied first by the application and shall be read first by the shell.

When a here-document is read from a terminal device and the shell is interactive, it shall write the contents of the variable PS2, processed as described in Shell Variables, to standard error before reading each line of input until the delimiter has been recognized.

Jorgensen

In your last example discussing <<- and <tab>a, it should be noted that the purpose was to allow normal indentation of code within the script while allowing heredoc text presented to the receiving process to begin in column 0. It is a not too commonly seen feature and a bit more context may prevent a good deal of head-scratching... – Vivianna

How should i escape expension if some of the content in between my EOF tags needs to be expanded and some don't? – Oven

...just use the backslash in front of the $ – Oven

@JeanmichelCote I don't see a better option :-) With regular strings you can also consider mixing up quotes like "$a"'$b'"$c", but there is no analogue here AFAIK. – Jorgensen

I have used <<- in my script but it does not ignores the leading tab characters. – Stunk

More on this, which character(s) needs to be escaped with this? (of course as you said the dollar sign $, but what else) – Crotty

@Crotty without quotes I think it is just like inside double quote "" strings except that " double quotes themselves don't need to be escaped, as more or less addressed on the "If no characters in word are quoted" paragraph. – Jorgensen

You can also prevent parameter expansion with a backslash: cat <<\EOF. I personally find this cleaner and with a more obvious intent, as it's inline with the general convention of using backslashes to escape things. – Commonly

Answer 5

Using tee instead of cat

Not exactly as an answer to the original question, but I wanted to share this anyway: I had the need to create a config file in a directory that required root rights.

The following does not work for that case:

$ sudo cat <<EOF >/etc/somedir/foo.conf
# my config file
foo=bar
EOF

because the redirection is handled outside of the sudo context.

I ended up using this instead:

$ sudo tee <<EOF /etc/somedir/foo.conf >/dev/null
# my config file
foo=bar
EOF

Pagel

in your case use sudo bash -c 'cat <<EOF >/etc/somedir/foo.conf # my config file foo=bar EOF' – Abixah

@Abixah how about POSIX shell? any convention of sudo bash -c? – Crotty

Thanks! That solved my confusion! Just to point out that tee -a file << EOF will append the multiline text to the file and not adding >/dev/null will print out the multiline text to the console – Crotty

Answer 6

<< EoF basically means:

<< - "read the multi-line input that begins from the next line onward, and treat it as if it's code in a separate file"

EoF - "stop reading immediately after the word EoF is found in the multi-line input"

As other answers have explained, the multi-line input is called a Here Document (i.e. it's a document that's being provided 'inline' or 'on the spot').

A Here Document is often used to generate output to be passed to a subsequent process. For example cat << EoF can be used to generate a desired output, using a Here Document.

Here's an example of using a Here Document to create a text document on the fly:

cat << EoF > ./my-document.txt
Hello world
Have a nice day
EoF

Eidetic

Answer 7

A little extension to the above answers. The trailing > directs the input into the file, overwriting existing content. However, one particularly convenient use is the double arrow >> that appends, adding your new content to the end of the file, as in:

cat <<EOF >> /etc/fstab
data_server:/var/sharedServer/authority/cert /var/sharedFolder/sometin/authority/cert nfs
data_server:/var/sharedServer/cert   /var/sharedFolder/sometin/vsdc/cert nfs
EOF

This extends your fstab without you having to worry about accidentally modifying any of its contents.

Thimble

Answer 8

note to mention that cat << \EOT (see the backslash) will not expand any variables inside, while cat << EOT will do.

examples:

FOO="bar"

cat << \EOT > foobar.txt
echo "$FOO"
EOT

will output: echo $FOO

while:

FOO="bar"

cat << EOT > foobar.txt
echo "$FOO"
EOT

will output: echo "bar"

Waverly

Answer 9

Example to create a json file:

cat << EoF > ./allaccess.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:*"
      ],
      "Resource": [
        "arn:aws:s3:::*"
      ]
    }
  ]
}
EoF

As a result:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:*"
      ],
      "Resource": [
        "arn:aws:s3:::*"
      ]
    }
  ]
}

Bemock

Answer 10

Long story short, EOF marker(but a different literal can be used as well) is a heredoc format that allows you to provide your input as multiline. A lot of confusion comes from how cat actually works it seems. You can use cat with >> or > as follows:

$ cat >> temp.txt
line 1
line 2

While cat can be used this way when writing manually into console, it's not convenient if I want to provide the input in a more declarative way so that it can be reused by tools and also to keep indentations, whitespaces, etc.
Heredoc allows to define your entire input as if you are not working with stdin but typing in a separate text editor. This is what Wikipedia article means by:

it is a section of a source code file that is treated as if it were a separate file.

Dorthydortmund

Answer 11

This isn't necessarily an answer to the original question, but a sharing of some results from my own testing. This:

<<test > print.sh
#!/bin/bash
echo \$PWD
echo $PWD
test

will produce the same file as:

cat <<test > print.sh
#!/bin/bash
echo \$PWD
echo $PWD
test

So, I don't see the point of using the cat command.

Gian

which shell? I tested with bash 4.4 on Ubuntu 18.04 as well as bash 3.2 on OSX. Both created an empty file when just using <<test without cat <<test. – Narrate

Answer 12

@kennytm is correct that this is known as a heredoc pattern (link).

There's no need for cat if you're writing to file. Use tee instead:

tee filename.txt <<EOF
line1
line2
line3
...
EOF

Hungry

Answer 13

Worth noting that here docs work in bash loops too. This example shows how-to get the column list of table:

export postgres_db_name='my_db'
export table_name='my_table_name'

# start copy 
while read -r c; do test -z "$c" || echo $table_name.$c , ; done < <(cat << EOF | psql -t -q -d $postgres_db_name -v table_name="${table_name:-}"
SELECT column_name
FROM information_schema.columns
WHERE 1=1
AND table_schema = 'public'
AND table_name   =:'table_name'  ;
EOF
)
# stop copy , now paste straight into the bash shell ...

output: 
my_table_name.guid ,
my_table_name.id ,
my_table_name.level ,
my_table_name.seq ,

or even without the new line

while read -r c; do test -z "$c" || echo $table_name.$c , | perl -ne 
's/\n//gm;print' ; done < <(cat << EOF | psql -t -q -d $postgres_db_name -v table_name="${table_name:-}"
 SELECT column_name
 FROM information_schema.columns
 WHERE 1=1
 AND table_schema = 'public'
 AND table_name   =:'table_name'  ;
 EOF
 )

 # output: daily_issues.guid ,daily_issues.id ,daily_issues.level ,daily_issues.seq ,daily_issues.prio ,daily_issues.weight ,daily_issues.status ,daily_issues.category ,daily_issues.name ,daily_issues.description ,daily_issues.type ,daily_issues.owner

Swallowtail