prepend **heredoc** to a file, in bash

Question

I was using:

cat <<<"${MSG}" > outfile

to begin with writing a message to outfile, then further processing goes on, which appends to that outfile from my awk script.

But now logic has changed in my program, so I'll have to first populate outfile by appending lines from my awk program (externally called from my bash script), and then as a final step prepend that ${MSG} heredoc to the head of my outfile..

How could I do that from within my bash script, not awk script?

EDIT

this is MSG heredoc:

read -r -d '' MSG << EOF
-----------------------------------------------
--   results of processing - $CLIST
--   used THRESHOLD ($THRESHOLD)
-----------------------------------------------
l
EOF
# trick to pertain newline at the end of a message
# see here: http://unix.stackexchange.com/a/20042
MSG=${MSG%l}

Eparchy

NB: cat with a herestring? The portable way to do that would have been printf '%s\n' "$MSG" > outfile. Avoid bashisms where you can. – Blitzkrieg

FYI, what you're doing here is necessarily inefficient -- "necessarily" meaning that POSIX doesn't provide any way (in any programming language) to prepend content to a file without rewriting that whole file. If it's a large file, then, you should try to avoid any process which would have you doing this frequently. – Obstruent

@CharlesDuffy yes you are right, in my script this occures only once. I tried with that MSG writing being handled with awk, but, as that awk script gets called more than once from within my bash script, putting that MSG into BEGIN block wouldn't help, because I would have many instances, and I want only one at the top. – Eparchy

As an aside: you can make do without l trick for preserving trailing newlines if you instead use the following: IFS= read -r -d '' MSG << EOF .... – Spelldown

@Spelldown Would I have to revert it back to old IFS value, or it doesn't matter, because it would go with the script being finished? – Eparchy

No, there is NO need to restore the value, because prepending a variable assignment directly to a command that way localizes it to that command - that is (loosely speaking), it automatically reverts to the previous value when the command finishes. – Spelldown

Oh yes, I forgot that, silly me. Thanks, your tip saves me two lines of code :) – Eparchy

Answer 1

You can use awk to insert a multiline string at beginning of a file:

awk '1' <(echo "$MSG") file

Or even this echo should work:

echo "${MSG}$(<file)" > file

Jaipur

$MSG contains more than one line, it's a heredoc banner-like message – Eparchy

can you explain a bit, will this work only for that number of lines, so it will break if I change my $MSG to have more lines? – Eparchy

MSG variable can have any number of lines actually. awk's substr function has been used to strip off $' from front and \n' from end (those are added by printf "%q"). I have tested it on my OSX before posting the answer. – Jaipur

I lost my newline at the end of $MSG. Why is that, and how to keep it? – Eparchy

Sure just change substr to substr(m,3,length(m)-3) – Jaipur

I'm a little worried that this relies a bit much on implementation-defined behavior -- printf '%q' promises only to create a string that bash can parse, not to do it in any specific way. – Obstruent

Actually I couldn't think of any other way of passing a multiline string to awk earlier. But I have edited it now to remove printf "%q" – Jaipur

+1 for simple solutions, but you should note that (a) awk is not the right tool for this job in general - it makes sense here specifically, because of the OP's specific requirements, and (b) the echo solution should only be used for small files, given that the entire string is built up in memory (right?). Also, it sounds like $MSG is already \n-terminated, so you should use <(printf '%s' "$MSG"). – Spelldown

@CharlesDuffy: To finish the discussion, even though printf %q is no longer in the mix: I share your concern; for instance, if the string happens to contain no characters that need escaping (though it will in this case), the result of printf %q is not $'...'-enclosed. Note that GNU awk and mawk actually DO accept strings with embedded newlines on the command line - FreeBSD/OSX awk doesn't (which is actually the POSIX-compliant behavior). The awk-POSIX-compliant solution is to replace newlines with '\n' strings: `-v MSG="${MSG//$'\n'/\\n}". See goo.gl/uT1oh9 for more. – Spelldown

Highly underrated answer. Thank you! For the sake of completion, the echo version just goes to stdout - it doesn't rewrite the original file (per the question). The adjustment is simple: echo "${MSG}$(<file)" > file – Sty

Answer 2

Use a command group:

{
    echo "$MSG"
    awk '...'
} > outfile

If outfile already exists, you have no choice but to use a temporary file and copy it over the original. This is due to how files are implemented by all(?) file systems; you cannot prepend to a stream.

{
     # You may need to rearrange, depending on how the original
     # outfile is used.
     cat outfile
     echo "$MSG"
     awk '...'
} > outfile.new && mv outfile.new outfile

Another non-POSIX feature you could use with cat is process substitution, which makes the output of an arbitrary command look like a file to cat:

cat <(echo $MSG) outfile <(awk '...') > outfile.new && mv outfile.new outfile

Reber

I cannot do that, because this happens in a moment when I already have outfile generated. I have a case checking if outfile exists, if it does, than I need to prepend my $MSG – Eparchy

I did exactly that: cat <<< $MSG outfile > outfile.tmp && mv outfile.tmp outfile, but ended up with outfile being empty, and still having that outfile.tmp in my directory? That was the reason I posted question here ;) – Eparchy

my bad.. now I saw, entered > between mv outfile.tmp and outfile, overseeing that, led me to post here.. sorry for taking your time – Eparchy

don't know why, but cat <<< $MSG outfile > outfile.tmp && mv outfile.tmp outfile doesn't prepend $MSG, outfile is there, but no $MSG prepended? – Eparchy

<<< provides standard input, and cat ignores standard input when it receives one or more filename arguments. – Reber

Oh, that was the case. Thanks. – Eparchy

Answer 3

Use - as a placeholder on the cat command line for the point where you want new content inserted:

{ cat - old-file >new-file && mv new-file old-file; } <<EOF
header
EOF

Obstruent

Can that <<EOF part, be replaced with <<<"${MSG}", as I already have it that way? – Eparchy

+1 for the prefix form, but it's simpler to use cat <<EOF >>old-file for the suffix form. – Spelldown

@mklement0, indeed, that was silly. I've removed it. – Obstruent

@branquito, indeed, using a herestring there will be fine. – Obstruent

I like your answer the best, but not shure If would be now ok to accept it as I already accepted another one earlier, before you posted. – Eparchy

@branquito, there's nothing wrong with the current answer, even if this one is a little bit cleaner, so perfectly fine to keep it. – Obstruent

Yes, that's why I said, your solution looks neat. Anyway I learned a few new and important things from all of you. – Eparchy