How to pipe input to a Bash while loop and preserve variables after loop ends

Question

Bash allows to use: cat <(echo "$FILECONTENT")

Bash also allow to use: while read i; do echo $i; done </etc/passwd

to combine previous two this can be used: echo $FILECONTENT | while read i; do echo $i; done

The problem with last one is that it creates sub-shell and after the while loop ends variable i cannot be accessed any more.

My question is:

How to achieve something like this: while read i; do echo $i; done <(echo "$FILECONTENT") or in other words: How can I be sure that i survives while loop?

Please note that I am aware of enclosing while statement into {} but this does not solves the problem (imagine that you want use the while loop in function and return i variable)

Transnational

mywiki.wooledge.org/BashFAQ/024 – Victual

Related: #37229558 . Explains all options including the below-mentioned process substitution and lastpipe and their pros and cons. – Swanhildas

Related: A variable modified inside a while loop is not remembered. – Morgenthaler

@Morgenthaler there is more the one dup... Related: Why piping input to "read" only works when fed into "while read ..." construct? and Read values into a shell variable from a pipe – Dopp

Answer 1

The correct notation for Process Substitution is:

while read i; do echo $i; done < <(echo "$FILECONTENT")

The last value of i assigned in the loop is then available when the loop terminates. An alternative is:

echo $FILECONTENT | 
{
while read i; do echo $i; done
...do other things using $i here...
}

The braces are an I/O grouping operation and do not themselves create a subshell. In this context, they are part of a pipeline and are therefore run as a subshell, but it is because of the |, not the { ... }. You mention this in the question. AFAIK, you can do a return from within these inside a function.

---

Bash also provides the shopt builtin and one of its many options is:

lastpipe

If set, and job control is not active, the shell runs the last command of a pipeline not executed in the background in the current shell environment.

Thus, using something like this in a script makes the modfied sum available after the loop:

FILECONTENT="12 Name
13 Number
14 Information"
shopt -s lastpipe   # Comment this out to see the alternative behaviour
sum=0
echo "$FILECONTENT" |
while read number name; do ((sum+=$number)); done
echo $sum

Doing this at the command line usually runs foul of 'job control is not active' (that is, at the command line, job control is active). Testing this without using a script failed.

Also, as noted by Gareth Rees in his answer, you can sometimes use a here string:

while read i; do echo $i; done <<< "$FILECONTENT"

This doesn't require shopt; you may be able to save a process using it.

Tarantella

Pardon my ignorance. I know this is right solution and I've marked it as answer so it worked for me. But now when I run while read i; do echo $i; done < <(cat /etc/passwd); echo $i It did not return last line two times. What I am doing wrong? – Transnational

@WakanTanka: I had to experiment a bit...I believe the answer is that the failing read resets i to empty, so the echo after the loop echoes a blank line. – Tarantella

Kudos for the Process Substitution reference. I was unaware of. – Posturize

@Wakan Tanka: got same result as yours, i use while read i; do x=$i; done < <(cat /etc/passwd); echo i=$i; echo x=$x worked, //maybe those days bash behavior changed ? – Kloster

You're a life saver! Been looking for hours for a similar solution. Yours is the only one that worked. – Chouest

I'm getting shopt: lastpipe: invalid shell option name in GNU bash, version 5.2.15(1)-release – Chert

Curious: the manual still documents it. – Tarantella

Answer 2

Jonathan Leffler explains how to do what you want using process substitution, but another possibility is to use a here string:

while read i; do echo "$i"; done <<<"$FILECONTENT"

This saves a process.

Coptic

Answer 3

This function makes duplicates $NUM times of jpg files (bash)

function makeDups() {
NUM=$1
echo "Making $1 duplicates for $(ls -1 *.jpg|wc -l) files"
ls -1 *.jpg|sort|while read f
do
  COUNT=0
  while [ "$COUNT" -le "$NUM" ]
  do
    cp $f ${f//sm/${COUNT}sm}
    ((COUNT++))
  done
done
}

Mantinea

Answer 4

The fastest was:

while read -r ITEM; do
    OPERATION
done < <(COMMAND)

Or in one line:

while read -r ITEM; do OPERATION; done < <(COMMAND)

Fraya

Answer 5

Bash provides var default behavior so:

while read -r line
do
  total=${total:-0}
  total=$((total + $(calculation code)))
done
echo $total

This allows the loop to easily capture stdin, for scripts and functions.

Tippets