Why the Value of a Variable Modified in a while Loop Is Not Remembered Outside

1. Introduction

Dealing with variable scope can sometimes lead to unexpected behavior in a Bash script. One such issue arises when modifying a variable inside a while loop that reads input from a command pipeline. The variable value is forgotten after the loop exits, as though a copy of the variable is being used within the loop instead of modifying the original.

In this tutorial, we’ll explore why the value of a variable modified in a while loop isn’t remembered outside.

Firstly, we’ll discuss this issue using a test program. After that, we’ll discuss how to use here string and here document to overcome this challenge. Moreover, we’ll look at using an explicit file descriptor to ensure that variable modifications persist outside the while loop.

2. Test Code

Let’s consider the whileVariable.sh script:

$ cat whileVariable.sh
#!/bin/bash
set -e
set -u
foo=0
bar="hello"
if [[ "$bar" == "hello" ]]
    then
    foo=1
    echo "Setting \$foo to 1: $foo"
fi
echo "Variable \$foo after if statement: $foo"
lines="first line\nsecond line\nthird line"
echo -e $lines | while read line
do
    if [[ "$line" == "second line" ]]
        then
        foo=2
        echo "Variable \$foo updated to $foo inside if inside while loop"
    fi
    echo "Value of \$foo in while loop body: $foo"
done
echo "Variable \$foo after while loop: $foo"

The intuitively expected outcome is that the variable foo retains the value 2 after the while loop. However, foo doesn’t retain the value in the output:

Setting $foo to 1: 1
Variable $foo after if statement: 1
Value of $foo in while loop body: 1
Variable $foo updated to 2 inside if inside while loop
Value of $foo in while loop body: 2
Value of $foo in while loop body: 2
Variable $foo after while loop: 1

Despite the updates inside the loop, foo reverts to its previous value of 1 after the loop ends.

This behavior occurs because of how Bash handles pipelines. When we use echo -e $lines | while read line, the while loop executes in a subshell.

A subshell is a separate child process created from the parent shell. Moreover, any changes to variables in a subshell aren’t reflected in the parent shell. Therefore, foo retains its previous value after the loop.

To verify this behavior, we print the process IDs inside and outside the loop in whileVariable.sh. The output shows different process IDs, proving that the while loop executes in a separate process.

3. Using a Here String

To ensure that the variable value persists outside a loop, one of the approaches is to use a here string.

In particular, instead of piping the output of echo -e $lines, we use a here string <<< and modify the end of the original whileVariable.sh to whileVariableHereString.sh:

$ cat whileVariableHereString.sh
…
lines=$'first line\nsecond line\nthird line'
while read line
do
    if [[ "$line" == "second line" ]]
    then
        foo=2
        echo "Variable \$foo updated to $foo inside if inside while loop"
    fi
    echo "Value of \$foo in while loop body: $foo"
    done <<< "$lines"
echo "Variable \$foo after while loop: $foo"

Let’s execute the script to see if the variable retains its value:

Setting $foo to 1: 1
Variable $foo after if statement: 1
Value of $foo in while loop body: 1
Variable $foo updated to 2 inside if inside while loop
Value of $foo in while loop body: 2
Value of $foo in while loop body: 2
Variable $foo after while loop: 2

Once we execute the script, we can see that variable changes inside the loop are now preserved. To elaborate, <<< prevents while from executing in a subshell, which is the main issue in the original script.

4. Using a Here Document

Another way to read multiple lines of input while preserving variable changes in a shell loop is to use a here document. This method provides a way to pass input directly into a loop, without relying on external commands like echo.

Let’s look at the change we made to the whileVariableHereString.sh to use <<EOT in a new whileVariableHereDoc.sh script:

$ cat whileVariableHereDoc.sh
...
while read line
do
    if [[ "$line" == "second line" ]]
    then
        foo=2
        echo "Variable \$foo updated to $foo inside if inside while loop"
    fi
    echo "Value of \$foo in while loop body: $foo"
done <<EOT
first line
second line
third line
EOT
echo "Variable \$foo after while loop: $foo"

In this example, we define the input directly inside the script using the <<EOT syntax. We can see that the code retains the value of the variable outside the loop:

Setting $foo to 1: 1
Variable $foo after if statement: 1
Value of $foo in while loop body: 1
Variable $foo updated to 2 inside if inside while loop
Value of $foo in while loop body: 2
Value of $foo in while loop body: 2
Variable $foo after while loop: 2

By doing this, we avoid creating an external process and ensure that the while loop runs within the same shell. Thus, the script preserves the value of foo even after the loop completes.

5. Using an Explicit File Descriptor

One more similar approach for preserving the variable value outside a loop is to use file redirection. This method avoids a subshell by passing the output of a command directly to the while loop.

Let’s look at the modified code:

$ cat whileVariableFileRedirect.sh
...
lines="first line
second line
third line"

while read line
do
    if [[ "$line" == "second line" ]]
    then
        foo=2
        echo "Variable \$foo updated to $foo inside if inside while loop"
    fi
    echo "Value of \$foo in while loop body: $foo"
done < <(echo -e "$lines")

echo "Variable \$foo after while loop: $foo"

In the above approach, we use echo -e “$lines” to output the lines stored in the lines variable. Moreover, we use < <() to redirect this output into the while loop. One of the key advantages of this method is that it prevents the loop from running in a subshell.

6. Conclusion

In this article, we covered how to retain the variable value outside a loop.

We discussed that the issue of variable scope inside a while loop in Bash is often caused by the subshell execution when using the pipe | operator. Since subshells don’t share variables with the parent shell, the parent shell doesn’t reflect the changes from within the while loop.

To target this issue, we explored here strings and here documents as a solution. Lastly, we discussed using file descriptors to preserve the variable value.