1. Overview
We often use templates to create documents that follow the same pattern or rule. For example, we fill dynamic parts of a web template to generate various web pages.
Usually, templates are text files with some variables. When we want to create a concrete document from a template, we replace the variables in the template with their actual values.
In this tutorial, we’ll learn how to substitute shell variables with their values in a text file in the Linux command line.
2. Introduction to the Problem
As usual, let’s explain the problem through an example.
First, let’s have a look at our template file:
$ cat template.txt
Hey $USER,
Nice to see you on this server!
Your home directory is: $HOME
Our template.txt file is a plain text file. It’s a message for a login user on the server.
As we can see in the file content above, the template contains shell variables to make the message dynamic. Therefore, to generate a concrete message, we need to replace the shell variables in the template file with their actual values.
Next, let’s see how to generate a message from the template.
We will extend our current template file to cover new requirements in the latter part of this tutorial.
3. eval Is Not the Right Way to Go
We know that the eval command can construct a command by concatenating arguments.
Therefore, to solve the problem, an idea is to read each line from our template file and build a command eval echo “$line”. If “$line” contains shell variables, eval will evaluate them as their values.
3.1. Building a Command Using eval
Now, let’s implement the idea and test if it can create an expected message from the template file:
$ rm -f message.txt; while read line
do
eval echo "$line" >> message.txt
done < "template.txt"
$ cat message.txt
Hey kent,
Nice to see you on this server!
Your home directory is: /home/kent
We’ve built a command to put the eval command in a while loop to append each line to an output file called message.txt.
After the execution, as cat‘s output shows, the message.txt file contains the expected content. In addition, all shell variables have been replaced with their values.
3.2. Some Problems
It seems that we’ve found the solution to the problem.
However, the eval approach may have some problems. Let’s add two more lines to our template.txt file:
$ cat template.txt
...
# If you like using Java, the Java home is: $JAVA_HOME
Also, you can use command substitution: $(YOUR_CMD)
Now, let’s rerun our command, and see what message it will create:
$ rm -f message.txt; while read line
do
eval echo "$line" >> message.txt
done < "template.txt"
zsh: command not found: YOUR_CMD
As we can see, we got an error message when we execute the command. This is because we’ve added $(YOUR_CMD) in the template. It’s not a variable, so we expect to see it unchanged in the created message.txt file. However, eval will evaluate the concatenated segments. That is to say, it will really execute $(YOUR_CMD) as command substitution. But, of course, there is no YOUR_CMD command available.
We added an invalid command in this example and saw the error message. We’re lucky since the error doesn’t do harm to the system. However, imagine if we added some valid command, such as “rm *” – we may have an unexpected or undesired result. Therefore, using eval to solve this problem can be dangerous.
Next, although the command execution has an error, let’s take a look at the message.txt it has generated:
$ cat message.txt
Hey kent,
Nice to see you on this server!
Your home directory is: /home/kent
Also, you can use command substitution:As the output above shows, the last line has been truncated due to the error we’ve been talking about.
Apart from that, we can see that the line starting with ‘#‘ becomes a blank line. This is because, during the evaluation, eval treats lines starting with ‘#‘ as comments and evaluates them as empty.
Therefore, eval is not the right tool to solve this problem.
4. Using the envsubst Command
We’ve learned why eval isn’t the right tool to use for this problem. So now, let’s introduce the envsubst command.
The envsubst utility is a member of the gettext package, which is pre-installed by default on most modern Linux distributions.
envsubst can substitute environment variables in shell format strings. Its syntax is pretty simple:
envsubst <TEMPLATE_FILEThe command above will print the output to Stdout. If it’s required, we can redirect the output to a file:
envsubst <TEMPLATE_FILE >OUTPUT_FILEFor simplicity, we’ll print the output to Stdout in this tutorial.
4.1. Environment Variable Substitution
Next, let’s test the envsubst command on our template file and see if it can produce the expected result:
$ envsubst <template.txt
Hey kent,
Nice to see you on this server!
Your home directory is: /home/kent
# If you like using Java, the Java home is: /usr/lib/jvm/default
Also, you can use command substitution: $(YOUR_CMD)Great, it does the job! As we can see in the output above, all variables have been substituted by their values.
It’s worth mentioning that envsubst ignores substitutions done by a shell, such as ${variable-default}, $(command-list), or `command-list`, due to security reasons.
4.2. Non-Environment Variable Substitution in a Shell Script
If we review our template file, we may realize that it contains only environment variables so far, such as $USER and $HOME.
In practice, our template can have non-environment variables. Let’s add one more line to the template file:
$ cat template.txt
Hey $USER,
...
Also, you can use command substitution: $(YOUR_CMD)
If you have any problem, please contact our admin: <$ADMIN1> or <$ADMIN2>.
As we can see, we’ve introduced two new variables in the template: $ADMIN1 and $ADMIN2. They are not environment variables.
Further, in the real world, we’ll likely create a script to automatically generate messages from a template.
Now, let’s create a simple script and declare the two “ADMIN” variables:
$ cat gen.sh
#!/bin/bash
[email protected]
[email protected]
envsubst <template.txt
Next, let’s test our script:
$ ./gen.sh
Hey kent,
Nice to see you on this server!
Your home directory is: /home/kent
# If you like using Java, the Java home is: /usr/lib/jvm/default
Also, you can use command substitution: $(YOUR_CMD)
If you have any problem, please contact our admin: <> or <>.
As we can see, all environment variables have been replaced with their values. But the two “ADMIN” variables are empty.
This is because envsubst only works with environment variables.
The fix to the problem is to export the two variables to make them available for all child processes created from that shell:
$ cat gen.sh
#!/bin/bash
export [email protected]
export [email protected]
envsubst <template.txt
$ ./gen.sh
Hey kent,
...
If you have any problem, please contact our admin: <[email protected]> or <[email protected]>.
5. Conclusion
In this article, we’ve learned how to use envsubst to replace shell variables in a template through examples.
Also, we’ve discussed why eval isn’t the right tool to solve the problem.