Authors Top

If you have a few years of experience in the Linux ecosystem, and you’re interested in sharing that experience with the community, have a look at our Contribution Guidelines.

1. Overview

Many programs and services need their configuration to be written down in a structured way. One of the ways to achieve this goal is using YAML files.

In the Bash shell, we need tools to deal with YAML content from the command line or script. 

In this tutorial, we’re going to learn about the yq utility.

2. yq Basics

The yq command is usually not a part of standard Linux distribution, so we need to install it manually. Then, let’s go through the command’s basics.

2.1. Version Check

First, let’s check the version:

$ yq -V #or --version
yq (https://github.com/mikefarah/yq/) version 4.24.2

We should be aware that two widely used implementations of yq exist. This used throughout the tutorial is known as the ‘Go implementation’, while the second one is based on Python. Both have very similar capabilities but differ slightly in syntax.

For more details, we should visit the project’s documentation page.

2.2. The Default Working Mode

Let’s notice that the default yq mode is eval, which allows reading, searching, and editing YAML files. So, let’s print the whole content of the file personal_data.yaml:

$ yq personal_data.yaml

name: "My name"
surname: "Foo"
street_address:
  city: "My city"
  street: "My street"
  number: 1

3. Accessing Properties

We should use the path in the YAML file to retrieve the property’s value. First, let’s print the name property in personal_data.yaml:

$ yq '.name' personal_data.yaml

My name

Next, the brackets [] provide all values from the structure street_address:

$ yq '.street_address[]' personal_data.yaml
My city
My street
1

3.1. Searching for Value

We can use the select operator to find a particular value. So let’s look for ‘Foo’ in the YAML file:

$ yq '.[] | select(. == "Foo")' personal_data.yaml

Foo

Let’s notice the use of the pipe operator | to pass the root node’s values to the select operator.

Now let’s find all values which start with ‘My’ in the street_address node. So, we apply the wildcard ‘*’:

$ yq '.street_address[] | select(. == "My*")' personal_data.yaml

My city
My street

3.2. Recursive Search

With the double dot .. operator, we can recursively traverse the document, starting from the given node. Thus, let’s find all values starting with ‘My’:

$ yq '.. | select(. == "My*")' personal_data.yaml

My name
My city
My street

Next, let’s narrow our search only to the street_address node’s children

$ yq '.street_address | .. | select(. == "My*") ' personal_data.yaml

My city
My street

3.3. Changing Values

We can change or update properties using the assign operator ‘=’. Then, let’s change the street number:

$ yq '.street_address.number = 256' personal_data.yaml

name: "My name"
surname: "Foo"
street_address:
  city: "My city"
  street: "My street"
  number: 256

The result shows up in the standard output only. Thus, we need to use the i option to modify the file in place.

4. Working With the Nodes of YAML

We can query and modify the YAML structure as well. Hence, we can add, delete and find nodes.

4.1. Adding and Deleting Nodes

Let’s create a new property zip_code in the street_address by simply adding it to the path:

$ yq -i '.street_address.zip_code = 16' personal_data.yaml && cat personal_data.yaml

name: "My name"
surname: "Bar"
street_address:
  city: "My city"
  street: "My street"
  number: 1
  zip_code: 16

Next, in a similarly simple way, let’s remove the node with the del operator:

$ yq -i 'del(.street_address.zip_code)' personal_data.yaml && cat personal_data.yaml

name: "My name"
surname: "Bar"
street_address:
  city: "My city"
  street: "My street"
  number: 1

4.2. Retrieving Nodes’ Names

The to_entries operator returns keys together with their values. Then, let’s use it on the root level:

$ yq 'to_entries' personal_data.yaml

- key: name
  value: "My name"
- key: surname
  value: "Foo"
- key: street_address
  value:
    city: "My city"
    street: "My street"
    number: 1

The results are provided as array elements. Further, we can extract the keys:

$ yq 'to_entries | .[] | .key' personal_data.yaml

name
surname
street_address

and values:

$ yq 'to_entries | .[] | .value' personal_data.yaml

My name
Foo
city: "My city"
street: "My street"
number: 1

4.3. Searching for Nodes With has

Let’s try to find Bash entry in the language data file languages.yaml:

$ yq languages.yaml

languages:
  - language:
      name: Bash the Bourne-Again shell
      feature: interpreted
  - language:
      name: C++
      feature: compiled, comes together with Bash well
  - language:
      name: Java
      feauture: compiled and interpreted, different from Bash and C++

Obviously, we should not search file-wide for a pattern like *Bash*, because we’d obtain Java and C++ data as well. Thus, we’re going to search only the name nodes:

$ yq  '.. | select(has("name")) | select(.name == "*Bash*")' languages.yaml

name: Bash
feature: interpreted

Let’s notice the use of operator has. It returns true for matching nodes. Further, the search is refined by the next select, which checks the node’s content.

5. Working With Multiple Files

yq can work with multiple files, which are provided as arguments. Moreover, we can index these files and refer to them.

As an example, let’s amend languages.yaml with languages’ versions from versions.yaml file:

$ yq versions.yaml

Java: openjdk 11.0.14.1 2022-02-08
C++: gcc (GCC) 11.2.1 20220127 (Red Hat 11.2.1-9)
Bash: GNU bash, version 5.1.8(1)-release (x86_64-redhat-linux-gnu)

We need to turn into the eval-all mode to read both files into memory:

$ yq eval-all '
    select(fi == 0) as $versions |
    select(fi == 1) |                                
    .languages[0].language.version = $versions.Bash|
    .languages[1].language.version = $versions.C++ |
    .languages[2].language.version = $versions.Java
' versions.yaml languages.yaml

languages:
  - language:
      name: Bash the Bourne-Again shell
      feature: interpreted
      version: GNU bash, version 5.1.8(1)-release (x86_64-redhat-linux-gnu)
  - language:
      name: C++
      feature: compiled, comes together with Bash well
      version: gcc (GCC) 11.2.1 20220127 (Red Hat 11.2.1-9)
  - language:
      name: Java
      feature: compiled and interpreted, different from Bash and C++
      version: openjdk 11.0.14.1 2022-02-08

Let’s highlight the use of file index fi to select the appropriate file. Further, with as $versions, we defined a variable with the first YAML content. In addition, to add the version node, we referred to the languages array elements by index.

6. Interaction With a Bash Script

When programming in Bash, we need to use the language’s constructs to access the YAML structure.

6.1. Injecting Bash Variables Into a YAML File

Let’s update the configuration in the YAML file with the bash variables. Hence, we’re going to use the env operator. An example, let’s create YAML content with the hostname:

$ yq --null-input '.hostname = env(HOSTNAME)'

hostname: fedora35

Let’s notice the null-input switch, which tells the command to create the YAML content without an input file.

6.2. Variable’s Value as a Search Target

Now let’s search the YAML content for a value or node matching the variable’s value. So first, let’s find the ‘Foo’ value in personal_data.yaml:

$ targetVal=Foo yq '.[] | select(. == env(targetVal))' personal_data.yaml

Foo

Next, let’s use the variable as a lookup key to the keys’ table:

$ targetKey=name yq  ' .[env(targetKey)] ' personal_data.yaml

My name

Finally, let’s search for a node that contains the city key with has:

$ targetKey=city yq '.. | select(has(env(targetKey)))' personal_data.yaml

city: "My city"
street: "My street"
number: 1

6.3. Variable Substitution With envsubst

Let’s consider a simple YAML template to collect basic system information:

$ yq system_data.yaml

hostname: ${HOSTNAME}
user: ${USER}
shell: ${SHELL}

Now we’re going to replace all ${} placeholders with actual values. Thus, we should use envsubst inside yq:

$ yq '.[] |= envsubst' system_data.yaml

hostname: fedora35
user: joe
shell: /bin/bash

6.4. Reading YAML Content Into an Array

Now let’s access YAML inside the script. Hence, we’re going to use an associative array, where the node’s name is a key. So, let’s import the street_address from personal_data.yaml with the yaml_reader script:

#!/bin/bash

declare -A content

while IFS="=" read -r key value; do content["$key"]=$value; done < <(
  yq '.street_address | to_entries | map([.key, .value] | join("=")) | .[]' personal_data.yaml
)

for key in "${!content[@]}"; do printf "key %s, value %s\n" "$key" "${content[$key]}"; done

We used the while IFS loop to fill the array. Furthermore, the to_entries results fed the map operator to concatenate each key-value pair with the join operator to the IFS format.

Now let’s check the results:

$ ./yaml_reader

key city, value My city
key number, value 1
key street, value My street

7. Conclusion

In this tutorial, we learned about the yq command. First, we went through its basics. Then, we focused on searching and modifying the properties in the YAML file. Next, we studied the example of multifile operation.

Meanwhile, we highlighted the importance of chaining the yq‘s operator to effectively achieve our goals. It’s especially useful as the command offers a wide choice of operators.

Finally, we looked through ways to interact between the bash script and the YAML content.

Authors Bottom

If you have a few years of experience in the Linux ecosystem, and you’re interested in sharing that experience with the community, have a look at our Contribution Guidelines.

Comments are closed on this article!