How to Unescape JSON String Using jq

1. Introduction

Working with JSON (JavaScript Object Notation) is a common task in modern software development, especially when exchanging data between systems. However, JSON strings can sometimes include escaped characters or embedded JSON structures, making them harder to read and manipulate.

The jq tool simplifies this task by allowing the unescaping or decoding of these strings. Through its fromjson function, jq can parse JSON strings and return their content in a readable format.

In this tutorial, we’ll explore how to unescape JSON strings using jq, illustrating the process with practical examples and real-world applications.

2. Understanding JSON Escaping

Before diving into how jq can unescape JSON strings, it’s important to understand what the concept of escaping means. We use escaping in JSON by adding a backslash (\) before certain characters, such as double quotes (“) or newlines (\n), to ensure that JSON parsers interpret these characters as part of the string rather than as JSON syntax.

Let’s view a simple example of an unescaped JSON string and subsequently illustrate escaping it:

{
  "package": {
    "tracking": "TRK123456789",
    "date": "2024-12-15"
  }
}

Next, we’ll escape the JSON data above using a free online tool that converts it to an escaped JSON string:

"{ \"package\": { \"tracking\": \"TRK123456789\", \"date\": \"2024-12-15\" } }"

This JSON object is in string format, with the double quotes (“) around keys and values escaped by backslashes (\). Notably, enclosing the entire JSON data in double quotes is important, as it converts it into a JSON string, making it easier to embed within other strings.

Another common format is having a JSON-encoded string as a value for a key object within JSON data:

{
  "package": "{ \"tracking\": \"TRK123456789\", \"date\": \"2024-12-15\" }" 
}

This is a JSON object containing an escaped JSON string field, a format commonly used when encoding escaped JSON strings within another JSON string.

This is where jq plays a role in assisting with unescaping and decoding this structure.

3. Steps to Unescaping JSON Strings Using jq

First, we need to install the jq tool if we don’t have it already installed on the system. Next, let’s create the file delivery.json containing the previous JSON string:

$ cat delivery.json 
"{ \"package\": { \"tracking\": \"TRK123456789\", \"date\": \"2024-12-15\" } }"

As seen above, the delivery.json file contains a JSON string with escaped fields.

For the JSON object containing an escaped JSON string field, let’s create a file named part_delivery.json:

$ cat part_delivery.json 
{ "package": "{ \"tracking\": \"TRK123456789\", \"date\": \"2024-12-15\" }" }

As shown above, the value of the package JSON object is an escaped JSON string containing the tracking and date fields.

So, to unescape the JSON string, we can use the fromjson function of the jq tool. In this case, the function tells jq to interpret the string as JSON.

3.1. Unescaping JSON String

We can extract all the fields from JSON data at once using a combination of the jq and fromjson commands. Let’s apply these commands to the delivery.json file:

$ jq fromjson delivery.json 
{
  "package": {
    "tracking": "TRK123456789",
    "date": "2024-12-15"
  }
}

The jq tool and its fromjson function read the content of the JSON string file and decode the JSON-encoded string back into a JSON object.

The fromjson filter processes a JSON-encoded string, identifies the escaped characters, and removes the backslashes used to escape special characters, such as double quotes.

To select a particular field out of a JSON string, we unescape the string using jq‘s fromjson command and use the dot notation method for selection:

$ jq 'fromjson | .package' delivery.json 
{
  "tracking": "TRK123456789",
  "date": "2024-12-15"
}

Here, we used the dot notation to extract the unescaped values of the package key field after using the fromjson function to decode the JSON string back into a JSON object.

3.2. Unescaping a JSON Object Containing an Escaped JSON String Field

With little adjustment, the jq command and fromjson function can be used to unescape JSON objects containing an escaped JSON string field.

First, we identify the value of a key JSON object within the JSON string and unescape it using the fromjson function. Let’s illustrate a practical way of doing this.

$ jq '.package | fromjson' part_delivery.json 
{
  "tracking": "TRK123456789",
  "date": "2024-12-15"
}

The .package part of the command filters out the JSON string value for the package key. The output is then piped into the fromjson function, which unescapes the result and returns it in JSON format.

4. Real-World Applications of JSON Unescaping

In real-world scenarios, developers find the ability to unescape JSON strings particularly valuable where systems transmit or store JSON data in escaped form.

4.1. API Responses

Many APIs return JSON strings as part of their response. In APIs, especially those designed to handle complex data types, strings are mostly escaped.

For instance, when we work with cloud services or data-processing platforms, these systems might return JSON-encoded responses that include nested JSON strings requiring decoding.

Here’s an example of a JSON API response:

{
  "data": {
    "user": "John Doe",
    "actions": [
      {
        "action": "login",
        "time": "2024-09-10"
      }
    ]
  }
}

Now, we convert the JSON data into a JSON object containing an escaped JSON string field, and save it to data.json:

$ cat data.json
{
  "data": "{\"user\":\"John Doe\",\"actions\":[{\"action\":\"login\",\"time\":\"2024-09-10\"}]}"
}

Next, let’s illustrate the use of jq and its fromjson function to unescape and extract information from the JSON string:

$ jq  '.data | fromjson' data.json
{
  "user": "John Doe",
  "actions": [
    {
      "action": "login",
      "time": "2024-09-10"
    }
  ]
}

As shown above, we use the fromjson function of the jq tool to unescape the data and reveal the user’s details and actions.

4.2. Database JSON Fields

Another example is JSON data from databases. In some cases, systems store JSON data as strings in fields, either for compatibility reasons or because of limitations in the database engine.

For example, let’s examine a Logs table that holds each log entry as a JSON string. Here’s an example of a log entry in JSON format:

{
  "log": {
    "timestamp": "2024-09-15T08:30:00Z",
    "event": "server_restart"
  }
}

This is log data that holds information on when a server restarted.

We converted and escaped the nested value into a JSON-encoded string and saved it as log.json:

$ cat log.json
{
  "log": "{\"timestamp\":\"2024-09-15T08:30:00Z\",\"event\":\"server_restart\"}"
}

Similarly, we can apply jq with the fromjson function to unescape and access the data within for further analysis:

$ jq -c '.log | fromjson' log.json
{
  "timestamp":"2024-09-15T08:30:00Z",
  "event":"server_restart"
}

Here, we unescape and extract the values of the log field from the escaped raw data. Now, we can easily automate, manipulate, or analyze the unescaped data.

5. Conclusion

In this article, we’ve explored the process of unescaping JSON strings with jq using its fromjson function. This is an essential skill for developers working with JSON data stored or transmitted in escaped formats. The versatility of jq makes it an invaluable tool for handling JSON data across various real-world applications.