How to Get JSON Response With Selenium

1. Overview

In web development, it’s not uncommon for web pages to fetch data dynamically from APIs using JavaScript. At times, we may want to capture the JavaScript Object Notation (JSON) response returned by these APIs to automate data extraction, validate data, or perform tests. For such a task, we can use a tool like Selenium WebDriver to capture the JSON response. However, this isn’t always straightforward.

In this tutorial, we’ll use a simple and practical approach to get JSON responses using Selenium.

2. Basics and Prerequisites

Selenium is designed for browser automation and usually interacts with elements such as buttons, input fields, and links. However, it doesn’t directly provide APIs to capture raw network responses.

To demonstrate a workaround, we can use Selenium to perform several tasks in succession:

1. load the page
2. wait for the JSON to be displayed on the page
3. use the driver to extract the content

To specify, the approach works for JSON returned via fetch or XHR that the page renders.

Before we proceed, we need to ensure several prerequisites are installed:

• Java 11 or higher
• Maven
• Chrome browser
• ChromeDriver matching the Chrome version

Now, let’s proceed to build a simple HTML page that fetches JSON objects, so we can capture the JSON using Java Selenium.

3. Project Setup

For this project, we use Maven to manage dependencies.

To begin with, let’s create a Maven project structure selenium-json-demo:

$ mvn archetype:generate -DgroupId=com.example -DartifactId=selenium-json-demo -DarchetypeArtifactId=maven-archetype-quickstart -DinteractiveMode=false
[INFO] Scanning for projects...
...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  4.150 s
[INFO] Finished at: 2025-09-02T08:01:51+03:00
[INFO] ------------------------------------------------------------------------

Once the command above creates the Maven project structure successfully, let’s navigate to it:

$ cd selenium-json-demo

In the project directory, we update the pom.xml file appropriately:

<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
                             http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.example</groupId>
    <artifactId>selenium-json-demo</artifactId>
    <packaging>jar</packaging>
    <version>1.0-SNAPSHOT</version>
    <name>selenium-json-demo</name>
    <url>http://maven.apache.org</url>

    <properties>
        <maven.compiler.source>11</maven.compiler.source>
        <maven.compiler.target>11</maven.compiler.target>
    </properties>

    <dependencies>
        <!-- Selenium WebDriver -->
        <dependency>
            <groupId>org.seleniumhq.selenium</groupId>
            <artifactId>selenium-java</artifactId>
            <version>4.25.0</version>
        </dependency>
        <!-- JUnit for testing (optional) -->
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>3.8.1</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.codehaus.mojo</groupId>
                <artifactId>exec-maven-plugin</artifactId>
                <version>3.5.1</version>
                <configuration>
                    <mainClass>com.example.App</mainClass>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

The pom.xml file sets up the Java 11 compiler, Selenium WebDriver dependency, and Maven exec plugin to run the main class. In this configuration, the <dependency> tags pull the Selenium library for driving the browser and JUnit, if we want to implement test cases later. Meanwhile, the <build> section adds the exec-maven-plugin, enabling us to run the main class (com.example.App) with mvn exec:java. Without this plugin, Maven doesn’t know how to launch the Selenium code directly from the command line.

4. Writing the HTML Test Page

In the project directory, let’s create a simple HTML page test.html:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Test JSON Fetch</title>
</head>
<body>
    <h1>Testing JSON Fetch</h1>
    <pre id="output"></pre>

    <script>
        fetch('https://jsonplaceholder.typicode.com/todos/1')
            .then(response => response.json())
            .then(data => {
                document.getElementById('output').textContent = JSON.stringify(data, null, 2);
            })
            .catch(error => console.error('Error fetching JSON:', error));
    </script>
</body>
</html>

So, let’s break this down:

• fetch: gets JSON from https://jsonplaceholder.typicode.com/todos/1
• document.getElementById(‘output’).textContent: renders the JSON
• <pre>: a tag to preserve formatting for easy reading

By serving this file locally, we simulate the exact scenario where JSON is fetched dynamically and rendered on the page. The <pre> tag keeps the formatting intact, so Selenium can later capture the entire JSON string in one shot. Thus, the approach mirrors real-world cases where dashboards or developer endpoints expose JSON responses in plain text.

5. Writing the Java Selenium Code

Now, let’s update the App.java file under src/main/java/com/example:

package com.example;

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;

public class App {
    public static void main(String[] args) {
        // Optional: Set ChromeDriver path if not in system PATH
        // System.setProperty("webdriver.chrome.driver", "/path/to/chromedriver");

        WebDriver driver = new ChromeDriver();

        try {
            // Open local test page
            driver.get("http://localhost:8000/test.html");

            // Wait briefly for JSON to load
            Thread.sleep(2000); // 2 seconds

            // Capture JSON from <pre> element
            String json = driver.findElement(By.id("output")).getText();
            System.out.println("Captured JSON:\n" + json);

        } catch (InterruptedException e) {
            e.printStackTrace();
        } finally {
            driver.quit();
        }
    }
}

Let’s analyze the code above:

• WebDriver driver = new ChromeDriver(): launches Chrome
• driver.get(“http://localhost:8000/test.html”): opens the page
• Thread.sleep(2000): waits for JSON to load
• driver.findElement(By.id(“output”)).getText(): extracts the JSON text

At this point, we should be able to run the project.

6. Running the Project

Of course, we need to serve the HTML file so ChromeDriver can load it via HTTP.

From the project directory, let’s start a local server for test.html:

$ python3 -m http.server 8000
Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000/) ...

Above, we use the Python built-in HTTP server. Any simple static server would work since the key point is that ChromeDriver can only load files via http:// or https://, not directly from our file system with file://.

Once the local server is set up, let’s compile the project:

$ mvn clean compile
[INFO] Scanning for projects...
...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  5.752 s
[INFO] Finished at: 2025-09-02T10:32:19+03:00
[INFO] ------------------------------------------------------------------------

After compiling the project, we can run the Selenium program:

$ mvn exec:java
[INFO] Scanning for projects...
...
Captured JSON:
{
  "userId": 1,
  "id": 1,
  "title": "delectus aut autem",
  "completed": false
}
...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  44.448 s
[INFO] Finished at: 2025-09-02T10:38:15+03:00
[INFO] ------------------------------------------------------------------------

In the output, it’s not uncommon to encounter CDP version warnings:

WARNING: Unable to find version of CDP to use for 139.0.7258.66. You may need to include a dependency on a specific version of the CDP using something similar to `org.seleniumhq.selenium:selenium-devtools-v86:4.25.0` where the version ("v86") matches the version of the chromium-based browser you're using and the version number of the artifact is the same as Selenium's.

In this case, any CDP version warnings are harmless.

[WARNING] thread Thread[UrlChecker-2,5,com.example.App] was interrupted but is still alive after waiting at least 14994msecs

Additionally, we can encounter thread warnings:

[WARNING] thread Thread[UrlChecker-2,5,com.example.App] was interrupted but is still alive after waiting at least 14994msecs

They occur because Maven tries to terminate threads created by Selenium.

7. Understanding the Output

The JSON we captured is exactly what the fetch API returned and rendered on the page.

To explain, this is how the approach works:

• Selenium interacts with the browser
• JSON is rendered in the DOM, specifically the <pre> element
• Selenium extracts the text content, so we get the full JSON response

Thus, the method functions across browsers as long as the page renders the JSON, and no complex proxies are needed, making it easy for beginners to implement.

Conversely, the limitation of the approach is that it only works for JSON rendered in the page. Additionally, it’s not suitable for background API calls that don’t update the DOM. If we need to intercept raw API responses that never appear in the DOM, we need advanced tools such as Selenium DevTools or a proxy like BrowserMob.

8. Conclusion

In this article, we examined how to obtain a basic JSON response from an HTML page using Selenium.

By leveraging how the page renders API responses, we can use Selenium WebDriver to load the page, wait for the response, and extract JSON directly from the DOM. In this case, we were able to fetch and print JSON directly in Java from serving a static test.html file and using Selenium WebDriver to locate and extract the text inside a <pre> tag.

We now have a minimal setup of a working Java Selenium project that captures JSON from a web page.