How to Implement Retry for JUnit Tests

1. Overview

Automated tests are an essential part of modern software development. However, not all test failures are caused by bugs. Some failures are flaky, occurring intermittently due to reasons like race conditions, network delays, or resource constraints.

To deal with such transient failures, implementing retry logic in tests can be a powerful tool for us. A retry mechanism allows our tests to be re-executed a specified number of times before being treated as failing, which helps stabilize our test runs and reduce false negatives in CI pipelines.

In this tutorial, we’ll explore how to implement retry logic in both JUnit 4 and JUnit 5, implementing custom and library-based approaches, and providing guidance on best practices.

2. Implementing Retry Logic in JUnit 5

JUnit 5 introduced a powerful extension model that makes it much easier for us to customize test behavior compared to JUnit 4. There are two common ways we can implement retry logic in JUnit 5: creating a custom extension that programmatically handles our retries when a test fails, and using an external library like JUnit Pioneer that provides built-in retry annotations.

Let’s explore both approaches.

2.1. Custom Retry Extension using TestExecutionExceptionHandler

We can create a custom extension in JUnit 5 by implementing the TestExecutionExceptionHandler interface, allowing retries when an exception is thrown during a test execution in our code.

Below is the implementation of a custom JUnit 5 extension using TestExecutionExceptionHandler:

public class RetryExtension implements TestExecutionExceptionHandler {
    private static final int MAX_RETRIES = 3;
    private static final ExtensionContext.Namespace NAMESPACE =
      ExtensionContext.Namespace.create("RetryExtension");

    @Override
    public void handleTestExecutionException(ExtensionContext context, Throwable throwable) throws Throwable {
        Store store = context.getStore(NAMESPACE);
        int retries = store.getOrDefault("retries", Integer.class, 0);

        if (retries < MAX_RETRIES) {
            retries++;
            store.put("retries", retries);
            System.out.println("Retrying test " + context.getDisplayName() + ", attempt " + retries);
            throw throwable;
        } else {
            throw throwable;
        }
    }
}

To use this retry mechanism in practice, let’s annotate our test class with @ExtendWith(RetryExtension.class) and write our test logic as shown below:

@ExtendWith(RetryExtension.class)
public class RetryTest {
    private static int attempt = 0;

    @Test
    public void testWithRetry() {
        attempt++;
        System.out.println("Test attempt: " + attempt);

        if (attempt < 3) {
            throw new RuntimeException("Failing test");
        }
    }
}

In this implementation, we use JUnit 5’s TestExecutionExceptionHandler to create a custom extension that retries a test up to a specified number of times (here, we use 3) when it fails. The retry count is stored in the test context using JUnit’s ExtensionContext.Store, and each retry is logged to help with debugging. If the test still fails after all attempts, the exception is rethrown to mark it as failed.

2.2. Using JUnit Pioneer’s @RetryingTest

For an easier, out-of-the-box solution, the JUnit Pioneer library provides us with a @RetryingTest annotation that automatically retries failed tests.

First, let’s add the junit-pioneer dependency to our pom.xml:

<dependency>
    <groupId>org.junit-pioneer</groupId>
    <artifactId>junit-pioneer</artifactId>
    <version>2.0.1</version>
    <scope>test</scope>
</dependency>

Then, let’s see how to use it in our tests:

public class RetryPioneerTest {
    private static int attempt = 0;

    @RetryingTest(maxAttempts = 3)
    void testWithRetry() {
        attempt++;
        System.out.println("Test attempt: " + attempt);

        if (attempt < 3) {
            throw new RuntimeException("Failing test");
        }
    }
}

The @RetryingTest from the JUnit Pioneer library provides an easy way to add retry logic without writing custom code. We simply specify the maximum number of attempts, and the library takes care of re-running the test until it passes or the limit is reached.

3. Implementing Retry Logic in JUnit 4

JUnit 4 lacks the modern extension model of JUnit 5, but we can achieve retry logic by creating a custom TestRule.

Let’s see an implementation of this:

public class RetryRule implements TestRule {
    private final int retryCount;

    public RetryRule(int retryCount) {
        this.retryCount = retryCount;
    }

    @Override
    public Statement apply(Statement base, Description description) {
        return new Statement() {
            @Override
            public void evaluate() throws Throwable {
                Throwable failure = null;

                for (int i = 0; i < retryCount; i++) {
                    try {
                        base.evaluate();
                        return;
                    } catch (Throwable t) {
                        failure = t;
                        System.out.println("Retry " + (i + 1) + "/" + retryCount +
                          " for test " + description.getDisplayName());
                    }
                }

                throw failure;
            }
        };
    }
}

And here’s how we would use the rule:

public class RetryRuleTest {
    @Rule
    public RetryRule retryRule = new RetryRule(3);

    private static int attempt = 0;

    @Test
    public void testWithRetry() {
        attempt++;
        System.out.println("Test attempt: " + attempt);

        if (attempt < 3) {
            throw new RuntimeException("Failing test");
        }
    }
}

In JUnit 4, we use a custom TestRule called RetryRule to implement retry logic. The rule wraps test execution and retries it the configured number of times if it fails. Each attempt is logged, and if all retries fail, the test is marked as failed by throwing the last encountered exception.

4. Best Practices for Test Retries

Let’s review some best practices for using retries:

• If a test fails intermittently due to timing or environment issues, retries make sense. But for consistent failures, retries only hide real problems.
• Always log each retry attempt for easier debugging.
• Too many retries can slow down test execution and mask issues. A common default is two or three attempts.
• Use retries as a temporary solution and aim to fix the root cause.

5. Conclusion

Retrying failed tests can significantly improve the reliability of our test suites, especially in CI environments where occasional flakiness is common. JUnit 4 and JUnit 5 both support retries, either via custom logic or third-party libraries like JUnit Pioneer. Use retries responsibly; not to ignore bugs, but to manage external flakiness while continuing to build high-quality software.