Continue the Test Even After Assertion Failure in TestNG

1. Overview

TestNG is a popular Java testing framework that’s an alternative to JUnit. While both frameworks offer their own paradigms, they both include the idea of assertions: logical statements that halt program execution if they evaluate to false, failing the test. A simple assertion in TestNG could look something like this:

@Test 
void testNotNull() {
    assertNotNull("My String"); 
}

But what happens if we need to make multiple assertions in a single test? In this article, we’ll explore TestNG’s SoftAssert, a technique for executing multiple assertions together.

2. Setup

For our exercise, let’s define a simple Book class:

public class Book {
    private String isbn;
    private String title;
    private String author;

    // Standard getters and setters...
}

We can also define an interface that models a simple service that looks up a Book based on its ISBN:

interface BookService {
    Book getBook(String isbn);
}

We can then mock this service in the unit test, which we’ll define later. This setup lets us define a scenario we can test in a realistic way: a service that returns an object that may be null or whose member variables may be null. Let’s start writing a unit test for this.

3. Basic Assertions Versus TestNG’s SoftAssert

To illustrate the benefits of SoftAssert, we’ll start by creating a unit test using basic TestNG assertions that fail and compare the feedback we get to the same test utilizing SoftAssert.

3.1. Using Traditional Assertions

To start, we’ll create a test using assertNotNull(), which takes a value to test and an optional message:

@Test
void givenBook_whenCheckingFields_thenAssertNotNull() {
    Book gatsby = bookService.getBook("9780743273565");

    assertNotNull(gatsby.isbn, "ISBN");
    assertNotNull(gatsby.title, "title");
    assertNotNull(gatsby.author, "author");
}

Then, we’ll define a mock implementation (using Mockito) of BookService that returns a Book instance:

@BeforeMethod
void setup() {
    bookService = mock(BookService.class);
    Book book = new Book();
    when(bookService.getBook(any())).thenReturn(book);
}

Running our test, we can see that we neglected to set the isbn field:

java.lang.AssertionError: ISBN expected object to not be null

Let’s fix this in our mock and run the test again:

@BeforeMethod void setup() {
    bookService = mock(BookService.class);
    Book book = new Book();
    book.setIsbn("9780743273565");
    when(bookService.getBook(any())).thenReturn(book);
}

We now receive a different error:

java.lang.AssertionError: title expected object to not be null

Again, we forgot to initialize a field in our mock, leading to another necessary change.

As we can see, this cycle of testing, making changes, and re-running the test isn’t only frustrating but time-consuming. This effect is, of course, multiplied by the size and complexity of the class. This problem is further compounded in the case of integration tests. Failures in remote deployment environments may be difficult or impossible to reproduce locally. Integration tests are typically more complex and, therefore, have longer execution times. Coupling this with the time needed to deploy test changes means the cycle time of each additional test re-run is costly.

Luckily, we can avoid this problem by using SoftAssert to evaluate multiple assertions without halting program execution immediately.

3.2. Grouping Assertions With SoftAssert

Let’s update our example above to use SoftAssert:

@Test void givenBook_whenCheckingFields_thenAssertNotNull() {
    Book gatsby = bookService.getBook("9780743273565"); 
    
    SoftAssert softAssert = new SoftAssert();
    softAssert.assertNotNull(gatsby.isbn, "ISBN");
    softAssert.assertNotNull(gatsby.title, "title");
    softAssert.assertNotNull(gatsby.author, "author");
    softAssert.assertAll();
}

Let’s break this down:

• first, we create an instance of SoftAssert
• next, we make a crucial change: we make our assertions against the instance of SoftAssert rather than using TestNG’s basic assertNonNull() method
• finally, it’s equally important to note we need to call the assertAll() method on the SoftAssert instance once we’re ready to get the result of all our assertions

Now, if we run this with our original mock that neglected to set any member variable values for Book, we’ll see a single error message containing all of the assertion failures:

java.lang.AssertionError: The following asserts failed:
    ISBN expected object to not be null,
    title expected object to not be null,
    author expected object to not be null

This shows how using SoftAssert is a good practice when a single test requires more than one assertion.

3.3. Considerations for SoftAssert

While SoftAssert is easy to setup and use, there is an important consideration to keep in mind: statefulness. Because SoftAssert records the failure of each assertion internally, it’s not suitable to share across multiple test methods. For this reason, we should make sure to create a new instance of SoftAssert in each test method.

4. Conclusion

In this tutorial, we’ve learned how to make multiple assertions using TestNG’s SoftAssert and how this can be a valuable tool for writing clean tests with reduced debugging time. We also learned that SoftAssert is stateful and instances shouldn’t be shared among multiple tests.