I just announced the new Spring 5 modules in REST With Spring:

>> CHECK OUT THE COURSE

1. Overview

JUnit and TestNG are undoubtedly the two most popular unit-testing frameworks in the Java ecosystem. While JUnit inspires TestNG itself, it provides its distinctive features, and unlike JUnit, it works for functional and higher levels of testing.

In this article, we will discuss and compare these frameworks by covering their features and common use cases.

2. Test Setup

While writing test cases, often we need to execute some configuration or initialization instructions before test executions, and also some cleanup after completion of tests. Let’s evaluate these in both frameworks.

JUnit offers initialization and cleanup at two levels, before and after each method and class. @Before and @After annotations at method level and @BeforeClass and @AfterClass at the class level:

public class SummationServiceTest {

    private static List<Integer> numbers;

    @BeforeClass
    public static void initialize() {
        numbers = new ArrayList<>();
    }

    @AfterClass
    public static void tearDown() {
        numbers = null;
    }

    @Before
    public void runBeforeEachTest() {
        numbers.add(1);
        numbers.add(2);
        numbers.add(3);
    }

    @After
    public void runAfterEachTest() {
        numbers.clear();
    }

    @Test
    public void givenNumbers_sumEquals_thenCorrect() {
        int sum = numbers.stream().reduce(0, Integer::sum);
        assertEquals(6, sum);
    }
}

Similar to JUnit, TestNG also provides initialization and cleanup at the method and class level. While @BeforeClass and @AfterClass remain the same at the class level, the method level annotations are @BeforeMethod and @AfterMethod:

@BeforeClass
public void initialize() {
    numbers = new ArrayList<>();
}

@AfterClass
public void tearDown() {
    numbers = null;
}

@BeforeMethod
public void runBeforeEachTest() {
    numbers.add(1);
    numbers.add(2);
    numbers.add(3);
}

@AfterMethod
public void runAfterEachTest() {
    numbers.clear();
}

TestNG also offers, @BeforeSuite, @AfterSuite, @BeforeGroup and @AfterGroup annotations, for configurations at suite and group levels:

@BeforeGroups("positive_tests")
public void runBeforeEachGroup() {
    numbers.add(1);
    numbers.add(2);
    numbers.add(3);
}

@AfterGroups("negative_tests")
public void runAfterEachGroup() {
    numbers.clear(); 
}

Also, we can use the @BeforeTest and @AfterTest if we need any configuration before or after test cases included in the <test> tag in TestNG XML configuration file:

<test name="test setup">
    <classes>
        <class name="SummationServiceTest">
            <methods>
                <include name="givenNumbers_sumEquals_thenCorrect" />
            </methods>
        </class>
    </classes>
</test>

Note that, the declaration of @BeforeClass and @AfterClass method has to be static in JUnit whereas, there is more flexibility in TestNG in the method declaration, it does not have these constraints.

3. Ignoring Tests

Both frameworks support ignoring test cases, though they do it quite differently. JUnit offers @Ignore annotation:

@Ignore
@Test
public void givenNumbers_sumEquals_thenCorrect() {
    int sum = numbers.stream().reduce(0, Integer::sum);
    Assert.assertEquals(6, sum);
}

while TestNG uses @Test with a parameter “enabled” with a boolean value true or false:

@Test(enabled=false)
public void givenNumbers_sumEquals_thenCorrect() {
    int sum = numbers.stream.reduce(0, Integer::sum);
    Assert.assertEquals(6, sum);
}

4. Running Tests Together

Running tests together as a collection is possible in both JUnit 4 and TestNG, but they do it in different ways.

The @RunWith and @Suite annotations are used to group test cases and run them as a suite. A suite is a collection of test cases that can be grouped together and run as a single test.

All the declarations are defined inside a single class, let’s look at the example:

@RunWith(Suite.class)
@Suite.SuiteClasses({ RegistrationTest.class, SignInTest.class })
public class SuiteTest {

}

In TestNG grouping tests in groups are done using XML file:

<suite name="suite">
    <test name="test suite">
        <classes>
            <class name="com.baeldung.RegistrationTest" />
            <class name="com.baeldung.SignInTest" />
        </classes>
    </test>
</suite>

This indicates RegistrationTest and SignInTest will run together.

Apart from grouping classes, TestNG can group methods as well. Methods are grouped with @Test(groups=”groupName”) annotation:

@Test(groups = "regression")
public void givenNegativeNumber_sumLessthanZero_thenCorrect() {
    int sum = numbers.stream().reduce(0, Integer::sum);
    Assert.assertTrue(sum < 0);
}

Let’s use an XML to execute the groups:

<test name="test groups">
    <groups>
        <run>
            <include name="regression" />
        </run>
    </groups>
    <classes>
        <class 
          name="com.baeldung.SummationServiceTest" />
    </classes>
</test>

This will execute the test method tagged with group regression.

5. Testing Exceptions

The feature for testing for exceptions using annotations is available in both JUnit 4:

@Test(expected = ArithmeticException.class) 
public void givenNumber_whenThrowsException_thenCorrect() { 
    int i = 1 / 0;
}

and TestNG:

@Test(expectedExceptions = ArithmeticException.class) 
public void givenNumber_whenThrowsException_thenCorrect() { 
    int i = 1 / 0;
}

This feature implies what exception is thrown from a piece of code, that is being unit tested.

6. Parameterized Tests

Parameterized unit tests are used for testing the same code under several conditions. With the help of parameterized unit tests, we can set up a test method that obtains data from some data source. The main idea is to make the unit test method reusable and to test with a different set of inputs.

In JUnit, the test class has to be annotated with @RunWith to make it a parameterized class and @Parameter to use the denote the parameter values for unit test. @Parameters will have to return a List[], and these parameters will be passed to the constructor of the test class as an argument:

@RunWith(value = Parameterized.class)
public class ParametrizedTests {

    private int value;
    private boolean isEven;

    public ParametrizedTests(int value, boolean isEven) {
        this.value = value;
        this.isEven = isEven;
    }

    @Parameters
    public static Collection<Object[]> data() {
        Object[][] data = new Object[][]{{1, false}, {2, true}, {4, true}};
        return Arrays.asList(data);
    }

    @Test
    public void givenParametrizedNumber_ifEvenCheckOK_thenCorrect() {
        Assert.assertEquals(isEven, value % 2 == 0);
    }
}

Note that the declaration of the parameter has to be static and it has to be passed to the constructor to initialize the class member as value for testing. This puts a limitation on the number of parameterized methods per class to only one.

In addition to that, the return type of parameter class must be a List, and the values are limited to String or primitive data types.

In TestNG, we can parametrize tests using @Parameter or @DataProvider annotation. While using the XML file annotate the test method with @Parameter:

@Test
@Parameters({"value", "isEven"})
public void 
  givenNumberFromXML_ifEvenCheckOK_thenCorrect(int value, boolean isEven) {
    Assert.assertEquals(isEven, value % 2 == 0);
}

and provide the data in the XML file:

<suite name="My test suite">
    <test name="numbersXML">
        <parameter name="value" value="1"/>
        <parameter name="isEven" value="false"/>
        <classes>
            <class name="baeldung.com.ParametrizedTests"/>
        </classes>
    </test>
</suite>

While using information in the XML file is simple and useful, in some cases, you might need to provide more complex data.

@DataProvider annotation is used to handle these scenarios, which can be used to map complex parameter types for testing methods.

@DataProvider for primitive data types:

@DataProvider(name = "numbers")
public static Object[][] evenNumbers() {
    return new Object[][]{{1, false}, {2, true}, {4, true}};
}

@Test(dataProvider = "numbers")
public void givenNumberFromDataProvider_ifEvenCheckOK_thenCorrect
  (Integer number, boolean expected) {
    Assert.assertEquals(expected, number % 2 == 0);
}

@DataProviderfor objects:

@Test(dataProvider = "numbersObject")
public void givenNumberObjectFromDataProvider_ifEvenCheckOK_thenCorrect
  (EvenNumber number) {
    Assert.assertEquals(number.isEven(), number.getValue() % 2 == 0);
}

@DataProvider(name = "numbersObject")
public Object[][] parameterProvider() {
    return new Object[][]{{new EvenNumber(1, false)},
      {new EvenNumber(2, true)}, {new EvenNumber(4, true)}};
}

In the same way, any particular objects that are to be tested can be created and returned using data provider. It’s useful when integrating with frameworks like Spring.

Notice that, in TestNG, since @DataProvider method need not be static, we can use multiple data provider methods in the same test class.

7. Test Timeout

Timed out tests means, a test case should fail if the execution is not completed within certain specified period. Both JUnit and TestNG support timed out tests, in the same way, annotating with @Test(timeout=1000):

@Test(timeOut = 1000)
public void givenExecution_takeMoreTime_thenFail() {
    while (true);
}

8. Dependent Tests

TestNG supports dependency testing. This means in a set of test methods, if the initial test fails, then all subsequent dependent tests will be skipped, not marked as failed as in the case for JUnit.

Let’s have a look at a scenario, where we need to validate email, and if it’s successful, will proceed to log in:

@Test
public void givenEmail_ifValid_thenTrue() {
    boolean valid = email.contains("@");
    Assert.assertEquals(valid, true);
}

@Test(dependsOnMethods = {"givenEmail_ifValid_thenTrue"})
public void givenValidEmail_whenLoggedIn_thenTrue() {
    LOGGER.info("Email {} valid >> logging in", email);
}

9. Order of Test Execution

There is no defined implicit order in which test methods will get executed in JUnit 4 or TestNG. The methods are just invoked as returned by the JAVA Reflection API. Since JUnit 4 it uses a more deterministic but not predictable order.

To have more control, we will annotate the test class with @FixMethodOrder annotation and mention a method sorter:

@FixMethodOrder(MethodSorters.NAME_ASCENDING)
public class SortedTests {

    @Test
    public void a_givenString_whenChangedtoInt_thenTrue() {
        assertTrue(
          Integer.valueOf("10") instanceof Integer);
    }

    @Test
    public void b_givenInt_whenChangedtoString_thenTrue() {
        assertTrue(
          String.valueOf(10) instanceof String);
    }

}

The MethodSorters.NAME_ASCENDING parameter sorts the methods by method name is lexicographic order. Apart from this sorter, we have MethodSorter.DEFAULT and MethodSorter.JVM as well. 

While TestNG also provides a couple of ways to have control in the order of test method execution. We provide the priority parameter in the @Test annotation:

@Test(priority = 1)
public void givenString_whenChangedToInt_thenCorrect() {
    Assert.assertTrue(
      Integer.valueOf("10") instanceof Integer);
}

@Test(priority = 2)
public void givenInt_whenChangedToString_thenCorrect() {
    Assert.assertTrue(
      String.valueOf(23) instanceof String);
}

Notice, that priority invokes test methods based on priority but does not guarantee that tests in one level are completed before invoking the next priority level.

Sometimes while writing functional test cases in TestNG, we might have an interdependent test where the order of execution must be same for every test run, to achieve that we should use the dependsOnMethods parameter to @Test annotation as shown in the earlier section.

10. Conclusion

Both JUnit and TestNG are modern tools for testing in the Java ecosystem.

In this article, we had a quick look at the various needs and ways of writing tests with each of these two test frameworks.

The implementation of all the code snippets can be found in TestNG and core-java Github project.

I just announced the new Spring 5 modules in REST With Spring:

>> CHECK OUT THE LESSONS

Sort by:   newest | oldest | most voted
Slava Semushin
Guest
>6. Parameterized Tests >Note that the declaration of the parameter has to be static and it has to be passed to the constructor in order to initialize the class member as value for testing. Effectively this limitation means that we can have only one parameterized method per class. With @DataProvider from TestNG it’s possible to have 5 methods with 5 data providers in the same class. Also very important point for me was an execution order of the test methods. TestNG executing test methods in the same order as they’re defined in the class, while JUnit doesn’t respect this order.… Read more »