JUnit 5 Conditional Test Execution with Annotations

1. Overview

In this tutorial, we’re going to take a look at conditional test execution with annotations in JUnit 5.

These annotations are from the JUnit Jupiter library’s condition package and allow us to specify different types of conditions under which our tests should or should not run.

2. Operating System Conditions

Sometimes, we need to change our test scenarios depending on the operating systems (OS) they’re running on. In these cases, the @EnabledOnOs annotation comes in handy.

The usage of @EnabledOnOs is simple – we just need to give it a value for the OS type. Furthermore, it also accepts an array argument for when we want to target multiple operating systems.

For example, let’s say we want to enable a test to run only on Windows and macOS:

@Test
@EnabledOnOs({OS.WINDOWS, OS.MAC})
public void shouldRunBothWindowsAndMac() {
    //...
}

Now, in contrast to the @EnabledOnOs, there is @DisabledOnOs. As the name implies, it disables tests according to the OS type argument:

@Test
@DisabledOnOs(OS.LINUX)
public void shouldNotRunAtLinux() {
    //...
}

3. Java Runtime Environment Conditions

We can also target our tests to run on specific JRE versions using the @EnableOnJre and @DisableOnJre annotations. These annotations also accept an array to enable or disable multiple Java versions:

@Test
@EnabledOnJre({JRE.JAVA_10, JRE.JAVA_11})
public void shouldOnlyRunOnJava10And11() {
    //...
}

As of JUnit 5.6, we can use @EnabledForJreRange to enable a test for a specific range of Java versions:

@Test
@EnabledForJreRange(min = JRE.JAVA_8, max = JRE.JAVA_13)
public void shouldOnlyRunOnJava8UntilJava13() {
    // this test will only run on Java 8, 9, 10, 11, 12, and 13.
}

By default the min value is JAVA_8 and the max value is the maximum possible JRE version. There is also a @DisabledForJreRange to disable a test for a specific range of Java versions:

@Test
@DisabledForJreRange(min = JRE.JAVA_14, max = JRE.JAVA_15)
public void shouldNotBeRunOnJava14AndJava15() {
    // this won't run on Java 14 and 15.
}

Moreover, if we want to disable our tests running with Java versions other than 8, 9, 10, and 11, we can use the JRE.OTHER enum property:

@Test
@DisabledOnJre(JRE.OTHER)
public void thisTestOnlyRunsWithUpToDateJREs() {
    // this test will only run on Java 8, 9, 10, and 11.
}

4. System Property Conditions

Now, if we want to enable our tests based on JVM system properties, we can use the @EnabledIfSystemProperty annotation.

To use it, we must provide named and matches arguments. The named argument is used to specify an exact system property. The matches is used for defining the pattern of property value with a regular expression.

For instance, let’s say we want to enable a test to run only when the virtual machine vendor name starts with “Oracle”:

@Test
@EnabledIfSystemProperty(named = "java.vm.vendor", matches = "Oracle.*")
public void onlyIfVendorNameStartsWithOracle() {
    //...
}

Likewise, we have the @DisabledIfSystemProperty to disable tests based on JVM system properties. To demonstrate this annotation, let’s take a look at an example:

@Test
@DisabledIfSystemProperty(named = "file.separator", matches = "[/]")
public void disabledIfFileSeperatorIsSlash() {
    //...
}

5. Environment Variable Conditions

We can also specify environment variable conditions for our tests with @EnabledIfEnvironmentVariable and @DisabledIfEnvironmentVariable annotations.

And, just like the annotations for system property conditions, these annotations take two arguments — named and matches — for specifying the environment variable name and regular expression to match against environment variable values:

@Test
@EnabledIfEnvironmentVariable(named = "GDMSESSION", matches = "ubuntu")
public void onlyRunOnUbuntuServer() {
    //...
}

@Test
@DisabledIfEnvironmentVariable(named = "LC_TIME", matches = ".*UTF-8.")
public void shouldNotRunWhenTimeIsNotUTF8() {
    //...
}

Furthermore, we can consult one of our other tutorials to learn more about system properties and system environment variables.

6. Native Image Conditions

When building native images with GraalVM, it’s important to consider the conditions under which certain containers or tests should be enabled or disabled. @EnabledInNativeImage and @DisabledInNativeImage annotations provide a straightforward way to achieve this.

By using @EnabledInNativeImage, we can ensure that a test is only executed within the context of a native image:

@Test
@EnabledInNativeImage
void shouldOnlyRunWithinNativeImage() {
    // ...
}

Conversely, @DisabledInNativeImage ensures that a test is never executed within a native image:

@Test
@DisabledInNativeImage
void shouldNeverRunWithinNativeImage() {
    // ...
}

By including these annotations in the test code, we can fine-tune the execution of the tests within the context of a GraalVM native image.

7. Script-Based Conditions

7.1. Deprecation Notice

Script-based condition APIs and their implementations were deprecated in JUnit 5.5 and removed from JUnit 5.6. In order to achieve the same result, it’s highly recommended to use a combination of built-in conditions or create a custom implementation of ExecutionCondition.

7.2. Conditions

Prior to JUnit 5.6, we can specify our test’s running conditions by writing scripts within @EnabledIf and @DisabledIf annotations.

These annotations accept three arguments:

• value – contains the actual script to run.
• engine (optional) – specifies the scripting engine to use; the default is Oracle Nashorn.
• reason (optional) – for logging purposes, specifies the message JUnit should print if our test fails.

So, let’s see a simple example where we specify only a one-line script, without additional arguments on the annotation:

@Test
@EnabledIf("'FR' == systemProperty.get('user.country')")
public void onlyFrenchPeopleWillRunThisMethod() {
    //...
}

Also, the usage of @DisabledIf is exactly the same:

@Test
@DisabledIf("java.lang.System.getProperty('os.name').toLowerCase().contains('mac')")
public void shouldNotRunOnMacOS() {
    //...
}

Furthermore, we can write multi-line scripts with the value argument.

Let’s write a brief example to check the month’s name before running the test.

We’ll define a sentence for reason with supported placeholders:

• {annotation} – the string for representing annotation instance.
• {script} – the script text that evaluated inside value argument.
• {result} – the string for representing the return value of the evaluated script.

For this instance, we will have a multi-line script in the value argument and values for engine and reason:

@Test
@EnabledIf(value = {
    "load('nashorn:mozilla_compat.js')",
    "importPackage(java.time)",
    "",
    "var thisMonth = LocalDate.now().getMonth().name()",
    "var february = Month.FEBRUARY.name()",
    "thisMonth.equals(february)"
},
    engine = "nashorn",
    reason = "On {annotation}, with script: {script}, result is: {result}")
public void onlyRunsInFebruary() {
    //...
}

We can use several script bindings when writing our scripts:

• systemEnvironment – to access system environment variables.
• systemProperty – to access system property variables.
• junitConfigurationParameter – to access configuration parameters.
  
• junitDisplayName – test or container display name.
• junitTags – to access tags on tests or container.
• anotherUniqueId – to get the unique id of test or container.
  

Finally, let’s look at another example to see how to use scripts with bindings:

@Test
@DisabledIf("systemEnvironment.get('XPC_SERVICE_NAME') != null" +
        "&& systemEnvironment.get('XPC_SERVICE_NAME').contains('intellij')")
public void notValidForIntelliJ() {
    //this method will not run on intelliJ
}

Moreover, please consult one of our other tutorials to learn more about @EnabledIf and @DisabledIf annotations.

8. Creating Custom Conditional Annotations

A very powerful feature that comes with JUnit 5 is the ability to create custom annotations. We can define custom conditional annotations by using a combination of existing conditional annotations.

For instance, suppose we want to define all our tests to run for specific OS types with specific JRE versions. We can write a custom annotation for this:

@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
@Test
@DisabledOnOs({OS.WINDOWS, OS.SOLARIS, OS.OTHER})
@EnabledOnJre({JRE.JAVA_9, JRE.JAVA_10, JRE.JAVA_11})
@interface ThisTestWillOnlyRunAtLinuxAndMacWithJava9Or10Or11 {
}

@ThisTestWillOnlyRunAtLinuxAndMacWithJava9Or10Or11
public void someSuperTestMethodHere() {
    // this method will run with Java9, 10, 11 and Linux or macOS.
}

Furthermore, we can use script-based annotations to create a custom annotation:

@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
@DisabledIf("Math.random() >= 0.5")
@interface CoinToss {
}

@RepeatedTest(2)
@CoinToss
public void gamble() {
    // this method run run roughly 50% of the time
}

9. Conclusion

In this article, we learned about conditional test execution with annotations in JUnit 5. Also, we walked through some examples of their usage.

Next, we saw how to create custom conditional annotations.

To learn more about this topic, we can consult JUnit’s documentation about conditional test execution with annotations.