Ignore Tests Conditionally in ScalaTest

1. Introduction

Testing is an essential part of software development to ensure code quality. Some of these tests might depend on external services such as Databases, Web Services, Message Queues, etc. As a result, in certain scenarios, we want to ignore running the tests if the required services are not available. 

In this tutorial, we’ll look at different ways to dynamically ignore or exclude tests in ScalaTest, Scala’s most popular testing library.

2. Ignore Tests Statically

Before ignoring the tests dynamically in ScalaTest, let’s have a quick look at ignoring the tests statically. We can use the @Ignore annotation on a test class to ignore all the tests in that particular class:

@Ignore
class ConditionalUnitTest extends AnyWordSpec {
  "Conditional tests" should {
    "ignore this test due to annotation" in {
      fail()
    }
  }
}

We can also ignore an individual test case using the ignore tag:

"ignore this test due to annotation" ignore {
  fail()
}

Please note that we utilized the ignore keyword instead of using the in keyword. As a result, only this specific test gets excluded from execution. Nevertheless, both of these methods enable the static ignoring of tests; hence, the tests get ignored.

3. Ignore Using Tags

Tags provide a means of grouping multiple tests based on specific behaviors within ScalaTest. Tags offer a convenient way to mark tests requiring specific external services. Additionally, they enable the exclusion of these tests from execution within particular environments. Let’s look at an example usage of tags and how to exclude running a specific tag:

object RequiresAvailableService extends Tag("RequiresAvailableService")
class ConditionalUnitTest extends AnyWordSpec {
  "Conditional Tests" should {
    "run this test only if the service is available by checking tag" taggedAs (RequiresAvailableService) in {
      succeed
    }
  }
}

Here, we created a tag, RequiresAvailableService, and tagged the test. Consequently, we can execute all tests except those that are tagged with this particular tag using runners:

sbt 'testOnly -- -l "RequiresAvailableService"'

When we execute the above command, it runs all tests except those tagged with the RequiresAvailableService tag. The flag -l takes the tag to be excluded from the execution.

This approach is useful in excluding a defined set of tests in different environments. However, this doesn’t allow us to control the execution conditionally.

Let’s modify the above example to support conditional execution:

object ServiceChecker {
  def isServiceAvailable: Boolean = false //logic to check service availability
}
object RequiresAvailableServiceConditional
  extends Tag(
    if (ServiceChecker.isServiceAvailable) "" else classOf[Ignore].getName
  )

We introduced a new method to check for the availability of the service. While creating the tag, we use this method conditionally into the tag name. If the service is unavailable, we pass the value as the full name of the Ignore class.

ScalaTest automatically ignores all the tests tagged with the name org.scalatest.Ignore and mark those tests as IGNORED in the test report.

We can use this tag within the test as before:

"run this test using the conditional if-else tag if service is available" taggedAs (RequiresAvailableServiceConditional) in {
  succeed
}

The above test is executed only if the service is available. Otherwise, it’s ignored. This way, we can conditionally execute certain tests. However, the tag value is determined at the time of the object creation. If the service becomes unavailable after the instance is created, this approach still tries to execute them.

4. Cancel Using if Condition

We can use a simple if condition to exclude tests from execution:

"run this test using simple if-else condition" in {
  if (!ServiceChecker.isServiceAvailable) {
    cancel("excluding this test due to service not available")
  }
  succeed
}

In this case, we used a simple if condition to check whether the service was available. If not, we exclude this test from executing using the cancel() method. Unlike using tags, this approach checks the condition during the test execution. Furthermore, the canceled tests are marked in the report as CANCELED.

If we need to use the same condition in multiple tests, we can extract the logic into a single method and use it:

private def skipTestIfServerUnavailable(test: => Assertion): Assertion = {
  if (ServiceChecker.isServiceAvailable) {
    test
  } else cancel("Not executing test since the service is not available")
}

Now, we can use this method in the test:

"run this test using skipTestIfServerUnavailable method" in skipTestIfServerUnavailable {
  succeed
}

Separating the service availability check from the test logic makes the tests more readable.

5. Using assume()

ScalaTest offers an alternative method for conditionally excluding tests from execution. By utilizing the assume() method, we can specify an assumption that must be true for the test to proceed. If the assumption fails, the test execution is canceled.

Let’s look at the usage of assume():

"run this test based on assume" in {
  assume(ServiceChecker.isServiceAvailable)
  // continue with tests
  succeed
}

In the above example, we invoked the isServiceAvailable() method in the assume() method. This method simplifies the code, eliminating the need for explicit cancellation logic within the test.

6. Canceled vs. Ignored Tests

The differences between canceled and ignored tests lie primarily in their reporting within ScalaTest. Tests marked as canceled occur when the cancel() method is explicitly invoked within the test. Conversely, if the condition within assume() fails, it automatically triggers the cancel() method. This means that partial execution of a test may occur if it is canceled. On the other hand, ignored tests result in the entire test case being excluded from execution. Both ignored and canceled tests are reported in yellow color in the reporting with either IGNORED or CANCELED message:

Here, we can notice that the counts for canceled and ignored tests are reported separately.

7. Conclusion

In this article, we explored different approaches in ScalaTest to exclude tests conditionally. We discussed the usage of tags, simple if-conditions, and the assume() method to achieve this requirement. The best approach will depend on your situation.

As always, the sample code used in this tutorial is available over on GitHub.