Introduction to ScalaCheck

1. Introduction

ScalaCheck is a library that automates stateful and stateless property-based testing, inspired by the Haskell library QuickCheck. This library enables us to write tests with random inputs without boilerplate code while also checking edge-case scenarios.

2. Installation

We can use ScalaCheck simply by adding it as a dependency in the sbt build file:

libraryDependencies += "org.scalacheck" %% "scalacheck" % "1.14.1" % "test"

3. Stateless Tests: Properties

The Properties class ships the functionality for stateless property testing. Specifically, it holds a collection of properties with convenient methods for checking.

Let’s see an example with a trivial check:

case class Simple(str: String) {
  def length: Int = str.length
}

object PropertiesUnitTest extends Properties("Simple") {
  property("length") = forAll { (str: String) =>
    Simple(str).length >= 0
  }
}

When we run it, we’ll see the console output:

+ Simple.length: OK, passed 100 tests.

As mentioned in the console output, the test was checked against 100 different input combinations. In fact, we can change the number of minimum successful tests needed, the number of workers for parallel execution, and other properties.

To illustrate the configuration capabilities, here, we change the minimumSuccessfulTests, and we set a callback function for onPropEval:

override def overrideParameters(p: Parameters): Parameters = {
  p.withMinSuccessfulTests(50)
    .withTestCallback(new TestCallback {
      override def onPropEval(name: String, threadIdx: Int, succeeded: Int, discarded: Int): Unit = {
        println(s"Evaluating prop with name: $name")
    }
  })
}

Since we added the onPropEval callback, the console output will change to:

...
Evaluating prop with name: CustomParameters.length
Evaluating prop with name: CustomParameters.length
...

4. Stateful Tests: Commands

On the contrary, the Commands trait gives us control over the input and its transitions. Admittedly, this approach looks more like a state machine since we need to specify the possible states and their transitions.

Let’s make a test that checks traffic light lifecycle.

First, we need to create the State and SystemUnderTest models:

sealed trait TrafficLightColorT {
  def color: String
}
case class TrafficLight(uuid: UUID, color: TrafficLightColorT)
case class SystemUnderTest(refId: UUID, date: Long, trafficLight: TrafficLight)

Now that we’ve created the test context model, we can proceed to the test:

object CommandsUnitTest extends Commands {
  override type State = TrafficLight
  override type Sut = SystemUnderTest

// initialization and precondition checks omitted for brevity

override def genInitialState: Gen[TrafficLight] = {
  for (
    trafficLightColor <- Gen.oneOf(model.Red, model.Yellow, model.Green)
  ) yield TrafficLight(UUID.randomUUID(), trafficLightColor)
}

override def genCommand(state: TrafficLight): Gen[Command] = {
  state.color match {
    case model.Green => TransitionToYellow(state)
    case model.Yellow => TransitionToRed(state)
    case model.Red => TransitionToGreen(state)
    case _ => throw new RuntimeException("Traffic lights have only green, yellow and red color.")
  }
}

case class TransitionToGreen(trafficLight: TrafficLight) extends Command {
  override type Result = Boolean

  override def run(sut: SystemUnderTest): Boolean = {
    println("going green")
    true
  }

  override def nextState(state: TrafficLight): TrafficLight = state.copy(color = model.Green)

  override def preCondition(state: TrafficLight): Boolean = state.color == model.Red

  override def postCondition(state: TrafficLight, result: Try[Boolean]): Prop = result == Success(true)
}

// Yellow and Red transitions are identical except from the color.

5. Generators

At this time, it’s obvious that ScalaCheck generates lots of random data to feed the test cases repeatedly. While the provided generators can come in quite handy, they often aren’t enough. There are cases where we need more control over the generated values.

Specifically, the Gen object has many helper functions that give us the ability to manipulate the generated values. The next section gives examples of various generators.

5.1. choose

Gen.choose picks random numbers from the inclusive range (num1, num2):

private val choiceGen = Gen.choose(-10, 10)
property("choiceGen") = forAll(choiceGen) { num => num.abs <= 10 }

5.2. pick

Gen.pick randomly picks a given number of elements from a list of values. As expected, the generated values are iterable:

property("randomPick") = forAll(Gen.pick(2, Seq(1, 2, 3, 4, 5))) { seq => seq.sum > 0 }

5.3. oneOf

Gen.oneOf selects a random element from the given list:

private val oneOfGen = Gen.oneOf(Seq(1, 2, -10, 40))
property("oneOf") = forAll(oneOfGen) { num => num.abs >= 0 }

5.4. sequence

Gen.sequence generates sequences, as the name implies. The sequence elements are created by the generators that the sequence argument consists of. Let’s see an example that uses the sequences that we already made above:

property("sequence") = forAll(Gen.sequence(Seq(choiceGen, oneOfGen))) { foo => foo.size >= 0 }

5.5. frequency

Gen.frequency creates a weighted random distribution. In our example, apples will be rarer than bananas, and bananas will be rarer than kiwis:

Gen.frequency((2, "apples"), (4, "bananas"), (6, "kiwis"))

6. Conclusion

In this tutorial, we presented ScalaCheck as a solution for property-based testing. Test cases that are written with ScalaCheck appear to be less verbose and less vulnerable to edge case failures.

As always, the code of the above examples is available over on GitHub.