Code Coverage Analysis Using sbt-scoverage

1. Introduction

In software development, the quality of the code is crucial. One of the significant ways to achieve this is by writing tests for each part of the code. To improve the quality, we should ensure that the tests adequately cover the code.

Test coverage is, without a doubt, one of the best ways to measure if we have enough tests and to detect if there are any untested areas in the application. Moreover, in this context, scoverage stands out as one of the most valuable tools for Scala developers.

In this tutorial, we’ll look into scoverage, how it works, its advantages, and how we can customize it for our needs.

2. Understanding Code Coverage

Code coverage is a very good metric for analyzing the coverage of the tests in the codebase. It helps to evaluate the effectiveness of the testing process. Moreover, it highlights the areas of code that aren’t covered with tests.

There are two main types of code coverage. Firstly, statement coverage checks the percentage of lines of code covered using the tests. Secondly, branch coverage calculates the test coverage for the different conditions within the codebase. By combining these two types, we can gain a comprehensive understanding of the total test coverage.

3. scoverage

scoverge is a free and open-source tool developed for calculating the test coverage in Scala codebases.

3.1. Advantages of scoverage

Some of the advantages of using scoverage are:

• Support for statement and branch coverage
• Integration with different build tools such as sbt, Maven, Gradle, and so on
• HTML and XML-based coverage reports
• Good customizable options
• Works with all Scala testing libraries such as ScalaTest, uTest, MUnit, and so on, as it relies on bytecode instrumentation for code coverage checks.

3.2. Integrations With Build Tools

scoverage provides integrations with all the popular build tools in the Scala ecosystem. There are scoverage plugins for SBT, Maven, Gradle, and Mill.

We’ll focus on the sbt plugin for scoverage since sbt is the most popular build tool for Scala applications.

4. sbt-scoverage Plugin

The sbt-scoverage plugin for SBT provides good integration of all features of scoverage with sbt. We can execute the coverage reports just like any other sbt command.

4.1. Setup

We can add the scoverage plugin to the project by adding it to the plugins.sbt file:

addSbtPlugin("org.scoverage" % "sbt-scoverage" % "2.0.8")

This adds support for scoverage in the project. However, sometimes there’s a chance of version conflict due to the different versions of sbt and plugins. If such an error occurs, we can add the following line to plugins.sbt and build.sbt to force the resolution of some of the transitive dependencies:

ThisBuild / libraryDependencySchemes += "org.scala-lang.modules" %% "scala-xml" % VersionScheme.Always

This instructs sbt to always include the dependency without creating an error for different available versions. Then, we can enable the scoverage in the build.sbt:

coverageEnabled := true

We are now ready to generate code coverage.

4.2. Generating the Coverage

Now that we have enabled coverage, we can execute the tests with coverage:

sbt compile coverage test

This runs the tests with the code coverage enabled. This generates the coverage data for each sub-module’s target directory. We can also generate a report:

sbt coverageReport

This generates the XML and HTML coverage reports inside the directory target/scala-<version>/scoverage-report. We can open the HTML file in a browser to analyze the contents.

In a multi-module build, it becomes easier to aggregate all the sub-module coverage reports into a single report. Here’s how we can do that:

sbt coverageAggregate

This aggregates all the coverage reports into a single report and places them under the root module’s target directory. Moreover, the other reporting tools can integrate the generated XML reports with scoverage.

5. Configurations

sbt-scoverge plugin provides different configurations to customize the coverage generation. Let’s look at some of the useful configurations in this section.

5.1. Exclude Files From Coverage

Sometimes, a need arises to exclude specific files from the coverage calculation. For instance, auto-generated Scala files for Slick tables may be ignored from coverage. Generally, tests typically don’t cover these files, and including them in the calculation would be redundant.

We can ignore such files from coverage by setting the configuration in the build.sbt:

ThisBuild / coverageExcludedFiles := """.*SlickTables;.*/scalaz/.*"""

The setting coverageExcludedFiles accepts the files to be excluded as multiple regexes separated by semi-column. We shouldn’t provide the .scala extension in the regex for the files, as scoverage automatically adds it.

Consequently, scoverage excludes the file SlickTables and all the files under the directory scalaz from the coverage. Notably, this doesn’t yet work with Scala 3.

5.2. Exclude Packages From Coverage

Similar to the exclusion by filename and path, we can exclude coverage by package. To achieve this, we need to modify the build.sbt configuration by adding the package to be excluded:

ThisBuild / coverageExcludedPackages := "<empty>;.*CollectionMonoid.*;"

The above configuration excludes all the classes whose fully qualified name matches the provided regex. In this case, scoverage excludes any classes that don’t have a package due to the special value <empty>. It also excludes any classes containing CollectionMonoid in their fully qualified path. This is also not yet supported for Scala 3.

5.3. Exclude a Part of the Code From Coverage

Apart from excluding an entire package or class, we can exclude specific parts of the codebase from the coverage calculation. This can be achieved by marking the areas to exclude using the directive $COVERAGE-OFF$ and $COVERAGE-ON$:

// $COVERAGE-OFF$
... all code here will be excluded from coverage calculation
// $COVERAGE-ON$

This is useful in ignoring large initialization code blocks without business logic that needs testing.

5.4. Configurations to Enforce Code Coverage

scoverage provides some settings that can be used to enforce a set of minimum coverage criteria. We can set thresholds for different configurations as per our requirements. We can add the following settings to the build.sbt to enable it:

coverageFailOnMinimum := true
coverageMinimumStmtTotal := 90
coverageMinimumBranchTotal := 90

When we run the test with coverage enabled, it marks the build status as a failure if the minimum coverage specified (90%) isn’t reached. This is very useful in ensuring the quality of the codebase.

6. Coverage Report

We can generate coverage reports for the tests executed:

sbt coverageReport

This generates a navigatable HTML report under target/scala-<version>/scoverage-report/index.html. Here’s a sample coverage report:

In a multi-module project, we can aggregate all the reports together:

sbt coverageAggregate

This generates an index.html at the project root level, aggregating all sub-module reports.

7. Conclusion

In this article, we looked at the scoverage sbt plugin and how we can use it to ensure the quality of the codebase. We also discussed the different configuration options to customize the coverage based on the requirements. Additionally, we learned how to generate coverage reports for single and multi-module builds.

In conclusion, sbt-scoverage is a very powerful plugin that helps developers to get valuable insight into the code base to evaluate and improve the code quality.

As always, the sample code used in this tutorial is available over on GitHub.