Code Coverage

1. Introduction

In this tutorial, we’re going to find out what code coverage is, how to measure it and how it improves the quality of developed software.

Code coverage describes the percentage of code covered by automated tests. In other words, it checks which parts of code run during the test suite and which don’t.

2. Importance of Code Coverage

The primary goal of the software developing process is to create high quality, faultless applications that meet our requirements.

One of the key parts of that process is testing. The big gain of unit tests is that they protect already implemented features from being broken as code changes. Also, unit tests give developers a sense of protecting the application from bugs.

But, is that all really true?

First of all, we can’t say that implementing unit tests gives our application reliable protection from bugs unless we’re using a code coverage approach. Code coverage is a metric we use to quantify how much of our code based is being reached by our unit tests.

Creating unit tests and not knowing for sure if they cover at least the most important scenarios, paths, and edge cases seems to be a profitless idea. Code coverage gives developers critical information about which part of codes are not tested, therefore it may result in errors or missed requirements.

Second, taking the code coverage approach from the very beginning of the project eliminates possible bugs in the early stage of the development life cycle. Consequently, we may take on less technical debt.

Third, all of the above-mentioned favors results in shorter development and maintenance time. And from a business point of view, all of that brings better ROI. It can also lead to improved customer satisfaction by reducing UAT and production problems.

As we can see, using code coverage improves the software development process in all stages and satisfies all groups of interest.

In the next section, we’ll see the most common ways to measure code coverage.

3. Ways to Measure Code Coverage

3.1. Statement Coverage

The main purpose of statement coverage is to run each executable statement in the program’s code at least once. It’s also often called line coverage.

So, let’s define what a statement is. A statement is the smallest unit in an imperative programming language that represents some action.

Percentage of statement coverage can be calculated as:

Statement coverage = Number of executed statements / Total number of statements * 100

That way of measuring code coverage is able to:

• verify the do’s and don’ts of the written code
• find dead code and unused statements
• test different flow paths and checks which ones are not covered

Let’s analyze a simple example. Below we’re given pseudocode of program that sums two numbers and prints information if the result is less than, greater than or equals zero:

algorithm SUM(a, b):
    // INPUT
    //   a, b = Numbers to add
    // OUTPUT
    //   Prints a message based on the sum of a and b
    result <- a + b
    if result > 0:
        print("Greater than zero")
    else if result < 0:
        print("Less than zero")
    else:
        print("Zero")

Now, let’s talk about possible tests. In the first option, we have given a = 3, b = 5. Statements at lines 1, 2, 3 and 4 will be executed during this test case. So we’re covering four statements out of eight. That means our statement coverage equals 50%.

Let’s discuss another option with a = 3, b = -5. In this scenario statements at lines 1, 2, 3, 5, 6 will be executed. Both tests altogether call lines 1 – 6. Total code coverage equals (6/8) * 100 = 75%.

3.2. Branch Coverage

Branch coverage ensures if each decision in a decision-making tree is executed at least once. By branches we mean: conditional statements, loops, switch statements. We can calculate branch coverage using the below formula:

Branch coverage = Number of executed branches / Total number of branches * 100

Advantages of branch coverage:

• verify if the execution of the test suite reaches all branches
• detects possible abnormal behavior of each branch
• can test areas of the source code that other approaches may discount

The pseudocode given below represents a program that prints passed numbers and also prints additional comments if the number is even:

algorithm EVEN(a):
    // INPUT
    //   a = the number to check for evenness
    // OUTPUT
    //   Prints a message if a is even and then prints the value of a
    if a mod 2 = 0:
        print("Even")
    print(a)

Below we can see the flow chart illustrating the above algorithm:

We can see three arrows in the diagram. Those are branches of a given program. Two of them described with words “Yes” and “No” are called conditional branches. Arrow without a label is an unconditional branch. Branch coverage takes into consideration both types.

Let’s now explore two test scenarios:

1. Given a = 1. The test case will go only through branch labeled with “No”. Branch coverage: 1/3 * 100 = 33%.
2. Given a = 4. The test case will go through a branch labeled with “Yes” and the unconditional one. Branch coverage: 2/3 * 100 = 67%

Clearly, both tests together cover 100% of branches.

3.3. Function Coverage

Function coverage verifies if each function of a program is being called at least once. It is also important to test functions with different input parameters. That way, test suites will also check if functions behave properly in different scenarios.

To calculate function coverage we can use the following formula:

Function coverage = Number of executed functions / Total number of functions * 100

Function coverage is the broadest compared to the rest.

Let’s say, for example, that our application consists only of a single method. Implementing a single unit test for that method will result in 100% function coverage. Obviously, one unit test is not able to cover all paths and scenarios. Despite 100% function coverage, our application is clearly not well-tested.

As we can see a high percentage of code coverage doesn’t always mean that code is ideal and faultless.

We’ll discuss that topic in the upcoming section.

4. What Percentage of Coverage to Target?

The answer to that question may seem obvious. Most often the first thing that comes to mind is that we should target as close to 100% as we can, thus it will bring our software near to perfection. Reality shows it’s not that simple.

First of all, it depends on the current project state. If we measure code coverage from the start of software development it’s possible to get coverage above 90%, which is a great result. Especially if we use the TDD approach.

Achieving the same code coverage in a large, legacy project with little or no tests may be nearly impossible. For such cases the best approach is to create tests for new features and consequently cover existing ones with tests while modifying, fixing or extending them. Hence, one could imagine even 30% as a success for their legacy project.

Secondly, enforcing 100% code coverage shouldn’t be the aim in itself.  It may be even harmful to software quality. Developers most often will write useless or misguided tests just to reach the goal. That behavior can result in missed or even wrongly implemented requirements because developers will be more focused on covering as much code as they can rather than on the valid business logic itself.

Finally, in the previous section, we’ve noticed that 100% code coverage doesn’t mean there are no bugs in the code. It depends on the chosen code coverage metric. Besides that, the features can be faultless and tested well but they might work not as intended. Verifying domain errors is a thing that code coverage may not handle.

Code coverage is a useful tool but we shouldn’t treat it as a primary goal. We should take care of our code quality by creating high quality, meaningful tests for features implemented as per requirements.

5. Code Coverage Tools

In this section, we’re going to briefly introduce some code coverage tools for common programming languages.

5.1. JaCoCo

JaCoCo is a great tool for Java developers. It is part of the Eclipse Foundation. It can easily integrate with the most important frameworks like Gradle, Maven, IntelliJ IDEA, Eclipse and many more. JaCoCo generates powerful and detailed code coverage reports. Moreover, as an open-source tool, it’s free for personal and commercial use.

5.2. Istambul

Javascript developers can use a convenient tool named Istambul. It supports ES5 and ES2015+. Istambul can integrate with a variety of unit tests frameworks like tap, mocha or AVA. The output is provided in terminal or HTML. Just like JaCoCo is open-source and free to use both for personal and commercial purposes.

More information can be found on Istambul’s official website.

5.3. Coverage.py

Python developers can use Coverage.py. It works from the command line, as an API, or can integrate with test runners. Coverage.py is open source and free for all use cases.

We can visit Coverage.py’s website for more details.

5.4. NCover

NCover provides enterprise code coverage solutions for the .NET platform. It was mainly designed for teams and organizations. It allows developers and QA teams to cooperate using collaborative features. NCover is an advanced, robust, enterprise tool that is not intended to use for personal purposes. Thus, it is neither free nor open-source.

6. Conclusion

In this article, we’ve defined what code coverage term means and we’ve analyzed various ways to measure it. Furthermore, we’ve discussed why code coverage is important in a software development process and how much code coverage should we target. In addition, a few code coverage tools were introduced.

To conclude, remember that 100% code coverage shouldn’t be blindly followed like a necessary trend. Focus on the quality of tests rather than quantity and implement features as per requirements.