Gray Box Testing Using the OAT Technique

1. Overview

Gray box testing helps us build adequate test coverage without testing every possible scenario.

In this tutorial, we’ll examine that approach and how to exercise it using the Orthogonal Array Testing (OAT) technique.

Finally, we’ll identify the advantages and disadvantages of using gray box testing.

2. What Is Gray Box Testing?

First, let’s compare white vs. black box testing approaches, and then understand gray box testing.

White box testing refers to testing parts of an algorithm fully known by us. Hence, we can test all paths of that algorithm. And because of that, white box testing might produce a high number of test scenarios. 

Black box testing means testing the external perspective of the application. In other words, we don’t know anything about the algorithm implemented, and it’s harder to test all paths of it. Thus, we focus on validating a limited number of test scenarios.

Gray box testing uses limited information, typically available in white box testing. Then, it uses a black box testing technique to produce the test scenarios with the available information.

Thus, we end up with fewer test scenarios than white box testing. However, those scenarios cover more functionalities than black box testing.

Thus, gray box testing is a mix of a black box testing techniques and white box testing knowledge.

3. Exercising Gray Box Testing

In this section, we’ll exercise gray box testing using the OAT technique on a commission calculator demo application.

3.1. Create the System Under Test

Before testing, let’s first create an application to calculate a salesperson’s average commission based on four attributes:

• Salesperson Level – L1, L2, or L3
• Contract Type – Full-time Commissioned, Contractor, or Freelancer
• Seniority – Junior, Mid, Senior
• Impact of the sales made – Low, Medium, High

To achieve that, let’s create the SalaryCommissionPercentageCalculator class to address the above requirements:

public class SalaryCommissionPercentageCalculator {
    public BigDecimal calculate(Level level, Type type, 
      Seniority seniority, SalesImpact impact) {
        return BigDecimal.valueOf(DoubleStream.of(
          level.getBonus(),
          type.getBonus(),
          seniority.getBonus(),
          impact.getBonus(),
          type.getBonus())
          .average()
          .orElse(0))
          .setScale(2, RoundingMode.CEILING);
    }

    public enum Level {
        L1(0.06), L2(0.12), L3(0.2);
        private double bonus;

        Level(double bonus) {
            this.bonus = bonus;
        }

        public double getBonus() {
            return bonus;
        }
    }

    public enum Type {
        FULL_TIME_COMMISSIONED(0.18), CONTRACTOR(0.1), FREELANCER(0.06);

        // bonus field, constructor and getter
    }

    public enum Seniority {
        JR(0.8), MID(0.13), SR(0.19);

        // bonus field, constructor and getter
    }

    public enum SalesImpact {
        LOW(0.06), MEDIUM(0.12), HIGH(0.2);

        // bonus field, constructor and getter
    }
}

The code above defines four enums to map the salesperson’s attributes. Each enum contains a bonus field representing the commission percentage of each attribute.

The calculate() method uses a double primitive stream to compute the average of all percentages.

Finally, we round the average result to two decimal places using the setScale() method from the BigDecimal class.

3.2. Brief Introduction to the OAT Technique

The OAT technique is based on Taguchi’s Design experiment proposed by Dr. Genichi Taguchi. That experiment allows us to consider only a subset of all input combinations to test interactions between variables. 

The idea is to consider only two-factor interactions between variables’ values and ignore repeated interactions when building experiments. That means each variable’s value interacts precisely once with another variable’s value in the subset of experiments. This will become clear when we build the test scenarios.

The variables and their values are used to construct Orthogonal Arrays. An Orthogonal Array is an array of numbers where each row represents a unique combination of variables. The columns represent the individual variable that can take on one of several values.

We can represent Orthogonal Arrays as val^var, where val is the number of values they assume, and var is the number of input variables. In our case, we have four variables where each assumes three values. Thus, val equals 3, and var equals 4.

Finally, the correct Orthogonal Array is the 3^4, also known as “L9: 3-level 4-factor” in Taguchi’s Design.

3.3. Get the Orthogonal Array

The computation of an Orthogonal Array can be too complex and computationally expensive. For that reason, designers of OAT tests typically use a list of already mapped arrays. Hence, we can use that catalog of arrays to find the correct one. In our case, the correct array in the catalog provided is the L9 3-level 4-factor array:

The table above contains two additional headers we’ve added to the 3^4 orthogonal array. The header in the first row defines the variables, whereas the first column defines the test scenario number.

Notably, among all scenarios, two values only interact with each other once. We don’t repeat the same pair of values in other scenarios. For instance, the pair where var1=val1 and var2=val1 only appears in the first test scenario.

3.4. Map the Variables and Their Values

Now, we must substitute the variables and their values to our orthogonal array in the same order they appear in the code. So, for example, the var 1 corresponds to the first enum defined, Level, where the val 0 below Level is its first value, L1.

After mapping all variables, we get the filled-in table below:

Each row of the table above corresponds to one test scenario using the values of each corresponding cell.

3.5. Configuring JUnit 5

The main focus of this article is to exercise gray box testing using the OAT gray box technique. So, for simplicity, we’ll use straightforward unit tests to illustrate it.

First, we need to configure JUnit 5 in our project. To do so, let’s add its latest dependency, junit-jupiter-engine, to our pom.xml file:

<dependency>
    <groupId>org.junit.jupiter</groupId>
    <artifactId>junit-jupiter-engine</artifactId>
    <version>5.9.2</version>
    <scope>test</scope>
</dependency>

3.6. Create the Test Class

Let’s define the SalaryCommissionPercentageCalculatorUnitTest class:

class SalaryCommissionPercentageCalculatorUnitTest {
    private SalaryCommissionPercentageCalculator testTarget = new SalaryCommissionPercentageCalculator();

    @ParameterizedTest
    @MethodSource("provideReferenceTestScenarioTable")
    void givenReferenceTable_whenCalculateAverageCommission_thenReturnExpectedResult(Level level,
      Type type, Seniority seniority, SalesImpact impact, double expected) {
        BigDecimal got = testTarget.calculate(level, type, seniority, impact);
        assertEquals(BigDecimal.valueOf(expected), got);
    }

    private static Stream<Arguments> provideReferenceTestScenarioTable() {
        return Stream.of(
                Arguments.of(L1, FULL_TIME_COMMISSIONED, JR, LOW, 0.26),
                Arguments.of(L1, CONTRACTOR, SR, MEDIUM, 0.12),
                Arguments.of(L1, FREELANCER, MID, HIGH, 0.11),
                Arguments.of(L2, FULL_TIME_COMMISSIONED, SR, HIGH, 0.18),
                Arguments.of(L2, CONTRACTOR, MID, LOW, 0.11),
                Arguments.of(L2, FREELANCER, JR, MEDIUM, 0.24),
                Arguments.of(L3, FULL_TIME_COMMISSIONED, MID, MEDIUM, 0.17),
                Arguments.of(L3, CONTRACTOR, JR, HIGH, 0.28),
                Arguments.of(L3, FREELANCER, SR, LOW, 0.12)
        );
    }
}

To understand what’s happening, let’s break down the code.

The test method uses JUnit 5 Parameterized Tests with the @MethodSource annotation to use a method as an input data provider.

provideReferenceTestScenarioTable() provides the same data in the Orthogonal Array of section 3.4. as a Stream of arguments. Each Argument.of() call corresponds to a test scenario and the calculate() call’s expected result.

Finally, we call calculate() using the provided parameters and use assertEquals() to compare the actual with the expected result.

4. Pros and Cons of Gray Box Testing

In our example, the total number of permutations for the input of calculate() is 81. We used OAT to reduce that number to 9 while maintaining good test coverage. 

Trying all input combinations may become difficult if the input size becomes too big. For example, in a system with 10 variables and 10 values, the total number of scenarios would be 10 e10. Testing such a high number of scenarios is impractical. We can reduce that number by using OAT and avoid a combinatorial explosion of the input.

Therefore, the main advantage of the OAT technique is that it improves test code maintainability and development speed without losing test coverage. 

On the other hand, the OAT technique and gray box testing generally have the downside of not covering all possible input permutations. Thus, we might miss an essential test scenario or a problematic edge case.

5. Conclusion

In this article, we’ve examined the OAT technique to understand gray box testing.

Using this technique, we drastically reduced the number of test scenarios. However, we must properly evaluate when to use it as we might miss important edge cases.

As always, the example code is available over on GitHub.