Generic Top

I just announced the new Learn Spring course, focused on the fundamentals of Spring 5 and Spring Boot 2:

>> CHECK OUT THE COURSE

1. Introduction

Although executing tests serially works just fine most of the time, we may want to parallelize them to speed things up.

In this tutorial, we'll cover how to parallelize tests using JUnit and Maven's Surefire Plugin. First, we'll run all tests in a single JVM process, then we'll try it with a multi-module project.

2. Maven Dependencies

Let's begin by importing the required dependencies. We'll need to use JUnit 4.7 or later along with Surefire 2.16 or later:

<dependency>
    <groupId>junit</groupId>
    <artifactId>junit</artifactId>
    <version>4.12</version>
    <scope>test</scope>
</dependency>
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>2.22.0</version>
</plugin>

In a nutshell, Surefire provides two ways of executing tests in parallel:

  • Multithreading inside a single JVM process
  • Forking multiple JVM processes

3. Running Parallel Tests

To run a test in parallel we should use a test runner that extends org.junit.runners.ParentRunner.

However, even tests that don't declare an explicit test runner work, as the default runner extends this class.

Next, to demonstrate parallel test execution, we'll use a test suite with two test classes each having a few methods. In fact, any standard implementation of a JUnit test suite would do.

3.1. Using Parallel Parameter

First, let's enable parallel behavior in Surefire using the parallel parameter. It states the level of granularity at which we'd like to apply parallelism.

The possible values are:

  • methods – runs test methods in separate threads
  • classes – runs test classes in separate threads
  • classesAndMethods – runs classes and methods in separate threads
  • suites – runs suites in parallel
  • suitesAndClasses – runs suites and classes in separate threads
  • suitesAndMethods – creates separate threads for classes and for methods
  • all – runs suites, classes as well as methods in separate threads

In our example, we use all:

<configuration>
    <parallel>all</parallel>
</configuration>

Second, let's define the total number of threads we want Surefire to create. We can do that in two ways:

Using threadCount which defines the maximum number of threads Surefire will create:

<threadCount>10</threadCount>

Or using useUnlimitedThreads parameter where one thread is created per CPU core:

<useUnlimitedThreads>true</useUnlimitedThreads>

By default, threadCount is per CPU core. We can use the parameter perCoreThreadCount to enable or disable this behavior:

<perCoreThreadCount>true</perCoreThreadCount>

3.2. Using Thread-Count Limitations

Now, let's say we want to define the number of threads to create at the method, class, and suite level. We can do this with the threadCountMethods, threadCountClasses and threadCountSuites parameters.

Let's combine these parameters with threadCount from the previous configuration: 

<threadCountSuites>2</threadCountSuites>
<threadCountClasses>2</threadCountClasses>
<threadCountMethods>6</threadCountMethods>

Since we used all in parallel, we've defined the thread counts for methods, suites, and classes. However, it isn't mandatory to define the leaf parameter. Surefire deduces the number of threads to use in case leaf parameters are omitted.

For example, if threadCountMethods is omitted, then we just need to make sure threadCount > threadCountClasses threadCountSuites.

Sometimes we may want to limit the number of threads created for classes or suites or methods even while we're using an unlimited number of threads.

We can apply thread-count limitations in such cases as well:

<useUnlimitedThreads>true</useUnlimitedThreads>
<threadCountClasses>2</threadCountClasses>

3.3. Setting Timeouts

Sometimes we may need to ensure that test execution is time-bounded.

To do that we can use the parallelTestTimeoutForcedInSeconds parameter. This will interrupt currently running threads and will not execute any of the queued threads after the timeout has elapsed:

<parallelTestTimeoutForcedInSeconds>5</parallelTestTimeoutForcedInSeconds>

Another option is to use parallelTestTimeoutInSeconds.

In this case, only the queued threads will be stopped from executing:

<parallelTestTimeoutInSeconds>3.5</parallelTestTimeoutInSeconds>

Nevertheless, with both options, the tests will end with an error message when the timeout has elapsed.

3.4. Caveats

Surefire calls static methods annotated with @Parameters, @BeforeClass, and @AfterClass in the parent thread. Thus make sure to check for potential memory inconsistencies or race conditions before running tests in parallel.

Also, tests that mutate shared state are definitely not good candidates for running in parallel.

4. Test Execution in Multi-Module Maven Projects

Till now, we've focused on running tests in parallel within a Maven module.

But let's say we have multiple modules in a Maven project. Since these modules are built sequentially, the tests for each module are also executed sequentially.

We can change this default behavior by using Maven's -T parameter which builds modules in parallel. This can be done in two ways.

We can either specify the exact number of threads to use while building the project:

mvn -T 4 surefire:test

Or use the portable version and specify the number of threads to create per CPU core:

mvn -T 1C surefire:test

Either way, we can speed up tests as well as build execution times.

5. Forking JVMs

With the parallel test execution via the parallel option, concurrency happens inside the JVM process using threads.

Since threads are sharing the same memory space, this can be efficient in terms of memory and speed. However, we may encounter unexpected race conditions or other subtle concurrency-related test failures. As it turns out, sharing the same memory space can be both a blessing and a curse.

To prevent thread-level concurrency issues, Surefire provides another parallel test execution mode: forking and process-level concurrency. The idea of forked processes is actually quite simple. Instead of spawning multiple threads and distributing the test methods between them, surefire creates new processes and does the same distribution.

Since there's no shared memory between different processes, we won't suffer from those subtle concurrency bugs. Of course, this comes at the expense of more memory usage and a little less speed.

Anyway, in order to enable forking, we just have to use the forkCount property and set it to any positive value:

<forkCount>3</forkCount>

Here, surefire will create at most three forks from the JVM and run the tests in them. The default value for forkCount is one, which means that maven-surefire-plugin creates one new JVM process to execute all tests in one Maven module.

The forkCount property supports the same syntax as -T. That is, if we append the to the value, that value will be multiplied with the number of available CPU cores in our system. For instance:

<forkCount>2.5C</forkCount>

Then in a two-core machine, Surefire can create at most five forks for parallel test execution.

By default, Surefire will reuse the created forks for other tests. However, if we set the reuseForks property to false, it'll destroy each fork after running one test class.

Also, in order to disable the forking, we can set the forkCount to zero.

6. Conclusion

To sum up, we started off by enabling multi-threaded behavior and defining the degree of parallelism using the parallel parameter. Subsequently, we applied limitations on the number of threads Surefire should create. Later, we set timeout parameters to control test execution times.

Finally, we looked at how we can reduce build execution times and therefore test execution times in multi-module Maven projects.

As always, the code presented here is available on GitHub.

Generic bottom

I just announced the new Learn Spring course, focused on the fundamentals of Spring 5 and Spring Boot 2:

>> CHECK OUT THE COURSE
2 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Jan
Jan
1 year ago

I cloned this code and ran mvn test and the tests did not run in parallel. I added sleeps to make sure that all tests should be started before any finishes. Can you confirm parallel running?

Loredana Crusoveanu
1 year ago
Reply to  Jan

Hey Jan,

Yes, this works for me. I added sleeps of 4 seconds in each method in ArithmeticFunctionTest and the total time of the test was still aprox 4 seconds.

For the multi-module run, I can confirm the same thing.

As for “all tests should be started before any finishes” – I don’t think that’s a guarantee. Each test starts when there is an available thread.

Comments are closed on this article!