Benchmark JDK Collections vs Eclipse Collections

1. Introduction

In this tutorial, we’re going to compare the performance of traditional JDK collections with Eclipse Collections. We’ll create different scenarios and explore the results.

2. Configuration

First, note that for this article, we’ll use the default configuration to run the tests. No flags or other parameters will be set on our benchmark.

We’ll use the following hardware and libraries:

• JDK 11.0.3, Java HotSpot(TM) 64-Bit Server VM, 11.0.3+12-LTS.
• MacPro 2.6GHz 6-core i7 with 16GB DDR4.
• Eclipse Collections 10.0.0 (latest available at the time of writing)
• We’ll leverage JMH (Java Microbenchmark Harness) to run our benchmarks
• JMH Visualizer to generate charts from JMH results

The easiest way to create our project is via the command-line:

mvn archetype:generate \
  -DinteractiveMode=false \
  -DarchetypeGroupId=org.openjdk.jmh \
  -DarchetypeArtifactId=jmh-java-benchmark-archetype \
  -DgroupId=com.baeldung \
  -DartifactId=benchmark \
  -Dversion=1.0

After that, we can open the project using our favorite IDE and edit the pom.xml to add the Eclipse Collections dependencies:

<dependency>
    <groupId>org.eclipse.collections</groupId>
    <artifactId>eclipse-collections</artifactId>
    <version>10.0.0</version>
</dependency>
<dependency>
    <groupId>org.eclipse.collections</groupId>
    <artifactId>eclipse-collections-api</artifactId>
    <version>10.0.0</version>
</dependency>

3. First Benchmark

Our first benchmark is simple. We want to calculate the sum of a previously created List of Integers.

We’ll test six different combinations while running them in serial and parallel:

private List<Integer> jdkIntList;
private MutableList<Integer> ecMutableList;
private ExecutorService executor;
private IntList ecIntList;

@Setup
public void setup() {
    PrimitiveIterator.OfInt iterator = new Random(1L).ints(-10000, 10000).iterator();
    ecMutableList = FastList.newWithNValues(1_000_000, iterator::nextInt);
    jdkIntList = new ArrayList<>(1_000_000);
    jdkIntList.addAll(ecMutableList);
    ecIntList = ecMutableList.collectInt(i -> i, new IntArrayList(1_000_000));
    executor = Executors.newWorkStealingPool();
}

@Benchmark
public long jdkList() {
    return jdkIntList.stream().mapToLong(i -> i).sum();
}

@Benchmark
public long ecMutableList() {
    return ecMutableList.sumOfInt(i -> i);
}

@Benchmark
public long jdkListParallel() {
    return jdkIntList.parallelStream().mapToLong(i -> i).sum();
}

@Benchmark
public long ecMutableListParallel() {
    return ecMutableList.asParallel(executor, 100_000).sumOfInt(i -> i);
}

@Benchmark
public long ecPrimitive() { 
    return this.ecIntList.sum(); 
}

@Benchmark
public long ecPrimitiveParallel() {
    return this.ecIntList.primitiveParallelStream().sum(); 
}

To run our first benchmark we need to execute:

mvn clean install
java -jar target/benchmarks.jar IntegerListSum -rf json

This will trigger the benchmark at our IntegerListSum class and save the result to a JSON file.

We’ll measure the throughput or number of operations per second in our tests, so the higher the better:

Benchmark                              Mode  Cnt     Score       Error  Units
IntegerListSum.ecMutableList          thrpt   10   573.016 ±    35.865  ops/s
IntegerListSum.ecMutableListParallel  thrpt   10  1251.353 ±   705.196  ops/s
IntegerListSum.ecPrimitive            thrpt   10  4067.901 ±   258.574  ops/s
IntegerListSum.ecPrimitiveParallel    thrpt   10  8827.092 ± 11143.823  ops/s
IntegerListSum.jdkList                thrpt   10   568.696 ±     7.951  ops/s
IntegerListSum.jdkListParallel        thrpt   10   918.512 ±    27.487  ops/s

Accordingly to our tests, Eclipse Collections’ parallel list of primitives had the highest throughput of all. Also, it was the most efficient with a performance almost 10x faster than the Java JDK running also in parallel.

Of course, a portion of that can be explained by the fact that when working with primitive lists, we don’t have the cost associated with boxing and unboxing.

We can use JMH Visualizer to analyze our results. The chart below shows a better visualization:

4. Filtering

Next, we’ll modify our list to get all elements that are multiple of 5. We’ll reuse a big portion of our previous benchmark and a filter function:

private List<Integer> jdkIntList;
private MutableList<Integer> ecMutableList;
private IntList ecIntList;
private ExecutorService executor;

@Setup
public void setup() {
    PrimitiveIterator.OfInt iterator = new Random(1L).ints(-10000, 10000).iterator();
    ecMutableList = FastList.newWithNValues(1_000_000, iterator::nextInt);
    jdkIntList = new ArrayList<>(1_000_000);
    jdkIntList.addAll(ecMutableList);
    ecIntList = ecMutableList.collectInt(i -> i, new IntArrayList(1_000_000));
    executor = Executors.newWorkStealingPool();
}

@Benchmark
public List<Integer> jdkList() {
    return jdkIntList.stream().filter(i -> i % 5 == 0).collect(Collectors.toList());
}

@Benchmark
public MutableList<Integer> ecMutableList() {
    return ecMutableList.select(i -> i % 5 == 0);
}


@Benchmark
public List<Integer> jdkListParallel() {
    return jdkIntList.parallelStream().filter(i -> i % 5 == 0).collect(Collectors.toList());
}

@Benchmark
public MutableList<Integer> ecMutableListParallel() {
    return ecMutableList.asParallel(executor, 100_000).select(i -> i % 5 == 0).toList();
}

@Benchmark
public IntList ecPrimitive() {
    return this.ecIntList.select(i -> i % 5 == 0);
}

@Benchmark
public IntList ecPrimitiveParallel() {
    return this.ecIntList.primitiveParallelStream()
      .filter(i -> i % 5 == 0)
      .collect(IntLists.mutable::empty, MutableIntList::add, MutableIntList::addAll);
}

We’ll execute the test just like before:

mvn clean install
java -jar target/benchmarks.jar IntegerListFilter -rf json

And the results:

Benchmark                                 Mode  Cnt     Score    Error  Units
IntegerListFilter.ecMutableList          thrpt   10   145.733 ±  7.000  ops/s
IntegerListFilter.ecMutableListParallel  thrpt   10   603.191 ± 24.799  ops/s
IntegerListFilter.ecPrimitive            thrpt   10   232.873 ±  8.032  ops/s
IntegerListFilter.ecPrimitiveParallel    thrpt   10  1029.481 ± 50.570  ops/s
IntegerListFilter.jdkList                thrpt   10   155.284 ±  4.562  ops/s
IntegerListFilter.jdkListParallel        thrpt   10   445.737 ± 23.685  ops/s

As we can see, the Eclipse Collections Primitive was the winner again. With a throughput more than 2x faster than the JDK parallel list.

Note that for filtering, the effect of parallel processing is more visible. Summing is a cheap operation for the CPU and we won’t see the same differences between serial and parallel.

Also, the performance boost that Eclipse Collections primitive lists got earlier begins to evaporate as the work done on each element begins to outweigh the cost of boxing and unboxing.

To finalize, we could see that operations on primitives are faster than objects:

5. Conclusion

In this article, we created a couple of benchmarks to compare Java Collections with Eclipse Collections. We’ve leveraged JMH to try to minimize the environment bias.