In this article, we'll make a simple comparison between two well-known Java frameworks, Spring Boot and Quarkus. At the end of it, we'll better understand their differences and similarities and some particularities.
Also, we'll perform some tests to measure their performance and observe their behaviour.
2. Spring Boot
Spring Boot is a Java-based framework focusing on enterprise applications. It connects all Spring projects and helps to accelerate developers' productivity by offering many production-ready integrations.
By doing this, it reduces the amount of configuration and boilerplate. Furthermore, thanks to its convention over configuration approach, which automatically registers default configurations based on the dependencies available at the classpath in the runtime, Spring Boot considerably reduces the time-to-market for many Java applications.
Quarkus is another framework with a similar approach as the Spring mentioned above Boot, but with an additional promise of delivering smaller artifacts with fast boot time, better resource utilization, and efficiency.
It's optimized for cloud, serverless, and containerized environments. But despite this slightly different focus, Quarkus also integrates well with the most popular Java frameworks.
As mentioned above, both frameworks integrate well with other projects and frameworks. However, their internal implementations and architectures are different. For example, Spring Boot offers web capabilities in two flavors: blocking (Servlets) and non-blocking (WebFlux).
On the other hand, Quarkus also offers both approaches, but unlike Spring Boot, it allows us to use both blocking and non-blocking strategies simultaneously. Moreover, Quarkus has the reactive approach embedded in its architecture.
For that reason, we'll use two entirely reactive applications implemented with Spring WebFlux and Quarkus reactive capabilities to have a more exact scenario in our comparison.
Also, one of the most significant features available in the Quarkus project is the ability to create native images (binary and platform-specific executables). So, we'll also include both native images in the comparison, but in the case of Spring, native image support is still in the experimental phase. To do this, we need the GraalVM.
4.1. Test Applications
Our application will expose three APIs: one allowing the user to create a zip code, the other to find the information of a particular zip code, and lastly, querying zip codes by city. These APIs were implemented using both Spring Boot and Quarkus entirely using the reactive approach, as already mentioned, and using a MySQL database.
The goal was to have a simple sample application but with a little more complexity than a HelloWorld app. Of course, this will affect our comparison as the implementation of things like database drivers and serialization frameworks will influence the result. However, most applications are likely to deal with those things as well.
So, our comparison doesn't aim to be the ultimate truth about which framework is better or more performant, but rather a case study that will analyze these particular implementations.
4.2. Test Planning
To test both implementations, we'll use Wrk to perform the test, and its metrics report to analyze our findings. Also, we'll use VisualVM to monitor the applications' resource utilization during the test execution.
The test will run for 7 minutes, where all APIs will be called, starting with a warmup period and after increasing the number of connections until reaching 100 of them. Wrk can generate a significant amount of load with this setup:
All the tests were performed on a machine with the following specifications:
Although not ideal because of the lack of isolation from other background processes, the test only aims to illustrate the proposed comparison. It's not the intention to provide an extensive and detailed analysis of the performance of both frameworks, as already mentioned.
Another point worth mentioning is that depending on our machine specification, we may need to adjust the number of connections, threads, etc.
4.3. Knowing Our Tests
It's essential to ensure we are testing the right thing, so to do that, we'll use Docker containers to deploy our infra. This will allow us to control the resource constraints of both the application and database. The goal is to stress the application now the underlying system, our the database. For this example, just limiting the number of available CPUs is enough, but this may change depending on the resources available in our machines.
To restrict the sources available, we can use the Docker settings, cpulimit command, or any other tool we prefer. Moreover, we may use the docker stats and top commands to monitor the system's resources. Last in regarding memory we will measure the heap usage and also the RSS and to that let's use the ps (ps -o pid,rss,command -p <pid>) command.
The developer experience was great for both projects, but it's worth mentioning that Spring Boot has better documentation and more material than we can find online. Quarkus is improving in this area and has a vast set of features that helps increase productivity. However, considering documentation and stack overflow issues, it's still behind.
In terms of metrics, we have:
With this experiment, we could observe that Quarkus was faster than Spring Boot in terms of startup time both in JVM and native versions. Furthermore, Quarkus build time was also much quicker in the case of native images. The build took 91 seconds (Quarkus) vs 113 seconds (Spring Boot), and the JVM build took 5.24 seconds (Quarkus) vs 1.75 seconds (Spring Boot), so point for Spring in this one.
Regarding artifact size, the runnable artifacts produced by Spring Boot and Quarkus were similar in terms of the JVM version, but in the case of native ones, Quarkus did a better job.
However, regarding other metrics, the conclusions are not straightforward. So, let's take a deeper look at some of them.
If we focus on the CPU usage, we'll see that the JVM versions consume more CPU at the beginning during the warmup phase. After that, the CPU usage stabilizes, and the consumption becomes relatively equal to all the versions.
Here are the CPU consumptions for Quarkus in JVM and Native versions, in that order:
Quarkus did better in both cases. However, the difference was so small that a tie could also be considered. Another point worth mentioning is that in the graph, we see the consumption based on the number of CPUs available in the machine. Still, to ensure we were stressing the option and not other parts of the system, we have limited the number of cores available to the application to three.
Regarding memory, it's even more complicated. First, the JVM versions of both frameworks reserve more memory for the heap, almost the same amount of memory. Regarding heap usage, the JVM versions consume more memory than the native ones, but looking at the pairs, Quarkus seems to consume slightly less than Spring in the JVM version. But, again, the difference is super tiny.
(Spring Boot JVM)
Then, looking at the native images, things seem to have changed. The Spring Native version seems to collect memory more frequently and keeps a lower memory footprint.
(Spring Boot Native)
Another important highlight is that Quarkus seems to overtake Spring in both versions when it comes to RSS memory measurement. We only added the RSS comparison at the startup time, but we can also use the same command during the tests.
Nevertheless, in this comparison, we only used the default parameters. Therefore, no changes were made to GC, JVM options, or any other parameters. Different applications may need different settings, we should have this in mind when using them in a real-life environment.
5.3. Response Time
Lastly, we'll use a different approach regarding response times as many of the benchmark tools available suffer from a problem called Coordinated Omission. We'll use hyperfoil, a tool designed to avoid this issue. During the test, many requests are created, but the idea is not to stress the application too much but rather just enough to measure its response time.
Though, the test structure is pretty much similar to the previous one.
(Spring Boot JVM)
Throughput and response time are not the same thing although related, they measure different things. Quarkus JVM version had a good performance under pressure and also when it comes to moderate load. It seems to have higher throughput and a slightly lower response time.
(Spring Boot Native)
Looking at the native versions, the numbers change again. Now, Spring seems to have a slightly lower response time and higher throughput overall. However, looking at all the numbers, we can see that the difference is too small to define any clear winner.
5.4. Connecting the Dots
All things considered, both frameworks proved to be great options for implementing Java applications.
The native apps have shown to be fast and to have low resource consumption, being excellent choices for serverless, short-living applications and environments where low resource consumption is critical.
On the other hand, the JVM apps seem to have more overhead but excellent stability and high throughput over time, ideal for robust, long-living applications.
Finally, regarding the performance, all the versions have robust performance when compared, at least for our example. The difference is so tiny that we can say they have similar performance. Of course, we can argue that the JVM versions handled the heavy load better in terms of throughput while consuming more resources, and on the other hand, the native versions consumed less. However, this difference may not even be relevant depending on the use case.
Last, I have to point out that in the Spring application, we had to switch the DB driver because one recommended by the documentation had an issue. In contrast, Quarkus worked out of the box without any problems.
This article compares the Spring Boot and Quarkus frameworks and their different deployment modes, JVM and Native. We also looked at other metrics and aspects of those applications. As usual, the code of the test application and scripts used to test them are available over on GitHub.