Spring Fruits Benchmark

Abstract

Benchmarks are tricky to do well, and the results are often hard to interpret. This analysis attempts to go beyond a simple headline number to explore how performance varies with data set size. The results show that while results might be disappointing for a given data set size, the performance characteristics can change as the size changes, leading to different conclusions about scalability and efficiency. The result is a deeper insight into how a framework handles increasing loads and the factors that influence performance.

For a data set size N, the cycle time (inverse throughput) is linear in N, so t = A + B*N, with A a baseline framework cost per HTTP request, and B the processing time per data item. We measure A and B for a simple endpoint that fetches and renders a complete data set in JSON. A is small (on the order of tens of microseconds). Probably for a "real" application (not just CRUD) the processing time B will increase, dominating performance for large data sets, and having a less dramatic impact for small data sets.

Upgrading the Spring Framework to version 7.0.6 and switching on virtual threads in the Spring Boot application both add meaningful improvements in throughput. Even with such simple changes, we are able to improve the throughput by as much as 50% for small data sets.

Introduction and Set Up

The application code is taken from some public benchmarks. The data represent fruits with links to the stores that stock them and the prices listed. The /fruits endpoint returns a list of fruits and prices per store in JSON format. As in the original code, the Spring Boot application has 3 important configuration tweaks. In addition we immediately removed the observability features from the Spring sample, which adds a few percent to the throughput over the original. The original benchmarks only covered one data set with 10 fruits; this analysis extends that to multiple data sets by increasing numbers of fruits to observe performance trends. The measurements are done with the Spring benchmarks harness, which is similar to the original one, and the baseline results for 10 fruits are similar to the original. The main difference in the harness is that the Spring one has a longer warm up time for the running application - it gives the JVM time to optimize itself. The numbers reported here were collected in a controlled environment on a Linux virtual machine with 16 vCPUs and 32 GB RAM, running Java 21. The application, database and the load generator were all running on the same VM to minimize network variability, but in separate containers, constrained to have a fixed amount of CPUs and memory (4 CPUs and 2GB for the application container unless mentioned otherwise).

Results

Here are the raw data for throughput (TPS) vs number of fruits (N). The larger data sets were generated simply by copying the original 10-fruit data set multiple times. The number of stores is constant and the number of prices per fruit is the same in each case, so the result set size scales linearly with the number of fruits.

Now we do some anaysis and plot the data along with a linear regression of the "cycle time" (inverse of throughput) vs number of fruits.

Optimizations

Profiling the Spring sample revealed that the transaction management was adding a significant overhead to the processing of each HTTP request, even though the operations were read-only. By annotating the repository methods with @Transactional(propagation = SUPPORTS, readOnly = true), we can avoid unnecessary transaction management overhead for read-only operations. This change significantly improved the performance of the sample, especially for smaller data sets, where the baseline overhead is more pronounced. (Note that this configuration is suited to read-only operations, and not appropriate in general for all applications and endpoints.) The results are shown below, demonstrating a noticeable improvement in throughput across all data set sizes.

There are some optimizations in the Spring Framework, adapting the HTTP headers to be more efficient, taking advantage of the underlying container implementation. This optimization is included in the results shown below (labelled "sha"), and it adds a few percent to the throughput across all data set sizes. You can pick up this change in Spring 7.0.5. Additional gains are also included from other optimizations in the Spring Framework, to be released shortly in version 7.0.6 (labelled "buf").

We can get some even better results by switching on virtual threads, which is trivial in a Spring Boot application.

Analysis and Interpretation

Here's a reasonable interpretation: time spent per fruit is roughly constant, and there is a fixed overhead per framework. So (cycle time) = 1 / throughput = A + B*N (A and B constant). The trendlines show a good fit to that for all data sets.

In a more realistic application where the cycle time per fruit is higher, the throughput will be dominated by real business processing even for small data sets. The baseline framework overhead would be less significant in that case, and since it is only a few tens of microseconds, any processing of a data item that took 100ms or a few times that would quickly overwhelm it even for small data sets.