A comparative study of application-level caching recommendations at the method level

Abstract

Performance and scalability requirements have a fundamental role in most large-scale software applications. To satisfy such requirements, caching is often used at various levels and infrastructure layers. Application-level caching—or memoization—is an increasingly used form of caching within the application boundaries, which consists of storing the results of computations in memory to avoid re-computing them. This is typically manually done by developers, who identify caching opportunities in the code and write additional code to manage the cache content. The task of identifying caching opportunities is a challenge because it requires the analysis of workloads and code locations where it is feasible and beneficial to cache objects. To aid developers in this task, there are approaches that automatically identify cacheable methods. Although such approaches have been individually evaluated, their effectiveness has not been compared. We thus in this paper present an empirical evaluation to compare the method recommendations made by the two existing application-level caching approaches at the method level, namely APLCache and MemoizeIt, using seven open-source web applications. We analyse the recommendations made by each approach as well as the hits, misses and throughput achieved with their valid caching recommendations. Our results show that the effectiveness of both approaches largely depends on the specific application, the presence of invalid recommendations and additional configurations, such as the time-to-live. By inspecting the obtained results, we observed in which cases the recommendations of each approach fail and succeed, which allowed us to derive a set of seven lessons learned that give directions for future approaches to support developers in the adoption of this type of caching.

Appendix: Reproducibility

To reproduce and/or change the parameters of our experiment, it is required to have two computers with sufficient resources, both running Linux-based systems and having Java 11+, Maven, Docker CE and docker-compose installed. In our repository, we provide a configure file that helps to install such dependencies. Also, this configuration script automates the cloning of all the repositories of each application version in the proper folder, as well as the tools that we implemented and rely on. It is also possible to run our experiment in standalone mode (with only one computer). However, it may not generate reliable results, due to the interference of the requester application that would compete on resources with the running application. Finally, compiling and installing our tools through Maven is also required, which is automatically made by a script named compile.sh.

To the first phase of the experiment, we automatically detect all the applications (including new ones) within the folder that holds the NOCACHE version of applications through the script trace.sh. The hostname for the server machine must be informed. We then check whether the traces of the application were already collected in the outputs folder. If not yet executed, we send a signal to the server machine (which could also be the same machine) to bring the application up via a Docker container. It is expected to the server machine to be listening for requests with our Java tool called RemoteExecutor, as well as having all the mentioned configurations done. New applications are expected to have a Dockerfile—as we did for our target applications—including its required steps to compile and to run, as well as a docker-compose describing its dependencies, such as a database and its initialisation SQL. We also look for the files whitelist, blacklist and ignored in order to configure which Java packages should be serialised by our JSONSerialiser tool for the tracings. Also, including the dependency of our ApplicationTracer in the pom or gradle file is required, so we can automatically inject code in every method of new applications. We then automatically wait for the application to start and once started we start firing requests. For new applications, describing the JSON file containing the workload graph as we did for our applications in the workloads folder is required for this step. When finished with the tracing, we automatically tear the application down and clean its created files and data.

To the recommendation of methods based on the collected traces, it is just needed to execute the approaches using as input the trace file generated in the first phase. Manual analysis of the recommendations, as well as the manual implementation of the caching with our Cache component, is required. Particularly for APL, the generated files containing the recommended inputs (which is already included in our repository) are required to be in the outputs folder. It is also important, for APL, that the same serialisation parameters adopted in the first phase are used in the second phase, so the inputs can match. For new applications, the Maven/Gradle dependency of our Cache component must be added. For new approaches, it is required that they are able to read and process the tracing file generated in the first phase so they can give their recommendations.

To the second phase of the experiment, we automatically detect all the applications and start firing the requests through our script named run.sh.The hostname for the server machine must be informed. If not yet generated the workloads, we automatically do it. We then start sampling all the ten executions for all the groups of simulated users collecting our metrics in the disk. Particular executions that already succeeded are skipped. It is expected in this phase that each application version is in the proper folder and includes the files, configurations and caching as mentioned before.

To aggregate the results produced in the outputs folder, the script reduce.sh helps on initialising the CSV files and executing our tools that calculate our metrics. The hostname for the server machine must be informed. In order to generate visualisations for the aggregated CSV files, the script plot.sh under the analysis folder may be invoked.