Skip to main content
SpringerLink
Log in

Controlling Automatic Experiment-Driven Systems Using Statistics and Machine Learning

  • Conference paper
  • First Online:
Software Architecture. ECSA 2022 Tracks and Workshops (ECSA 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13928))

Included in the following conference series:

  • 305 Accesses

Abstract

Experiments are used in many modern systems to optimize their operation. Such experiment-driven systems are used in various fields, such as web-based systems, smart-* systems, and various self-adaptive systems. There is a class of these systems that derive their data from running simulations or another type of computation, such as in digital twins, online planning using probabilistic model-checking, or performance benchmarking. To obtain statistically significant results, these systems must repeat the experiments multiple times. As a result, they consume extensive computation resources. The GraalVM benchmarking project detects performance changes in the GraalVM compiler. However, the benchmarking project has an extensive usage of computational resources and time. The doctoral research project proposed in this paper focuses on controlling the experiments with the goal of reducing computation costs. The plan is to use statistical and machine learning approaches to predict the outcomes of experiments and select the experiments yielding more useful information. As an evaluation, we are applying these methods to the GraalVM benchmarking project; the initial results confirm that these methods have the potential to significantly reduce computation costs.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    www.mongodb.com.

  2. 2.

    www.graalvm.org.

  3. 3.

    https://graal.d3s.mff.cuni.cz.

References

  1. Abdullah, M., Bulej, L., Bures, T., Hnetynka, P., Horky, V., Tuma, P.: Reducing experiment costs in automated software performance regression detection. In: 2022 48th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), Gran Canaria, Spain, pp. 56–59. IEEE (2022). https://doi.org/10.1109/SEAA56994.2022.00017

  2. Ali, S., Hafeez, Y., Hussain, S., Yang, S.: Enhanced regression testing technique for agile software development and continuous integration strategies. Software Qual. J. 28(2), 397–423 (2019). https://doi.org/10.1007/s11219-019-09463-4

    Article  Google Scholar 

  3. ALShoaibi, D., Gupta, H., Mendelson, M., Jenhani, I., Mrad, A.B., Mkaouer, M.W.: Learning to characterize performance regression introducing code changes. In: Proceedings of the 37th ACM/SIGAPP Symposium on Applied Computing, Virtual Event, pp. 1590–1597. ACM (2022). https://doi.org/10.1145/3477314.3507150

  4. Altiero, F., Colella, G., Corazza, A., Di Martino, S., Peron, A., Starace, L.L.L.: Change-aware regression test prioritization using genetic algorithms. In: 2022 48th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), Gran Canaria, Spain, pp. 125–132. IEEE (2022). https://doi.org/10.1109/SEAA56994.2022.00028

  5. Barrett, E., Bolz-Tereick, C.F., Killick, R., Mount, S., Tratt, L.: Virtual machine warmup blows hot and cold. Proc. ACM Program. Lang. 1(OOPSLA), 1–27 (2017). https://doi.org/10.1145/3133876

  6. Bukh, P.N.D.: Review of the art of computer systems performance analysis, techniques for experimental design, measurement, simulation and modeling. Interfaces 22(4), 113–115 (1992)

    Google Scholar 

  7. Bulej, L., et al.: Unit testing performance with stochastic performance logic. Autom. Softw. Eng. 24(1), 139–187 (2016). https://doi.org/10.1007/s10515-015-0188-0

    Article  Google Scholar 

  8. Bulej, L., Horký, V., Tůma, P.: Tracking Performance of Graal on Public Benchmarks, p. 7258108 Bytes (2021). https://doi.org/10.6084/M9.FIGSHARE.14447148. Artwork Size: 7258108 Bytes Publisher: figshare

  9. Costa, D., Bezemer, C.P., Leitner, P., Andrzejak, A.: What’s wrong with my benchmark results? Studying bad practices in JMH benchmarks. IEEE Trans. Softw. Eng. 47(7), 1452–1467 (2021). https://doi.org/10.1109/TSE.2019.2925345

  10. De Oliveira, A.B., Fischmeister, S., Diwan, A., Hauswirth, M., Sweeney, P.F.: Perphecy: performance regression test selection made simple but effective. In: 2017 IEEE International Conference on Software Testing, Verification and Validation (ICST), Tokyo, pp. 103–113. IEEE (2017). https://doi.org/10.1109/ICST.2017.17

  11. Deng, A., Lu, J., Chen, S.: Continuous monitoring of A/B tests without pain: optional stopping in Bayesian testing. In: 2016 IEEE International Conference on Data Science and Advanced Analytics (DSAA), Montreal, QC, Canada, pp. 243–252. IEEE (2016). https://doi.org/10.1109/DSAA.2016.33

  12. Eimer, T., Biedenkapp, A., Reimer, M., Adriansen, S., Hutter, F., Lindauer, M.: DACBench: a benchmark library for dynamic algorithm configuration. In: Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, pp. 1668–1674. International Joint Conferences on Artificial Intelligence Organization, Montreal, Canada (2021). https://doi.org/10.24963/ijcai.2021/230

  13. Georges, A., Eeckhout, L., Buytaert, D.: Java performance evaluation through rigorous replay compilation. In: Proceedings of the 23rd ACM SIGPLAN Conference on Object-Oriented Programming Systems Languages and Applications, Nashville, TN, USA, pp. 367–384. ACM (2008). https://doi.org/10.1145/1449764.1449794

  14. Gerostathopoulos, I., Plasil, F., Prehofer, C., Thomas, J., Bischl, B.: Automated online experiment-driven adaptation-mechanics and cost aspects. IEEE Access 9, 58079–58087 (2021). https://doi.org/10.1109/ACCESS.2021.3071809

    Article  Google Scholar 

  15. Grano, G., Laaber, C., Panichella, A., Panichella, S.: Testing with fewer resources: an adaptive approach to performance-aware test case generation. IEEE Trans. Software Eng. 47(11), 2332–2347 (2021). https://doi.org/10.1109/TSE.2019.2946773

    Article  Google Scholar 

  16. Gupta, S., et al.: Top challenges from the first practical online controlled experiments summit. ACM SIGKDD Explor. Newsl. 21(1), 20–35 (2019). https://doi.org/10.1145/3331651.3331655

    Article  Google Scholar 

  17. Habib, M.K., Ayankoso, S.A., Nagata, F.: Data-driven modeling: concept, techniques, challenges and a case study. In: 2021 IEEE International Conference on Mechatronics and Automation (ICMA), Takamatsu, Japan, pp. 1000–1007. IEEE (2021). https://doi.org/10.1109/ICMA52036.2021.9512658

  18. Johari, R., Koomen, P., Pekelis, L., Walsh, D.: Always valid inference: continuous monitoring of A/B tests. Oper. Res. 70(3), 1806–1821 (2022). https://doi.org/10.1287/opre.2021.2135

    Article  MathSciNet  MATH  Google Scholar 

  19. Laaber, C., Basmaci, M., Salza, P.: Predicting unstable software benchmarks using static source code features. Empir. Softw. Eng. 26(6), 1–53 (2021). https://doi.org/10.1007/s10664-021-09996-y

    Article  Google Scholar 

  20. Moghadam, M.H., Saadatmand, M., Borg, M., Bohlin, M., Lisper, B.: An autonomous performance testing framework using self-adaptive fuzzy reinforcement learning. Softw. Qual. J. (6), 1–33 (2021). https://doi.org/10.1007/s11219-020-09532-z

  21. Pecorelli, F., Grano, G., Palomba, F., Gall, H.C., De Lucia, A.: Toward Granular Automatic Unit Test Case Generation (2022). https://doi.org/10.48550/ARXIV.2204.05561

  22. Quin, F., Weyns, D.: SEAByTE: a self-adaptive micro-service system artifact for automating A/B testing. In: Proceedings of the 17th Symposium on Software Engineering for Adaptive and Self-Managing Systems, Pittsburgh, Pennsylvania, pp. 77–83. ACM (2022). https://doi.org/10.1145/3524844.3528081

  23. Reichelt, D.G., Kuhne, S., Hasselbring, W.: PeASS: a tool for identifying performance changes at code level. In: 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE), San Diego, CA, USA, pp. 1146–1149. IEEE (2019). https://doi.org/10.1109/ASE.2019.00123

  24. Rothermel, G., Untch, R., Chu, C., Harrold, M.: Prioritizing test cases for regression testing. IEEE Trans. Software Eng. 27(10), 929–948 (2001). https://doi.org/10.1109/32.962562

  25. Saputri, T.R.D., Lee, S.W.: The application of machine learning in self-adaptive systems: a systematic literature review. IEEE Access 8, 205948–205967 (2020). https://doi.org/10.1109/ACCESS.2020.3036037

    Article  Google Scholar 

  26. Sheskin, D.J.: Handbook of Parametric and Nonparametric Statistical Procedures, 5 edn. Chapman and Hall/CRC, Boca Raton (2011). https://doi.org/10.1201/9780429186196

  27. Smith, C.U.: Software performance antipatterns in cyber-physical systems. In: Proceedings of the ACM/SPEC International Conference on Performance Engineering, Edmonton, AB, Canada, pp. 173–180. ACM (2020). https://doi.org/10.1145/3358960.3379138

  28. Traini, L., Cortellessa, V., Di Pompeo, D., Tucci, M.: Towards effective assessment of steady state performance in Java software: are we there yet? Empir. Softw. Eng. 28(1), 13 (2023). https://doi.org/10.1007/s10664-022-10247-x

    Article  Google Scholar 

  29. Vemulapati, J., Khastgir, A.S., Savalgi, C.: AI based performance benchmarking & analysis of big data and cloud powered applications: an in depth view. In: Proceedings of the 2019 ACM/SPEC International Conference on Performance Engineering, Mumbai, India, pp. 103–109. ACM (2019). https://doi.org/10.1145/3297663.3309676

Download references

Acknowledgment

This work has been partially supported by the EU project ExtremeXP grant agreement 101093164, partially by Charles University institutional funding SVV 260698/2023, and partially by the Charles University Grant Agency project 408622.

Author information

Authors and Affiliations

  1. Charles University, Prague, Czech Republic

    Milad Abdullah

Authors
  1. Milad Abdullah
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Milad Abdullah .

Editor information

Editors and Affiliations

  1. Federal University of Rio Grande do Norte, Natal, Brazil

    Thais Batista

  2. Charles University, Prague, Czech Republic

    Tomáš Bureš

  3. Vrije Universiteit Amsterdam, Amsterdam, The Netherlands

    Claudia Raibulet

  4. University of L’Aquila, L’Aquila, Italy

    Henry Muccini

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Abdullah, M. (2023). Controlling Automatic Experiment-Driven Systems Using Statistics and Machine Learning. In: Batista, T., Bureš, T., Raibulet, C., Muccini, H. (eds) Software Architecture. ECSA 2022 Tracks and Workshops. ECSA 2022. Lecture Notes in Computer Science, vol 13928. Springer, Cham. https://doi.org/10.1007/978-3-031-36889-9_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-36889-9_9

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-36888-2

  • Online ISBN: 978-3-031-36889-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation