Comparing the Accuracy of Resource Demand Measurement and Estimation Techniques

  • Felix WillneckerEmail author
  • Markus Dlugi
  • Andreas Brunnert
  • Simon Spinner
  • Samuel Kounev
  • Wolfgang Gottesheim
  • Helmut Krcmar
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9272)


Resource demands are a core aspect of performance models. They describe how an operation utilizes a resource and therefore influence the systems performance metrics: response time, resource utilization and throughput. Such demands can be determined by two extraction classes: direct measurement or demand estimation. Selecting the best suited technique depends on available tools, acceptable measurement overhead and the level of granularity necessary for the performance model. This work compares two direct measurement techniques and an adaptive estimation technique based on multiple statistical approaches to evaluate strengths and weaknesses of each technique. We conduct a series of experiments using the SPECjEnterprise2010 industry benchmark and an automatic performance model generator for architecture-level performance models based on the Palladio Component Model. To compare the techniques we conduct two experiments with different levels of granularity on a standalone system, followed by one experiment using a distributed SPECjEnterprise2010 deployment combining both extraction classes for generating a full-stack performance model.


Performance model generation Resource demand measurements Resource demand estimations 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Becker, S., Koziolek, H., Reussner, R.: The Palladio Component Model for Model-Driven Performance Prediction. Journal of Systems and Software 82(1), 3–22 (2009). special Issue: Software Performance - Modeling and AnalysisCrossRefGoogle Scholar
  2. 2.
    Brosig, F., Gorsler, F., Huber, N., Kounev, S.: Evaluating approaches for performance prediction in virtualized environments. In: 2013 IEEE 21st International Symposium on Modeling, Analysis Simulation of Computer and Telecommunication Systems (MASCOTS), pp. 404–408, August 2013Google Scholar
  3. 3.
    Brosig, F., Kounev, S., Krogmann, K.: Automated extraction of palladio component models from running enterprise java applications. In: Proceedings of the 1st International Workshop on Run-time Models for Self-managing Systems and Applications (ROSSA 2009). ACM, New York (2009)Google Scholar
  4. 4.
    Brunnert, A., Krcmar, H.: Continuous Performance Evaluation and Capacity Planning Using Resource Profiles (under review). Journal of Systems and Software (2015)Google Scholar
  5. 5.
    Brunnert, A., Neubig, S., Krcmar, H.: Evaluating the Prediction Accuracy of Generated Performance Models in Up- and Downscaling Scenarios, pp. 113–130 (2014)Google Scholar
  6. 6.
    Brunnert, A., Vögele, C., Krcmar, H.: Automatic performance model generation for java enterprise edition (EE) applications. In: Balsamo, M.S., Knottenbelt, W.J., Marin, A. (eds.) EPEW 2013. LNCS, vol. 8168, pp. 74–88. Springer, Heidelberg (2013) CrossRefGoogle Scholar
  7. 7.
    Dynatrace: Dynatrace Agent Timers. (accessed: May 14, 2015)
  8. 8.
    Harchol-Balter, M.: Performance Modeling and Design of Computer Systems. Cambridge University Press, New York (2013) zbMATHGoogle Scholar
  9. 9.
    Hofer, P., Hörschläger, F., Mössenböck, H.: Sampling-based steal time accounting under hardware virtualization. In: Proceedings of the 6th ACM/SPEC International Conference on Performance Engineering (ICPE 2015), pp. 87–90. ACM, New York (2015)Google Scholar
  10. 10.
    Huber, N., von Quast, M., Hauck, M., Kounev, S.: Evaluating and modeling virtualization performance overhead for cloud environments. In: Proceedings of the 1st International Conference on Cloud Computing and Services Science (CLOSER 2011), pp. 563–573. SciTePress, May 2011Google Scholar
  11. 11.
    Kuperberg, M.: Quantifying and Predicting the Influence of Execution Platform on Software Component Performance, vol. 5. KIT Scientific Publishing (2010)Google Scholar
  12. 12.
    Menascé, D.A.: Computing missing service demand parameters for performance models. In: Proceedings of the 2008 Computer Measurement Group Conference (CMG 2008), Las Vegas, NV, USA, pp. 241–248 (2008)Google Scholar
  13. 13.
    Rolia, J., Vetland, V.: Parameter estimation for performance models of distributed application systems. In: Proceedings of the 1995 Conference of the Centre for Advanced Studies on Collaborative Research (CASCON 1995), p. 54. IBM (1995)Google Scholar
  14. 14.
    Spinner, S.: Evaluating Approaches to Resource Demand estimation. Master’s thesis, Karlsruhe Institute of Technology (KIT) (2011)Google Scholar
  15. 15.
    Spinner, S., Casale, G., Zhu, X., Kounev, S.: LibReDE: A library for resource demand estimation. In: Proceedings of the 5th ACM/SPEC International Conference on Performance Engineering (ICPE 2014), pp. 227–228. ACM, New York (2014)Google Scholar
  16. 16.
    Wang, W., Huang, X., Qin, X., Zhang, W., Wei, J., Zhong, H.: Application-level CPU consumption estimation: towards performance isolation of multi-tenancy web applications. In: Proceedings of the 5th International Conference on Cloud Computing (CLOUD), pp. 439–446. IEEE (2012)Google Scholar
  17. 17.
    Willnecker, F., Brunnert, A., Gottesheim, W., Krcmar, H.: Using dynatrace monitoring data for generating performance models of java EE applications. In: Proceedings of the 6th ACM/SPEC International Conference on Performance Engineering (ICPE 2015), pp. 103–104. ACM, New York (2015)Google Scholar
  18. 18.
    Zheng, T., Woodside, M., Litoiu, M.: Performance Model Estimation and Tracking Using Optimal Filters. IEEE Transactions on Software Engineering 34(3), 391–406 (2008)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Felix Willnecker
    • 1
    Email author
  • Markus Dlugi
    • 1
  • Andreas Brunnert
    • 1
  • Simon Spinner
    • 2
  • Samuel Kounev
    • 2
  • Wolfgang Gottesheim
    • 3
  • Helmut Krcmar
    • 4
  1. 1.Fortiss GmbHMünchenGermany
  2. 2.Universität WürzburgWürzburgGermany
  3. 3.Dynatrace Austria GmbHLinzAustria
  4. 4.Technische Universität MünchenGarchingGermany

Personalised recommendations