An Empirical Investigation of the Component-Based Performance Prediction Method Palladio

  • Ralf ReussnerEmail author
  • Steffen Becker
  • Anne Koziolek
  • Heiko Koziolek


Model-based performance prediction methods aim at evaluating the expected response time, throughput, and resource utilization of a software system at design time, before implementation, to achieve predictability of the system’s performance characteristics. Existing performance prediction methods use monolithic, throw-away prediction models or component-based, reusable prediction models. While it is intuitively clear that the development of reusable models requires more effort, the actual higher amount of effort had not been quantified or analyzed systematically yet. Furthermore, the achieved prediction accuracy of the methods when applied by developers had not yet been compared. To study this effort, we conducted a controlled experiment with 19 computer science students who predicted the performance of two example systems applying an established, monolithic method (Software Performance Engineering) as well as our own component-based method (Palladio) in 2007. This paper summarizes two earlier papers on this study. The results show that the effort of model creation with Palladio is approximately 1.25 times higher than with SPE in our experimental setting, with the resulting models having comparable prediction accuracy. Therefore, in some cases, the creation of reusable prediction models can already be justified, provided they are reused at least once.



We would like to thank Walter Tichy, Lutz Prechelt, and Wilhelm Hasselbring for their kind review of the experimental design and fruitful comments. Furthermore, we thank all members of the SDQ Chair for helping to prepare and conduct the experiment. Last, but not least, we thank all students who volunteered to participate in our experiment.


  1. 1.
    Balsamo, S., Di Marco, A., Inverardi, P., Simeoni, M.: Model-based performance prediction in software development: a survey. IEEE TSE 30(5), 295–310 (2004)Google Scholar
  2. 2.
    Balsamo S., Marzolla, M.: A simulation-based approach to software performance modeling. In: Proceedings of ESEC/FSE, Helsinki, pp. 363–366. ACM (2003)Google Scholar
  3. 3.
    Balsamo, S., Marzolla, M., Di Marco, A., Inverardi, P.: Experimenting different software architectures performance techniques. In: Proceedings of WOSP, Redwood Shores, pp. 115–119. ACM (2004)Google Scholar
  4. 4.
    Basili, V.R., Caldiera, G., Rombach, H.D.: The goal question metric approach. In: Marciniak, J.J. (ed.) Encyclopedia of Software Engineering – 2 Volume Set, pp. 528–532. Wiley, Chichester (1994)Google Scholar
  5. 5.
    Becker, S., Grunske, L., Mirandola, R., Overhage, S.: Performance prediction of component-based systems: a survey from an engineering perspective. In: Architecting Systems with Trustworthy Components. Volume 3938 of LNCS, pp. 169–192. Springer, Berlin/New York (2006)Google Scholar
  6. 6.
    Becker, S., Koziolek, H., Reussner, R.: Model-based performance prediction with the Palladio component model. In: Proceedings of WOSP, Buenos Aires, pp. 54–65. ACM Sigsoft, 5–8 Feb 2007Google Scholar
  7. 7.
    Bertolino A., Mirandola, R.:CB-SPE tool: putting component-based performance engineering into practice. In: Proceedings of CBSE, Edinburgh. Volume 3054 of LNCS, pp. 233–248. Springer (2004)Google Scholar
  8. 8.
    Bondarev, E., Muskens, J., de With, P.H.N., Chaudron, M.R.V., Lukkien, J.: Predicting real-time properties of component assemblies: a scenario-simulation approach. In: Proceedings of 30th EUROMICRO-Conference, Rennes, pp. 40–47 (2004)Google Scholar
  9. 9.
    Gorton I., Liu, A.: Performance evaluation of alternative component architectures for enterprise JavaBean applications. IEEE Internet Comput. 7(3), 18–23. Rennes, France (2003)Google Scholar
  10. 10.
    Höst, M., Regnell, B., Wohlin, C.: Using students as subjects – a comparative study of students and professionals in lead-time impact assessment. Empir. Softw. Eng. 5(3), 201–214 (2000)zbMATHCrossRefGoogle Scholar
  11. 11.
    Koziolek, H., Firus, V.: Empirical evaluation of model-based performance predictions methods in software development. In: Proceedings of QoSA, Erfurt. Volume 3712 of LNCS, pp 188–202, Sept 2005Google Scholar
  12. 12.
    Martens, A.: Empirical validation of the model-driven performance prediction approach Palladio. Master’s thesis, Carl-von-Ossietzky Universität Oldenburg (2007)Google Scholar
  13. 13.
    Martens, A., Becker, S., Koziolek, H., Reussner, R.: An empirical investigation of the applicability of a component-based performance prediction method. In: Thomas N., Juiz, C. (eds.) Proceedings of the 5th European Performance Engineering Workshop (EPEW’08), Palma de Mallorca. Volume 5261 of Lecture Notes in Computer Science, pp. 17–31. Springer, Berlin/Heidelberg (2008)Google Scholar
  14. 14.
    Martens, A., Becker, S., Koziolek, H., Reussner, R.: An empirical investigation of the effort of creating reusable models for performance prediction. In: Proceedings of the 11th International Symposium on Component-Based Software Engineering (CBSE’08), Karlsruhe. Volume 5282 of Lecture Notes in Computer Science, pp. 16–31. Springer, Berlin/Heidelberg (2008)Google Scholar
  15. 15.
    Martens, A., Koziolek, H., Prechelt, L., Reussner, R.: From monolithic to component-based performance evaluation of software architectures. Empir. Soft. Eng. 16(5), 587–622 (2011)CrossRefGoogle Scholar
  16. 16.
    Menascé, D.A., Almeida, V.A.F., Dowdy, L.W.: Performance by Design. Prentice Hall, Upper Saddle River (2004)Google Scholar
  17. 17.
    Petriu, D.C. Wang, X.: From UML description of high-level software architecture to LQN performance models. In: Nagl, M., Schürr, A., Münch, M. (eds.) Proceedings of AGTIVE’99, Kerkrade, vol. 1779. Springer, Berlin/New York (2000)Google Scholar
  18. 18.
    Sachs, L.: Applied Statistics: A Handbook of Techniques. Springer, New York (1982)zbMATHGoogle Scholar
  19. 19.
    Smith, C.U.: Performance Engineering of Software Systems. Addison-Wesley, Reading (1990)Google Scholar
  20. 20.
    Smith, C.U., Williams, L.G.: Performance Solutions: A Practical Guide to Creating Responsive, Scalable Software. Addison-Wesley, Boston (2002)Google Scholar
  21. 21.
    Szyperski, C.: Component Software: Beyond Object-Oriented Programming. ACM,New York/Addison-Wesley, Reading (1998)Google Scholar
  22. 22.
    Welch, B.L.: The generalization of student’s problem when several different population variances are involved. Biometrika 34, 28–35 (1947)MathSciNetzbMATHGoogle Scholar
  23. 23.
    Wohlin, C., Runeson, P., Höst, M., Ohlsson, M.C., Regnell, B., Wesslén, A.: Experimentation in Software Engineering: An Introduction. Kluwer Academic, Norwell (2000)CrossRefGoogle Scholar
  24. 24.
    Wu X., Woodside, M.: Performance Modeling from Software Components. SIGSOFT SE Notes 29(1), 290–301 (2004)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Ralf Reussner
    • 1
    Email author
  • Steffen Becker
    • 2
  • Anne Koziolek
    • 1
  • Heiko Koziolek
    • 3
  1. 1.Karlsruher Institut für Technologie (KIT)Institut für Programmstrukturen und Datenorganisation (IPD)KarlsruheGermany
  2. 2.Fachgruppe Softwaretechnik, Heinz Nixdorf InstitutUniversität PaderbornPaderbornGermany
  3. 3.ABB Corporate ResearchLadenburgGermany

Personalised recommendations