Abstract
Model-based performance prediction methods aim at evaluating the expected response time, throughput, and resource utilization of a software system at design time, before implementation, to achieve predictability of the system’s performance characteristics. Existing performance prediction methods use monolithic, throw-away prediction models or component-based, reusable prediction models. While it is intuitively clear that the development of reusable models requires more effort, the actual higher amount of effort had not been quantified or analyzed systematically yet. Furthermore, the achieved prediction accuracy of the methods when applied by developers had not yet been compared. To study this effort, we conducted a controlled experiment with 19 computer science students who predicted the performance of two example systems applying an established, monolithic method (Software Performance Engineering) as well as our own component-based method (Palladio) in 2007. This paper summarizes two earlier papers on this study. The results show that the effort of model creation with Palladio is approximately 1.25 times higher than with SPE in our experimental setting, with the resulting models having comparable prediction accuracy. Therefore, in some cases, the creation of reusable prediction models can already be justified, provided they are reused at least once.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
Note that the percentages for the two systems do not equally influence the results, but are weighted by the number of decisions by definition of the metric (cf. [12, p. 41]).
References
Balsamo, S., Di Marco, A., Inverardi, P., Simeoni, M.: Model-based performance prediction in software development: a survey. IEEE TSE 30(5), 295–310 (2004)
Balsamo S., Marzolla, M.: A simulation-based approach to software performance modeling. In: Proceedings of ESEC/FSE, Helsinki, pp. 363–366. ACM (2003)
Balsamo, S., Marzolla, M., Di Marco, A., Inverardi, P.: Experimenting different software architectures performance techniques. In: Proceedings of WOSP, Redwood Shores, pp. 115–119. ACM (2004)
Basili, V.R., Caldiera, G., Rombach, H.D.: The goal question metric approach. In: Marciniak, J.J. (ed.) Encyclopedia of Software Engineering – 2 Volume Set, pp. 528–532. Wiley, Chichester (1994)
Becker, S., Grunske, L., Mirandola, R., Overhage, S.: Performance prediction of component-based systems: a survey from an engineering perspective. In: Architecting Systems with Trustworthy Components. Volume 3938 of LNCS, pp. 169–192. Springer, Berlin/New York (2006)
Becker, S., Koziolek, H., Reussner, R.: Model-based performance prediction with the Palladio component model. In: Proceedings of WOSP, Buenos Aires, pp. 54–65. ACM Sigsoft, 5–8 Feb 2007
Bertolino A., Mirandola, R.:CB-SPE tool: putting component-based performance engineering into practice. In: Proceedings of CBSE, Edinburgh. Volume 3054 of LNCS, pp. 233–248. Springer (2004)
Bondarev, E., Muskens, J., de With, P.H.N., Chaudron, M.R.V., Lukkien, J.: Predicting real-time properties of component assemblies: a scenario-simulation approach. In: Proceedings of 30th EUROMICRO-Conference, Rennes, pp. 40–47 (2004)
Gorton I., Liu, A.: Performance evaluation of alternative component architectures for enterprise JavaBean applications. IEEE Internet Comput. 7(3), 18–23. Rennes, France (2003)
Höst, M., Regnell, B., Wohlin, C.: Using students as subjects – a comparative study of students and professionals in lead-time impact assessment. Empir. Softw. Eng. 5(3), 201–214 (2000)
Koziolek, H., Firus, V.: Empirical evaluation of model-based performance predictions methods in software development. In: Proceedings of QoSA, Erfurt. Volume 3712 of LNCS, pp 188–202, Sept 2005
Martens, A.: Empirical validation of the model-driven performance prediction approach Palladio. Master’s thesis, Carl-von-Ossietzky Universität Oldenburg (2007)
Martens, A., Becker, S., Koziolek, H., Reussner, R.: An empirical investigation of the applicability of a component-based performance prediction method. In: Thomas N., Juiz, C. (eds.) Proceedings of the 5th European Performance Engineering Workshop (EPEW’08), Palma de Mallorca. Volume 5261 of Lecture Notes in Computer Science, pp. 17–31. Springer, Berlin/Heidelberg (2008)
Martens, A., Becker, S., Koziolek, H., Reussner, R.: An empirical investigation of the effort of creating reusable models for performance prediction. In: Proceedings of the 11th International Symposium on Component-Based Software Engineering (CBSE’08), Karlsruhe. Volume 5282 of Lecture Notes in Computer Science, pp. 16–31. Springer, Berlin/Heidelberg (2008)
Martens, A., Koziolek, H., Prechelt, L., Reussner, R.: From monolithic to component-based performance evaluation of software architectures. Empir. Soft. Eng. 16(5), 587–622 (2011)
Menascé, D.A., Almeida, V.A.F., Dowdy, L.W.: Performance by Design. Prentice Hall, Upper Saddle River (2004)
Petriu, D.C. Wang, X.: From UML description of high-level software architecture to LQN performance models. In: Nagl, M., Schürr, A., Münch, M. (eds.) Proceedings of AGTIVE’99, Kerkrade, vol. 1779. Springer, Berlin/New York (2000)
Sachs, L.: Applied Statistics: A Handbook of Techniques. Springer, New York (1982)
Smith, C.U.: Performance Engineering of Software Systems. Addison-Wesley, Reading (1990)
Smith, C.U., Williams, L.G.: Performance Solutions: A Practical Guide to Creating Responsive, Scalable Software. Addison-Wesley, Boston (2002)
Szyperski, C.: Component Software: Beyond Object-Oriented Programming. ACM,New York/Addison-Wesley, Reading (1998)
Welch, B.L.: The generalization of student’s problem when several different population variances are involved. Biometrika 34, 28–35 (1947)
Wohlin, C., Runeson, P., Höst, M., Ohlsson, M.C., Regnell, B., Wesslén, A.: Experimentation in Software Engineering: An Introduction. Kluwer Academic, Norwell (2000)
Wu X., Woodside, M.: Performance Modeling from Software Components. SIGSOFT SE Notes 29(1), 290–301 (2004)
Acknowledgements
We would like to thank Walter Tichy, Lutz Prechelt, and Wilhelm Hasselbring for their kind review of the experimental design and fruitful comments. Furthermore, we thank all members of the SDQ Chair for helping to prepare and conduct the experiment. Last, but not least, we thank all students who volunteered to participate in our experiment.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this chapter
Cite this chapter
Reussner, R., Becker, S., Koziolek, A., Koziolek, H. (2013). An Empirical Investigation of the Component-Based Performance Prediction Method Palladio. In: Münch, J., Schmid, K. (eds) Perspectives on the Future of Software Engineering. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-37395-4_13
Download citation
DOI: https://doi.org/10.1007/978-3-642-37395-4_13
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-37394-7
Online ISBN: 978-3-642-37395-4
eBook Packages: Computer ScienceComputer Science (R0)