A Calibration Framework for Capturing and Calibrating Software Performance Models

  • Xiuping Wu
  • Murray Woodside
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5261)

Abstract

Software performance engineering could benefit from combining modeling and testing techniques, if performance models could be derived more cheaply and more easily. This work investigates how known testing and estimation methodologies can be combined in a calibration framework, to provide and maintain performance models in sync with a developing product or component library. There are two main aspects. The first addresses a major barrier in practice, the calibration of model parameters that represent quantities that cannot easily be measured directly. This work calibrates these “hidden parameters” efficiently using a Kalman Filter. The second is the exploitation of the filter estimator to control the calibration framework, for example to terminate a test when accuracy is sufficient, and to design tests for parameter coverage. The technique is demonstrated on simulated data and on an implemented Voice-over-IP (VoIP) system.

Keywords

Performance modeling Performance testing Parameter estimation Software performance engineering 
This is a preview of subscription content, log in to check access.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Avritzer, A., Weyuker, E.J.: The Role of Modeling in the Performance Testing of E-Commerce Applications. IEEE Trans. Software Eng. 30(12) (December 2004)Google Scholar
  2. 2.
    Avritzer, A., Kondek, J., Liu, D., Weyuker, E.J.: Software performance testing based on workload characterization. In: Proc. 3rd Int. Workshop on Software and Performance (WOSP 2002), Rome, pp. 17–24 (2002)Google Scholar
  3. 3.
    Balsamo, S., DiMarco, A., Inverardi, P., Simeoni, M.: Model-based Performance Prediction in Software Development. IEEE Trans. Software Eng. 30, 295–310 (2004)CrossRefGoogle Scholar
  4. 4.
    Barber, S.: Creating Effective Load Models for Performance Testing with Incomplete Empirical Data. In: Proc. Sixth IEEE International Workshop on Web Site Evolution (WSE 2004) (2004)Google Scholar
  5. 5.
    Bertolino, A., Mirandola, R.: CB-SPE Tool: Putting Component-Based Performance Engineering into Practice. In: Proc. 7th Int. Symposium on Component-Based Software Engineering, Edinburgh (2004)Google Scholar
  6. 6.
    Bondarev, E., Muskens, J., de With, P., Chaudron, M., Lukkien, J.: Predicting Real-Time Properties of Component Assemblies. In: Proc. 30th Euro. Micro. Conf. IEEE, Los Alamitos (2004)Google Scholar
  7. 7.
    Dang, L., Jennings, C., Kelly, D.: Practical VoIP Using VOCAL. O’Reilly Media, Sebastopol (2002)Google Scholar
  8. 8.
    Denaro, A.P.G., Emmerich, W.: Early Performance Testing of Distributed Software Applications. In: Proc. 4th Int. Workshop on Software and Performance, Redwood Shores, California, January 2004, pp. 94–103 (2004)Google Scholar
  9. 9.
    Eskenazi, E., Fioukov, A.V., Hammer, D.K., Obbink, H.: Performance prediction for industrial software with the APPEAR method. In: Proc. 4th Progess Symp. on Embedded Systems (2003)Google Scholar
  10. 10.
    Franks, R.G., et al.: Performance Analysis of Distributed Server Systems. In: Proc. 6th Int. Conf. on Software Quality, Ottawa, October 28-30, 1996, pp. 15–26 (1996)Google Scholar
  11. 11.
    Franks, R.G., et al.: Layered Queueing Network Solver and Simulator User Manual, Dept. of Systems and Computer Engineering, Carleton University (December 2005)Google Scholar
  12. 12.
    Hissam, S.A., et al.: Packaging Predictable Assembly. In: Bishop, J.M. (ed.) CD 2002. LNCS, vol. 2370, pp. 108–124. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  13. 13.
    Israr, T., Woodside, M., Franks, R.G.: Interaction tree algorithms to extract effective architecture and layered performance models from traces. J. Systems and Software 80(4), 474–492 (2007)CrossRefGoogle Scholar
  14. 14.
    Jazwinski, A.H.: Stochastic Processes and Filtering Theory. Academic Press, London (1970)MATHGoogle Scholar
  15. 15.
    Kutner, M.H., Nachtsheim, C.J., Neter, J., Li, W.: Applied Linear Statistical Models, 5th edn. McGraw-Hill, New York (2005)Google Scholar
  16. 16.
    Liu, Y., Fekete, A., Gorton, I.: Design-Level Performance Prediction of Component-Based Applications. IEEE Trans. on Software Eng. 31(11) (November 2005)Google Scholar
  17. 17.
    Miller, B.P., Callaghan, M.D., Cargille, J.M., Hollingsworth, J.K., Irvin, R.B., Karavanic, K.L., Kunchithapadam, K., Newhall, T.: The Paradyn Parallel Performance Measurement Tool. IEEE Computer 28(11) (November 1995)Google Scholar
  18. 18.
    Musa, J.D.: The Operational Profile in Software Reliability Engineering: An Overview. IEEE Software 10, 14–32 (1993)CrossRefGoogle Scholar
  19. 19.
    Muskens, J., Chaudron, M.: Prediction of Run-Time Resource Consumption in Multi-task Component-Based Software Systems. In: Proc. 7th Int. Symp. on Component-Based Software Engineering, Edinburgh, May 24-25 (2004)Google Scholar
  20. 20.
    Object Management Group, UML Profile for Schedulability, Performance, and Time Specification, Version 1.1, OMG document formal/05-01-02 (January 2005)Google Scholar
  21. 21.
    Rosenberg, J., et al.: The Session Initiation Protocol (SIP), IETF RFC 3261 (June 2002), http://www.ietf.org/rfc/rfc3261.txt
  22. 22.
    Smith, C.U., Williams, L.G.: Performance Solutions. Addison-Wesley, Reading (2002)Google Scholar
  23. 23.
    Woodside, C.M., Neilson, J.E., Petriu, D.C., Majumdar, S.: The Stochastic Rendezvous Network Model for Performance of Synchronous Client-Server-Like Distributed Software. IEEE Trans. on Computers 44(1), 20–34 (1995)MATHCrossRefGoogle Scholar
  24. 24.
    Woodside, M., Petriu, D.C., Petriu, D.B., Shen, H., Israr, T., Merseguer, J.: Performance by Unified Model Analysis (PUMA). In: Proc. 5th Int. Workshop on Software and Performance, Palma de Mallorca, pp. 1–12 (2005)Google Scholar
  25. 25.
    Woodside, M., Franks, R.G., Petriu, D.C.: The Future of Software Performance Engineering. In: Briand, L., Wolf, A. (eds.) Proc. Future of Software Engineering 2007, May 2007, vol. P2829, pp. 171–187. IEEE Computer Society, Los Alamitos (2007)CrossRefGoogle Scholar
  26. 26.
    Woodside, M., Zheng, T., Litoiu, M.: Service system resource management based on a tracked layered performance model. In: Proc. 3rd Int. Conf. on Autonomic Computing (ICAC 2006), pp. 123–133 (2006)Google Scholar
  27. 27.
    Woodside, M.: The Relationship of Performance Models to Data. In: Proc. SPEC Int. Performance Evaluation Workshop (SIPEW 2008). LNCS, vol. 5119, Darmstadt. Springer, Heidelberg (2008)Google Scholar
  28. 28.
    Wu, X., Woodside, M.: Performance Modeling from Software Components. In: Proc. 4th Int. Workshop on Software and Performance, Redwood Shores, CA, pp. 290–301 (2004)Google Scholar
  29. 29.
    Zheng, T., Woodside, M., Litoiu, M.: Performance Model Estimation and Tracking using Optimal Filters. IEEE Trans. on Software Eng. (June 2008)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Xiuping Wu
    • 1
  • Murray Woodside
    • 1
  1. 1.Dept. of Systems & Computer Eng.Carleton UniversityOttawaCanada

Personalised recommendations