Quality Assurance in Performance: Evaluating Mono Benchmark Results

  • Tomas Kalibera
  • Lubomir Bulej
  • Petr Tuma
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3712)


Performance is an important aspect of software quality. To prevent performance degradation during software development, performance can be monitored and software modifications that damage performance can be reverted or optimized. Regression benchmarking provides means for an automated monitoring of performance, yielding a list of software modifications potentially associated with performance changes. We focus on locating individual modifications as causes of individual performance changes and present three methods that help narrow down the list of modifications potentially associated with a performance change. We illustrate the entire process on a real world project.


Virtual Machine Performance Change Operation Duration Performance Regression Benchmark Result 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Jeffries, R.E., Anderson, A., Hendrickson, C.: Extreme Programming Installed. Addison-Wesley Longman Publishing Co., Inc., Boston (2000)Google Scholar
  2. 2.
    Bulej, L., Kalibera, T., Tuma, P.: Repeated results analysis for middleware regression benchmarking. Performance Evaluation 60, 345–358 (2005)CrossRefGoogle Scholar
  3. 3.
    Bulej, L., Kalibera, T., Tuma, P.: Regression benchmarking with simple middleware benchmarks. In: Hassanein, H., Olivier, R.L., Richard, G.G., Wilson, L.L. (eds.) International Workshop on Middleware Performance, IPCCC 2004, pp. 771–776 (2004)Google Scholar
  4. 4.
    Kalibera, T., Bulej, L., Tuma, P.: Generic environment for full automation of benchmarking. In: Beydeda, S., Gruhn, V., Mayer, J., Reussner, R., Schweiggert, F. (eds.) SOQUA/TECOS. LNI, GI vol. 58, pp. 125–132 (2004)Google Scholar
  5. 5.
    Novell, Inc.: The Mono Project (2005),
  6. 6.
    ECMA: ECMA-335: Common Language Infrastructure (CLI). ECMA (European Association for Standardizing Information and Communication Systems), Geneva, Switzerland (2002)Google Scholar
  7. 7.
    Distributed Systems Research Group: Mono regression benchmarking (2005),
  8. 8.
    DOC Group: TAO performance scoreboard (2005),
  9. 9.
    Prochazka, M., Madan, A., Vitek, J., Liu, W.: RTJBench: A Real-Time Java Benchmarking Framework. In: Component And Middleware Performance Workshop, OOPSLA 2004 (2004)Google Scholar
  10. 10.
    Dillenseger, B., Cecchet, E.: CLIF is a Load Injection Framework. In: Workshop on Middleware Benchmarking: Approaches, Results, Experiences, OOPSLA 2003 (2003)Google Scholar
  11. 11.
    Memon, A.M., Porter, A.A., Yilmaz, C., Nagarajan, A., Schmidt, D.C., Natarajan, B.: Skoll: Distributed continuous quality assurance. In: ICSE, pp. 459–468. IEEE Computer Society, Los Alamitos (2004)Google Scholar
  12. 12.
    Courson, M., Mink, A., Marçais, G., Traverse, B.: An automated benchmarking toolset. In: Bubak, M., Afsarmanesh, H., Williams, R., Hertzberger, L.O. (eds.) HPCN-Europe 2000. LNCS, vol. 1823, pp. 497–506. Springer, Heidelberg (2000)CrossRefGoogle Scholar
  13. 13.
    Kalibera, T., Bulej, L., Tuma, P.: Benchmark precision and random initial state. In: accepted for 2005 International Symposium on Performance Evaluation of Computer and Telecommunications Systems, SPECTS 2005 (2005)Google Scholar
  14. 14.
    Buble, A., Bulej, L., Tuma, P.: CORBA benchmarking: A course with hidden obstacles. In: IPDPS, p. 279. IEEE Computer Society, Los Alamitos (2003)Google Scholar
  15. 15.
    Hauswirth, M., Sweeney, P., Diwan, A., Hind, M.: The need for a whole-system view of performance. In: Component And Middleware Performance workshop, OOPSLA 2004 (2004)Google Scholar
  16. 16.
    Gu, D., Verbrugge, C., Gagnon, E.: Code layout as a source of noise in JVM performance. In: Component And Middleware Performance Workshop, OOPSLA 2004 (2004)Google Scholar
  17. 17.
    The FreeBSD Documentation Project: FreeBSD Developers’ Handbook (2005),
  18. 18.
    Jain, R.: The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling. Wiley-Interscience, New York (1991)zbMATHGoogle Scholar
  19. 19.
    Re, C., Vogels, W.: SciMark – C# (2004),
  20. 20.
    Pozo, R., Miller, B.: SciMark 2.0 benchmark (2005),

Copyright information

© Springer-Verlag Berlin Heidelberg 2005

Authors and Affiliations

  • Tomas Kalibera
    • 1
  • Lubomir Bulej
    • 1
    • 2
  • Petr Tuma
    • 1
  1. 1.Distributed Systems Research Group,Department of Software Engineering, Faculty of Mathematics and PhysicsCharles UniversityPragueCzech Republic
  2. 2.Institute of Computer ScienceCzech Academy of SciencesPragueCzech Republic

Personalised recommendations