Skip to main content

Performance Regression Unit Testing: A Case Study

  • Conference paper

Part of the Lecture Notes in Computer Science book series (LNPSE,volume 8168)


Including performance tests as a part of unit testing is technically more difficult than including functional tests – besides the usual challenges of performance measurement, specifying and testing the correctness conditions is also more complex. In earlier work, we have proposed a formalism for expressing these conditions, the Stochastic Performance Logic. In this paper, we evaluate our formalism in the context of performance unit testing of JDOM, an open source project for working with XML data. We focus on the ability to capture and test developer assumptions and on the practical behavior of the built in hypothesis testing when the formal assumptions of the tests are not met.


  • Stochastic Performance Logic
  • regression testing
  • performance testing
  • unit testing
  • performance evaluation

This is a preview of subscription content, access via your institution.

Buying options

USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-642-40725-3_12
  • Chapter length: 15 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
USD   49.99
Price excludes VAT (USA)
  • ISBN: 978-3-642-40725-3
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   64.99
Price excludes VAT (USA)


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Bergmann, V.: ContiPerf 2 (2013),

  2. Bulej, L., Bures, T., Keznikl, J., Koubkova, A., Podzimek, A., Tuma, P.: Capturing Performance Assumptions using Stochastic Performance Logic. In: Proc. ICPE 2012. ACM (2012)

    Google Scholar 

  3. Clark, M.: JUnitPerf (2013),

  4. Foo, K., Jiang, Z.M., Adams, B., Hassan, A., Zou, Y., Flora, P.: Mining performance regression testing repositories for automated performance analysis. In: Proc. QSIC 2010. IEEE (2010)

    Google Scholar 

  5. Ghaith, S., Wang, M., Perry, P., Murphy, J.: Profile-based, load-independent anomaly detection and analysis in performance regression testing of software systems. In: Proc. CSMR 2013. IEEE (2013)

    Google Scholar 

  6. Heger, C., Happe, J., Farahbod, R.: Automated root cause isolation of performance regressions during software development. In: Proc. ICPE 2013. ACM (2013)

    Google Scholar 

  7. JDOM (2013),

  8. hunterhacker/jdom [Git] (2013),

  9. hunterhacker/jdom: Verifier performance (2013),

  10. JUnit (April 2013),

  11. Kalibera, T., Bulej, L., Tůma, P.: Benchmark Precision and Random Initial State. In: Proc. SPECTS 2005. SCS (2005)

    Google Scholar 

  12. Kalibera, T., Tůma, P.: Precise Regression Benchmarking with Random Effects: Improving Mono Benchmark Results. In: Horváth, A., Telek, M. (eds.) EPEW 2006. LNCS, vol. 4054, pp. 63–77. Springer, Heidelberg (2006)

    CrossRef  Google Scholar 

  13. Oliveira, A., Petkovich, J.-C., Reidemeister, T., Fischmeister, S.: Datamill: Rigorous performance evaluation made easy. In: Proc. ICPE 2013. ACM (2013)

    Google Scholar 

  14. Porter, A., Yilmaz, C., Memon, A.M., Schmidt, D.C., Natarajan, B.: Skoll: A process and infrastructure for distributed continuous quality assurance. IEEE Trans. Softw. Eng. 33(8), 510–525 (2007)

    CrossRef  Google Scholar 

  15. Puchko, T.: Retrotranslator (2013),

  16. SPL Tools (2013),

  17. Welch, B.L.: The Generalization of Student’s Problem when Several Different Population Variances are Involved. Biometrika 34(1/2), 28–35 (1947)

    MathSciNet  MATH  CrossRef  Google Scholar 

Download references

Author information

Authors and Affiliations


Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2013 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Horký, V., Haas, F., Kotrč, J., Lacina, M., Tůma, P. (2013). Performance Regression Unit Testing: A Case Study. In: Balsamo, M.S., Knottenbelt, W.J., Marin, A. (eds) Computer Performance Engineering. EPEW 2013. Lecture Notes in Computer Science, vol 8168. Springer, Berlin, Heidelberg.

Download citation

  • DOI:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-40724-6

  • Online ISBN: 978-3-642-40725-3

  • eBook Packages: Computer ScienceComputer Science (R0)