Quality Impact of Introducing Component-Level Test Automation and Test-Driven Development

  • Lars-Ola Damm
  • Lars Lundberg
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4764)


Companies spend significant efforts on testing their products to achieve a sufficient quality level. This paper presents results from evaluating the quality impact of implementing a framework for component-level test automation and Test-Driven Development. The evaluation comprised six projects for two products at a software development department at Ericsson. The paper suggests how an existing measurement approach can be used for evaluating the quality impact of improvements in early phases, i.e. by classifying faults reported on released products after which phase they should have been caught in. Based on this measurement approach, the evaluation determined that the ratio of reported faults in the released products decreased significantly after implementing the framework. That is, the ratio of faults belonging to component-level testing decreased from between 60-70 percent to less than 20 percent in the two studied products.


Unit Test Fault Slip Fault Distribution Test Framework Quality Impact 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bassin, K.A., Kratschmer, T., Santhanam, P.: Evaluating Software Objectively. IEEE Software 15(6), 66–74 (1998)CrossRefGoogle Scholar
  2. 2.
    Beck, K.: Test Driven Development - by example. Addison-Wesley, Boston, MA (2003)Google Scholar
  3. 3.
    Beizer, B.: Software Testing Techniques, 2nd edn. Van Nostrand Reinhold Company, New York (1990)Google Scholar
  4. 4.
    Bhat, T., Nagappan, N.: Evaluating the Efficacy of Test-Driven Development: Industrial Case Studies. In: Proceedings of the 5th International Symposium on Empirical Software Engineering, pp. 356–363. ACM Press, New York (2006)CrossRefGoogle Scholar
  5. 5.
    Bundell, G.A., Lee, G., Morris, J., Parker, K., Peng, L.: A software component verification tool. In: Proceedings of International Conference on Software Methods and Tools, pp. 137–146. IEEE Computer Soc., Los AlamitosGoogle Scholar
  6. 6.
    Chillarege, R., Bhandari, I., Chaar, M., Halliday, D., Moebus, B., Ray, B., Wong, M.Y.: Orthogonal Defect Classification-A concept for In-Process Measurement. IEEE Transactions on Software Engineering 18, 943–956 (1992)CrossRefGoogle Scholar
  7. 7.
    Cockburn, A.: Agile Software Development. Addison-Wesley, Boston, MA (2002)Google Scholar
  8. 8.
    Crispin, L.: Driving Software Quality: How Test-Driven Development Impacts Software Quality. IEEE Software 23(6), 70–71 (2006)CrossRefGoogle Scholar
  9. 9.
    Damm, L.-O., Lundberg, L.: Introducing Test Automation and Test-Driven Development: An Experience Report. In: Proceedings of the International Workshop on Test and Analysis of Component-Based Systems. Electronic Notes in Theoretical Computer Science, vol. 316, pp. 3–15. Elsevier, Amsterdam (2004)Google Scholar
  10. 10.
    Damm, L.-O., Lundberg, L.: Identification of Test Process Improvements by Combining ODC Triggers and Faults-Slip-Through. In: Proceedings of the 4th International Symposium on Empirical Software Engineering, pp. 152–161. IEEE, Los Alamitos (2005)Google Scholar
  11. 11.
    Damm, L.-O., Lundberg, L.: Results from Introducing Component-Level Test Automation and Test-Driven Development. Journal of Systems and Software 79, 1001–1014 (2006)CrossRefGoogle Scholar
  12. 12.
    Damm, L.-O., Lundberg, L., Wohlin, C.: Faults-Slip-Through - A Concept for Measuring the Efficiency of the Test Process. Wiley Journal of Software: Process Improvement and Practice 11, 47–59 (2006)CrossRefGoogle Scholar
  13. 13.
    Erdogmus, H., Morisio, M.: On the Effectiveness of the Test-First Approach to Programming. IEEE Transactions on Software Engineering 31(3), 226–237 (2005)CrossRefGoogle Scholar
  14. 14.
    Fenton, N., Pfleeger, S.L.: Software Metrics: A Rigorous Approach. PWS Publishing Company (1997)Google Scholar
  15. 15.
    Gao, J., Tsao, J., Wu, Y.: Testing and Quality Assurance for Component-Based Software. Artech House Publishers (2003)Google Scholar
  16. 16.
    George, B., Williams, L.: A structured experiment of test-driven development. Information and Software Technology 46, 337–342 (2004)CrossRefGoogle Scholar
  17. 17.
    Geras, A., Smith, M.R., Miller, J.: A Prototype Empirical Evaluation of Test Driven Development. In: Proceedings of the 10th IEEE International Software Metrics Symposium, pp. 405–416. IEEE Computer Society, Los Alamitos (2004)CrossRefGoogle Scholar
  18. 18.
    Harrold, J.: Testing: a roadmap. In: International Conference on Software Engineering, pp. 61–72. ACM, New York (2000)Google Scholar
  19. 19.
    Hevner, A.R.: Phase Containment for Software Quality Improvement. Information and Software Technology 39, 867–877 (1997)CrossRefGoogle Scholar
  20. 20.
    Howles, T., Daniels, S.: Widespread Effects of Defects. Quality Progress 36(8), 58–62 (2003)Google Scholar
  21. 21.
    Maximilien, E.M., Williams, L.: Assessing Test-Driven Development at IBM. In: Proceedings of the 25th International Conference on Software Engineering, pp. 564–569. IEEE Computer Soc. Press, Los Alamitos (2003)CrossRefGoogle Scholar
  22. 22.
    Robson, C.: Real World Research, 2nd edn. Blackwell Publishers, Oxford, UK (2002)Google Scholar
  23. 23.
    Runeson, P., Andersson, C., Thelin, T., Andrews, A., Berling, T.: What Do We Know about Defect Detection Methods? IEEE Software 23(3), 82–90 (2006)CrossRefGoogle Scholar
  24. 24.
    Shull, F., Basili, V., Boehm, B., Brown, W., Costa, P., Lindwall, M., Port, D., Rus, I., Tesoriero, R., Zelkowitz, M.: What We Have Learned About Fighting Defects. In: Proceedings of the Eight IEEE Symposium on Software Metrics, pp. 249–258. IEEE, Los Alamitos (2002)CrossRefGoogle Scholar
  25. 25.
    Teiniker, E., Mitterdorfer, S., Johnson, L.M., Johnson, L.M., Kreiner, C., Kovacs, Z., Weiss, R.: A Test-Driven Component Development Framework Based On The Corba Component Model. In: Proceedings of the 27th Annual International Computer Software and Applications Conference, pp. 400–405. IEEE, Los Alamitos (2003)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2007

Authors and Affiliations

  • Lars-Ola Damm
    • 1
  • Lars Lundberg
    • 1
  1. 1.School of Engineering, Blekinge Institute of Technology, Box 520, SE-372 25 RonnebySweden

Personalised recommendations