Accuracy of software quality models over multiple releases

Abstract

Many evolving mission‐critical systems must have high software reliability. However, it is often difficult to identify fault‐prone modules early enough in a development cycle to guide software enhancement efforts effectively and efficiently. Software quality models can yield timely predictions of membership in the fault‐prone class on a module‐by‐module basis, enabling one to target enhancement techniques. However, it is an open empirical question, “Can a software quality model remain useful over several releases?” Most prior software quality studies have examined only one release of a system, evaluating the model with modules from the same release. We conducted a case study of a large legacy telecommunications system where measurements on one software release were used to build models, and three subsequent releases of the same system were used to evaluate model accuracy. This is a realistic assessment of model accuracy, closely simulating actual use of a software quality model. A module was considered fault‐prone if any of its faults were discovered by customers. These faults are extremely expensive due to consequent loss of service and emergency repair efforts. We found that the model maintained useful accuracy over several releases. These findings are initial empirical evidence that software quality models can remain useful as a system is maintained by a stable software development process.

This is a preview of subscription content, access via your institution.

References

  1. Basili, V.R., L.C. Briand, and W. Melo (1996), “A Validation of Object-Oriented Design Metrics as Quality Indicators,” IEEE Transactions on Software Engineering 22, 10, 751–761.

    Article  Google Scholar 

  2. Biyani, S. and P. Santhanam (1998), “Exploring Defect Data from Development and Customer Usage on Software Modules over Multiple releases,” In Proceedings the Ninth International Symposium on Software Reliability Engineering, IEEE Computer Society Press, Los Alamitos, CA, pp. 316–320.

    Google Scholar 

  3. Briand, L.C. and V.R. Basili (1992), “A Classification Procedure for the Effective Management of Changes during the Maintenance Process,” In Proceedings of the International Conference on Software Maintenance, IEEE Computer Society Press, Los Alamitos, CA, pp. 328–336.

    Google Scholar 

  4. Briand, L.C., V.R. Basili, and C.J. Hetmanski (1993), “Developing Interpretable Models with Optimized Set Reduction for Identifying High-Risk Software Components,” IEEE Transactions on Software Engineering 19, 11, 1028–1044.

    Article  Google Scholar 

  5. Coleman, D., B. Lowther, and P. Oman (1995), “The Application of Software Maintainability Models in Industrial Software,” Journal of Systems and Software 29, 1, 3–16.

    Article  Google Scholar 

  6. Ebert, C. (1996), “Classification Techniques for Metric-Based Software Development,” Software Quality Journal 5, 4, 255–272.

    MathSciNet  Article  Google Scholar 

  7. Fayyad, U.M. (1996), “Data Mining and Knowledge Discovery: Making Sense Out of Data,” IEEE Expert 11, 4, 20–25.

    Article  Google Scholar 

  8. Hosmer, D.W., Jr. and S. Lemeshow (1989), “Applied Logistic Regression,” Wiley, New York.

    Google Scholar 

  9. Hudepohl, J.P., S.J. Aud, T.M. Khoshgoftaar, E.B. Allen, and J. Mayrand (1996), “EMERALD: Software Metrics and Models on the Desktop,” IEEE Software 13, 5, 56–60.

    Article  Google Scholar 

  10. Jones, W.D., J.P. Hudepohl, T.M. Khoshgoftaar, and E.B. Allen (1999), “Application of a Usage Profile in Software QualityModels,” In Proceedings of the Third European Conference on Software Maintenance and Reengineering, IEEE Computer Society Press, Los Alamitos, CA, pp. 148–157.

    Google Scholar 

  11. Khoshgoftaar, T.M. and E.B. Allen (1999), “A Practical Classification Rule for Software Quality Models,” IEEE Transactions on Reliability 48.

  12. Khoshgoftaar, T.M., E.B. Allen, K.S. Kalaichelvan, and N. Goel (1996), “Early Quality Prediction: A Case Study in Telecommunications,” IEEE Software 13, 1, 65–71.

    Article  Google Scholar 

  13. Khoshgoftaar, T.M., E.B. Allen, R. Halstead, G.P. Trio, and R. Flass (1998a), “Process Measures for Predicting Software Quality,” Computer 31, 4, 66–72.

    Article  Google Scholar 

  14. Khoshgoftaar, T.M., E.B. Allen, W.D. Jones, and J.P. Hudepohl (1998b), “Return on Investment of Software Quality Models,” In Proceedings 1998 IEEE Workshop on Application-Specific Software Engineering and Technology, IEEE Computer Society Press, Los Alamitos, CA, pp. 145–150.

    Google Scholar 

  15. Khoshgoftaar, T.M., E.B. Allen, A. Naik, W.D. Jones, and J.P. Hudepohl (1998c), “Using Classification Trees for Software Quality Models: Lessons Learned,” In Proceedings of the Third IEEE International High-Assurance Systems Engineering Symposium, IEEE Computer Society Press, Los Alamitos, CA, pp. 82–89.

    Google Scholar 

  16. Khoshgoftaar, T.M., E.B. Allen, W.D. Jones, and J.P. Hudepohl (1999), “Which Software Modules Have Faults that Will Be Discovered by Customers?” Journal of Software Maintenance: Research and Practice 11, 1, 1–18.

    Article  Google Scholar 

  17. Khoshgoftaar, T.M. and D.L. Lanning (1995), “A Neural Network Approach for Early Detection of Program Modules having High Risk in the Maintenance Phase,” Journal of Systems and Software 29, 1, 85–91.

    Article  Google Scholar 

  18. Mayrand, J. and F. Coallier (1996), “System Acquisition Based on Software Product Assessment,” In Proceedings of the Eighteenth International Conference on Software Engineering, IEEE Computer Society Press, Los Alamitos, CA, pp. 210–219.

    Google Scholar 

  19. Munson, J.C. and T.M. Khoshgoftaar (1992), “The Detection of Fault-Prone Programs,” IEEE Transactions on Software Engineering 18, 5, 423–433.

    Article  Google Scholar 

  20. Myers, R.H. (1990), Classical and Modern Regression with Applications, Duxbury Series, PWS-KENT Publishing, Boston.

    Google Scholar 

  21. Schneidewind, N.F. (1995), “Software Metrics Validation: Space Shuttle Flight Software Example,” Annals of Software Engineering 1, 287–309.

  22. Schneidewind, N.F. (1998), “An Integrated Process and Product Model,” In Proceedings Fifth International Software Metrics Symposium, IEEE Computer Society Press, Los Alamitos, CA, pp. 224–234.

    Google Scholar 

  23. Seber, G.A.F. (1984), Multivariate Observations, Wiley, New York.

    Google Scholar 

  24. Selby, R.W. and A.A. Porter (1988), “Learning from Examples: Generation and Evaluation of Decision Trees for Software Resource Analysis,” IEEE Transactions on Software Engineering 14, 12, 1743–1756.

    Article  Google Scholar 

Download references

Author information

Affiliations

Authors

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Khoshgoftaar, T.M., Allen, E.B., Jones, W.D. et al. Accuracy of software quality models over multiple releases. Annals of Software Engineering 9, 103–116 (2000). https://doi.org/10.1023/A:1018972607783

Download citation

Keywords

  • IEEE Computer Society
  • Software Quality
  • Software Metrics
  • Multiple Release
  • Open Empirical Question