Predicting defective modules in different test phases
Defect prediction is a well-established research area in software engineering . Prediction models in the literature do not predict defect-prone modules in different test phases. We investigate the relationships between defects and test phases in order to build defect prediction models for different test phases. We mined the version history of a large-scale enterprise software product to extract churn and static code metrics. We used three testing phases that have been employed by our industry partner, namely function, system and field, to build a learning-based model for each testing phase. We examined the relation of different defect symptoms with the testing phases. We compared the performance of our proposed model with a benchmark model that has been constructed for the entire test phase (benchmark model). Our results show that building a model to predict defect-prone modules for each test phase significantly improves defect prediction performance and shortens defect detection time. The benefit analysis shows that using the proposed model, the defects are detected on the average 7 months earlier than the actual. The outcome of prediction models should lead to an action in a software development organization. Our proposed model gives a more granular outcome in terms of predicting defect-prone modules in each testing phase so that managers may better organize the testing teams and effort.
KeywordsSoftware testing Testing phase Defect prediction
This research is supported in part by Turkish State Planning Organization (DPT) under the Project Number 2007K120610 and by NSERC Project number 402003-2012. We would like to thank IBM Canada Lab – Toronto site for making their development data available for research and strategic help during all phases of this research. The opinions expressed in this paper are those of the authors and not necessarily of IBM Corporation.
- Alpaydin, E. (2004). Introduction to machine learning. Cambridge: The MIT Press.Google Scholar
- Boehm, B., Basili, V.R., (2001). Software defect reduction top 10 list. Computer 34(1), 135–137. doi:http://doi.ieeecomputersociety.org/10.1109/2.962984.
- Brooks, F., (1995). The mythical man-month: Essays on software engineering. Reading, MA: Addison-Wesley Pub. Co., http://books.google.ca/books?id=fUYPAQAAMAAJ.
- Caglayan, B., & Bener, A. (2014). DSL-2014-02 metric distributions for the enterprise software dataset for modules in different test phases. Tech. rep., Ryerson University, Data Science Laboratory.Google Scholar
- Caglayan, B., Bener, A., Koch, S., (2009). Merits of using repository metrics in defect prediction for open source projects. In Proceedings of the 2009 ICSE workshop on emerging trends in Free/Libre/Open source software research and development, IEEE Computer Society, Washington, DC, USA, FLOSS ’09, pp. 31–36.Google Scholar
- Caglayan, B., Tosun, A., Miranskyy, A., Bener, A., Ruffolo, N. (2010). Usage of multiple prediction models based on defect categories. In Proceedings of the 6th International conference on predictive models in software engineering, ACM, New York, NY, USA, PROMISE ’10, pp 8:1–8:9.Google Scholar
- Dallmeier, V., Zimmermann, T. (2007). Extraction of bug localization benchmarks from history. In Proceedings of the 22nd IEEE/ACM international conference on automated software engineering.Google Scholar
- Di Fatta, G., Leue, S., & Stegantova, E. (2006). Discriminative pattern mining in software fault detection. In SOQUA ’06: Proceedings of the 3rd international workshop on software quality assurance (pp. 62–69). New York, NY, USA: ACM.Google Scholar
- Gelman, A., Hill, J. (2007). Data analysis using regression and multilevel/hierarchical models. Analytical methods for social research, Cambridge: Cambridge University Press. http://books.google.com.tr/books?id=c9xLKzZWoZ4C.
- IEEE Standard Computer Dictionary. (1991). A compilation of IEEE standard computer glossaries. IEEE Std 610, pp 1. 10.1109/IEEESTD.1991.106963.
- Jiang, Y., Cukic, B., Menzies, T., (2008). Can data transformation help in the detection of fault-prone modules? In DEFECTS ’08: Proceedings of the 2008 workshop on defects in large software systems, ACM, New York, NY, USA, pp. 16–20. doi: 10.1145/1390817.1390822.
- Kocaguneli, E., Tosun, A., Bener, A.B., Turhan, B., Caglayan, B. (2009). Prest: An intelligent software metrics extraction, analysis and defect prediction tool. In SEKE, pp. 637–642.Google Scholar
- Lessmann, S., Baesens, B., Mues, C., Pietsch, S. (2008). Benchmarking classification models for software defect prediction: A proposed framework and novel findings. IEEE Transactions on Software Engineering 34(4), 485–496. http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=4527256.
- Li, N., Li, Z., Zhang, L. (2010). Mining frequent patterns from software defect repositories for black-box testing. In Intelligent systems and applications (ISA), 2010 2nd international workshop on, pp. 1–4. doi: 10.1109/IWISA.2010.5473578.
- Maloof, M.A. (2003). Learning when data sets are imbalanced and when costs are unequal and unknown. In ICML-2003 workshop on learning from imbalanced data sets II.Google Scholar
- Meneely, A., Williams, L., Snipes, W., & Osborne, J. (2008). Predicting failures with developer networks and social network analysis. In SIGSOFT ’08/FSE-16: Proceedings of the 16th ACM SIGSOFT international symposium on foundations of software engineering (pp. 13–23). New York, NY, USA: ACM.Google Scholar
- Misirli, A. T., Bener, A. B., & Kale, R. (2011). Ai-based software defect predictors: Applications and benefits in a case study. AI Magazine, 32(2), 57–68.Google Scholar
- Misirli, A. T., Caglayan, B., Bener, A., Turhan, B. (2013). A retrospective study of software analytics projects: In-depth interviews with practitioners. IEEE Software 30(5):54–61. doi: 10.1109/MS.2013.93.
- Myers, G. J., Badgett, T., Thomas, T., & Sandler, C. (2004). The art of software testing (2nd ed.). New York: Wiley.Google Scholar
- Nagappan, N., & Ball, T. (2007). Using software dependencies and churn metrics to predict field failures: An empirical case study. In ESEM ’07: Proceedings of the first international symposium on empirical software engineering and measurement (pp. 364–373). Washington, DC, USA: IEEE Computer Society.Google Scholar
- Nagappan, N., Ball, T., & Zeller, A. (2006). Mining metrics to predict component failures. In Proceedings of the 28th international conference on software engineering, ACM, Shanghai, China.Google Scholar
- Ostrand, T. J., Weyuker, E. J., & Bell, R. M. (2004). Where the bugs are. In Proceedings of the 2004 ACM SIGSOFT international symposium on software testing and analysis. ACM, New York, NY, USA, ISSTA ’04, pp. 86–96. doi: 10.1145/1007512.1007524.
- Ostrand, T. J., Weyuker, E. J., & Bell, R. M. (2007). Automating algorithms for the identification of fault-prone files. In Proceedings of the 2007 international symposium on Software testing and analysis (pp. 219–227). ACM.Google Scholar
- Pressman, R. (2010). Software engineering: a practitioner’s approach. McGraw-Hill higher education, McGraw-Hill Higher Education. http://books.google.ca/books?id=y4k_AQAAIAAJ.
- Ratzinger, J., Pinzger, M., & Gall, H. (2007). EQ-Mine: Predicting short-term defects for software evolution. In M. B. Dwyer & A. Lopes (Eds.), Proceedings of the fundamental approaches to software engineering at the European joint conference on theory and practice of software (pp. 12–26). Berlin: Springer.Google Scholar
- Schroter, A., Zimmermann, T., Premraj, R., Zeller, A. (2006). If your bug database could talk. In Proceedings of the 5th international symposium on empirical software engineering, Volume II: Short papers and posters, pp 18–20.Google Scholar
- Shull, F., Basili, V., Boehm, B., Brown, A.W., Costa, P., Lindvall, M., Port, D., Rus, I., Tesoriero, R., Zelkowitz, M. (2002). What we have learned about fighting defects. In Proceedings of 8th international software metrics symposium, pp 249–258.Google Scholar
- Tosun, A., Turhan, B., & Bener, A. (2009a). Practical considerations in deploying ai for defect prediction: A case study within the turkish telecommunication industry. In PROMISE ’09: Proceedings of the 5th international conference on predictor models in software engineering (pp. 1–9). New York, NY, USA: ACM.Google Scholar
- Tosun, A., Turhan, B., & Bener, A. (2009b). Validation of network measures as indicators of defective modules in software systems. In PROMISE ’09: Proceedings of the 5th international conference on predictor models in software engineering (pp. 1–9). New York, NY, USA: ACM.Google Scholar
- Turhan, B., Bener, A. (2007). A multivariate analysis of static code attributes for defect prediction. In Quality software, 2007. QSIC ’07. Seventh international conference on, pp 231–237.Google Scholar
- Weyuker, E. J., Ostrand, T. J., Bell, R. M. (2007). Using developer information as a factor for fault prediction. In Proceedings of the third international workshop on predictor models in software engineering, IEEE computer society. doi: 10.1109/PROMISE.2007.14. http://portal.acm.org/citation.cfm?id=1269056.
- Zimmermann, T., & Nagappan, N. (2008). Predicting defects using network analysis on dependency graphs. In ICSE ’08: Proceedings of the 30th international conference on software engineering (pp. 531–540). New York, NY, USA: ACM.Google Scholar
- Zimmermann, T., Nagappan, N. (2009). Predicting defects with program dependencies. 2009 3rd international symposium on empirical software engineering and measurement pp 435–438. doi:10.1109/ESEM.2009.5316024. http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=5316024.
- Zimmermann, T., Premraj, R., Zeller, A. (2007). Predicting defects for eclipse. In PROMISE ’07: Proceedings of the third international workshop on predictor models in software engineering, IEEE computer society, Washington, DC, USA.Google Scholar