Skip to main content
SpringerLink
Log in

All complaints are not created equal: text analysis of open source software defect reports

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

As the use of Open Source Software (OSS) systems increases in the corporate environment, it is important to examine the maintenance process of these projects. OSS projects allow end users to directly submit reports in case of any operational issues. Timely resolution of these defect reports requires effective management of maintenance resources. This study analyzes the usefulness of the textual content of the defect reports as an early indicator of their resolution time. Text Mining techniques are used to categorize defect reports of five OSS projects. Significant variation in the defect resolution time amongst the resulting categories, for each of the sample projects, indicates that a text based classification of defect reports can be useful in early assessment of resolution time before source code level analysis. Such technique can assist in allocation of sufficient maintenance resources to targeted defects and also enable project teams to manage customer expectations regarding defect resolution times.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2

Similar content being viewed by others

Notes

  1. Some OSS projects do offer paid versions that include software maintenance services as well.

  2. The term significant hereafter, in this section, is used to indicate statistical significance.

  3. Asterisk indicates statistical significance at p < 0.05.

References

  • Anvik J, Hiew L, Murphy GC (2005) Coping with an open bug repository. In: Proceedings of the OOPSLA workshop on eclipse technology eXchange. San Diego, California, pp 35–39, 16–17 Oct 2005

  • Anvik J, Hiew L, Murphy GC (2006) Who should fix this bug? In: Proceedings of the 28th international conference on software engineering. ACM, Shanghai, China, pp 361–370, 20–28 May 2006

  • Anvik J, Murphy GC (2007) Determining implementation expertise from bug reports. In: Proceedings of the 4th intl. Workshop on mining software repositories. Washington, DC, USA, p 2

  • Berry MW, Browne M (2005) Understanding search engines: mathematical modeling and text retrieval. Soc for Industrial & Applied Math, Philadelphia, PA

    Book  MATH  Google Scholar 

  • Bettenburg N, Premraj R, Zimmermann T, Kim S (2008) Duplicate bug reports considered harmful really? In: IEEE international conference on software maintenance, ICSM, pp 337–345, 28 Sept 2008–4 Oct 2008

  • Brettschneider R (1989) Is your software ready for release? IEEE Softw 6(4):100, 102, 108

    Google Scholar 

  • Canfora G, Cerulo L (2005) How software repositories can help in resolving a new change request. In: Proceedings of the workshop on empirical studies in reverse engineering

  • Chiarini-Tremblay M, Berndt DJ, Foulis P, Luther S (2005) Utilizing text mining techniques to identify fall related injuries. In: Eleventh Americas conference on information systems, Omaha, NE

  • Chu-Ti L, Chin-Yu H, Chin-Yu H (2006) Software release time management: how to use reliability growth models to make better decisions. In: 2006 IEEE international conference on management of innovation and technology, vol 2, pp 658–662

  • Chulani S, Ray B, Santhanam P, Leszkowicz R (2003) Metrics for managing customer view of software quality. In: Proceedings ninth international software metrics symposium, pp 189–198

  • Cochran WG (1947) Some consequences when the assumptions for the analysis of variances are not satisfied. Biometrics 3(1):22–38

    Article  MathSciNet  Google Scholar 

  • Crowston K, Annabi H, Howison J (2003) Defining open source project success. In: International conference of information systems. Seattle, WA

  • Cubranic D, Murphy GC (2004) Automatic bug triage using text categorization. In: Proceedings of the sixteenth international conference on software engineering & knowledge engineering (SEKE 2004), pp 92–97

  • Deerwester EA (1990) Indexing by latent semantic analysis. J Am Soc Inf Sci Technol 41(6):391–401

    Article  Google Scholar 

  • Di Lucca G, Di Penta M, Gradara S (2002) An approach to classify software maintenance requests. In: Proceedings international conference onSoftware maintenance, pp 93–102

  • Dinh-Trong TT (2005) The freebsd project: a replication case study of open source development. IEEE Trans Softw Eng 31(6):481–494

    Article  Google Scholar 

  • Dit B, Poshyvanyk D, Marcus A (2008) Measuring the semantic similarity of comments in bug reports. In: Proceedings 1st international workshop on semantic technologies in system maintenance (STSM’08)

  • Everitt BS (1998) The Cambridge dictionary of statistics. Cambridge University Press, Cambridge

    MATH  Google Scholar 

  • Goldberg J (1995) CDM: an approach to learning in text categorization. Tools with artificial intelligence, 1995. In: Proceedings seventh international conference, pp 258–265

  • Huang C-Y, Lin C-T (2006) Software reliability analysis by considering fault dependency and debugging time lag. IEEE Trans Reliab 55(3):436–450

    Article  MathSciNet  Google Scholar 

  • Ito PK (1980) Handbook of statistics 1: analysis of variance. Amsterdam: North-Holland, Ch. robustness of ANOVA and MANOVA test procedures, pp 199–236

  • Jensen C, Scacchi W (2004) Data mining for software process discovery in open source software development communities. In: International workshop on mining software repositories (MSR 2004) workshop - 26th international conference on software engineering, pp 96–100

  • Jones C (1996) Software defect-removal efficiency. Computer 29(4):94–95

    Article  Google Scholar 

  • Kemerer CF, Slaughter SA (1997) Determinants of software maintenance profiles: an empirical investigation. J Softw Maint Evol: Research and Practice 9(4):235–251

    Article  Google Scholar 

  • Kemerer CF, Slaughter SA (1999) An empirical approach to studying software evolution. IEEE Trans Softw Eng 25:493–509

    Article  Google Scholar 

  • Kerlinger FN, Lee HB (1999) Foundations of behavioral research, 4th edn. Wadsworth Publishing, New York, NY

    Google Scholar 

  • Ko Y, Seo J (2000) Automatic text categorization by unsupervised learning. In: Linguistics ICOC (ed) Proceedings of the 18th conference on computational linguistics, vol 1. Saarbrcken, Germany, pp 453–459

    Chapter  Google Scholar 

  • Lehman M, Bennett KH (2002) Feast: feedback, evolution and software technology, pp 1996–2001

  • Levendel Y (1990) Reliability analysis of large software systems: defect data modeling. IEEE Trans Softw Eng 16(2):141–152

    Article  Google Scholar 

  • Lewis D, Reguette M (1994) A comparison of two learning algorithms for text categorization. In: Proceedings of SDAIR-94, 3rd annual symposium on document analysis and information retrieval. Las Vegas, NV, pp 81–93

  • Matter D, Adrian Kuhn ON (2009) Assigning bug reports using a vocabulary-based expertise model of developers. In: Proceedings of the working conference on mining software repositories. Los Alamitos, CA, USA, pp 131–140

  • McCallum A, Nigam K, Ungar LH (2000) Efficient clustering of high-dimensional data sets with application to reference matching. In: KDD ’00: proceedings of the sixth ACM SIGKDD international conference on knowledge discovery and data mining. ACM, New York, NY, USA, pp 169–178

    Chapter  Google Scholar 

  • Melouk SH, Raja U, Keskin BB (2010) Managing resource allocation and task prioritization decisions in large scale virtual collaborative development projects. Inf Resour Manage J 23(2):53–76

    Article  Google Scholar 

  • Mockus A, Fielding RT, Herbsleb J (2000) A case study of open source software development: the apache server. In: Proceedings of the 2000 international conference on software engineering, pp 263–272

  • Mockus A, Fielding RT, Herbsleb J (2002) Two case studies of open source software development: Apache and Mozilla. ACM Trans Softw Eng Methodol (TOSEM) 11(3):309–346

    Article  Google Scholar 

  • Mockus A, Votta L (2000) Identifying reasons for software changes using historic databases. In: ICSM ’00: proceedings of the 2000 international conference on software maintenance. IEEE Computer Society, San Jose, CA, p 120

    Google Scholar 

  • Musa JD, Ackerman AF (1989) Quantifying software validation: when to stop testing? IEEE Softw 6:19–27

    Article  Google Scholar 

  • Neter J, Wasserman W, Kutner MH (2004) Applied linear regression models, vol 4. McGraw-Hill/Irwin, Boston, MA

    Google Scholar 

  • Newman M (2002) Software errors cost u.s. economy $59.5 billion annually. Tech. Rep. NIST 2002-10, National Institute of Standards

  • Ohtera H, Yamada S (1990) Optimum software-release time considering an error-detection phenomenon during operation. IEEE Trans Reliab 39:596–599

    Article  MATH  Google Scholar 

  • Pankaj B, Far BH, Ruhe G, Far BH (2005) Explorative study to provide decision support for software release decisions. In: Proceedings of the 21st IEEE international conference on software maintenance 2005, ICSM’05, pp 617–620

  • Porter MF (1980) An algorithm for suffix stripping. Program 14(3):130–137

    Article  Google Scholar 

  • Poulsen K (2004) Software bug contributed to blackout. Tech. rep., Security Focus

  • Raja U, Tretter MJ (2009) Antecedents of open source software defects: a data mining approach to model formulation, validation and testing. Information Technology and Management 10(4):235–251

    Article  Google Scholar 

  • Raja U, Tretter MJ (2010) Classification of software patches: a text mining approach. J Softw Maint: Research and Practice 23(2):69–87

    Article  Google Scholar 

  • Scariano SM, Davenport JM (1987) The effects of violations of independence assumptions in the one-way ANOVA. Am Stat 41(2):123–129

    MathSciNet  Google Scholar 

  • Shadish WR, Cook TD, Campbell DT (2001) Experimental and Quasi-Experimental Designs for Generalized Causal Inference, 2nd edn. Wadsworth Publishing, New York, NY

    Google Scholar 

  • Wang X, Zhang L, Xie T, Anvik J, Sun J (2008) An approach to detecting duplicate bug reports using natural language and execution information. In: ICSE ’08: Proceedings of the 30th international conference on software engineering. ACM, New York, NY, USA, pp 461–470

    Chapter  Google Scholar 

  • Weiss C, Premraj R, Zimmermann T, Zeller, May 20 – 26 2007. How long will it take to fix this bug? In: ICSE workshops MSR ’07. Fourth international workshop on mining software repositories. Minneapolis, MN

  • Williams CC, Hollingsworth JK (2005) Automatic mining of source code repositories to improve bug finding techniques. IEEE Trans Softw Eng 31(6):466–480

    Article  Google Scholar 

  • Xie AM, Hu QP (2007) A study of the modeling and analysis of software fault-detection and fault-correction processes. Qual Reliab Eng Int 23(4):459–470

    Article  MathSciNet  Google Scholar 

  • Xie M, Yang, B (2003) A study of the effect of imperfect debugging on software develpment cost. IEEE Trans Softw Eng 29(5):471–473

    Article  Google Scholar 

  • Ying AT, Murphy GC, Ng R, Chu-Carroll MC (2004) Predicting source code changes by mining change history. IEEE Trans Softw Eng 30(9):574–586

    Article  Google Scholar 

  • Zhang X, Teng X, Pham H (2003)Considering fault removal efficiency in software reliability assessment. IEEE Trans Syst Man Cybern, Part A, Syst Humans 33(1):114–120

    Article  Google Scholar 

  • Zimmermann T, Zeller A, Weissgerber P, Diehl S (2005) Mining version histories to guide software changes. IEEE Trans Softw Eng 31(6):429–445

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Information Systems, Statistics and Management Science, The University of Alabama, Tuscaloosa, AL, 35487, USA

    Uzma Raja

Authors
  1. Uzma Raja
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Uzma Raja.

Additional information

Editor: Andreas Zeller

Rights and permissions

Reprints and permissions

About this article

Cite this article

Raja, U. All complaints are not created equal: text analysis of open source software defect reports. Empir Software Eng 18, 117–138 (2013). https://doi.org/10.1007/s10664-012-9197-9

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-012-9197-9

Keywords

Search

Navigation