Empirical Software Engineering

, Volume 19, Issue 3, pp 582–618 | Cite as

On using machine learning to automatically classify software applications into domain categories

  • Mario Linares-Vásquez
  • Collin McMillan
  • Denys Poshyvanyk
  • Mark Grechanik


Software repositories hold applications that are often categorized to improve the effectiveness of various maintenance tasks. Properly categorized applications allow stakeholders to identify requirements related to their applications and predict maintenance problems in software projects. Manual categorization is expensive, tedious, and laborious – this is why automatic categorization approaches are gaining widespread importance. Unfortunately, for different legal and organizational reasons, the applications’ source code is often not available, thus making it difficult to automatically categorize these applications. In this paper, we propose a novel approach in which we use Application Programming Interface (API) calls from third-party libraries for automatic categorization of software applications that use these API calls. Our approach is general since it enables different categorization algorithms to be applied to repositories that contain both source code and bytecode of applications, since API calls can be extracted from both the source code and byte-code. We compare our approach to a state-of-the-art approach that uses machine learning algorithms for software categorization, and conduct experiments on two large Java repositories: an open-source repository containing 3,286 projects and a closed-source repository with 745 applications, where the source code was not available. Our contribution is twofold: we propose a new approach that makes it possible to categorize software projects without any source code using a small number of API calls as attributes, and furthermore we carried out a comprehensive empirical evaluation of automatic categorization approaches.


Closed-source Open-source Software categorization Machine learning 



We are grateful to anonymous EMSE and ICSM’11 reviewers for their relevant detailed comments and suggestions, which helped us in significantly improving the initial version of this paper. We also would like to thank Marty White from the College of William and Mary for his useful suggestions that we implemented in this paper. This work is supported in part by NSF CCF-1016868, CCF-0916260, CCF-0916139, CCF-1017633, CCF-1218129 and Accenture. Any opinions, findings and conclusions expressed herein are the authors’ and do not necessarily reflect those of the sponsors.


  1. Aha DW, Kibler D, Albert MK (1991) Instance-based learning algorithms. Mach Learn 6:37–66Google Scholar
  2. Alpaydin E (2010) Introduction to machine learning, 2nd edn. The MIT Press, Cambridge, MassachusettsGoogle Scholar
  3. Antoniol G, Ayari K, Di Penta M, Khomh F, Guéhéneuc Y-G (2008) Is it a bug or an enhancement?: a text-based approach to classify change requests. 18th Conference of the Centre for Advanced Studies on Collaborative Research Meeting of Minds (CASCON’08), Ontario, Canada, pp 304–318Google Scholar
  4. Anvik J, Hiew L, Murphy GC (2006) Who should fix this bug? 28th International Conference on Software Engineering (ICSE’06), pp 361–370Google Scholar
  5. Anvik J, Murphy GC (2011) Reducing the effort of bug report triage: recommenders for development-oriented decisions. ACM Transactions on Software Engineering and Methods 20(3):10:1–10:35Google Scholar
  6. Bajracharya S, Ossher J, Lopes CV (2010) Leveraging usage similarity for effective retrieval of examples in code repositories. 18th International Symposium on the Foundations of Software Engineering (FSE’10)Google Scholar
  7. Bruno M, Canfora G, Di Penta M, Scognamiglio R (2005) An approach to support web service classification and annotation. IEEE International Conference on e-Technology, e-Commerce and e-Services (EEE’05), pp 138–143Google Scholar
  8. Bugde S, Nagappan N, Rajamani S, Ramalingam G (2008) Global software servicing: observational experiences at Microsoft. 2008 IEEE International Conference on Global Software Engineering (ICGSE’08), pp 182–191Google Scholar
  9. Cohen WW (1995) Fast effective rule induction. 12th International Conference on Machine Learning, pp 115–123Google Scholar
  10. Crammer K, Singer Y (2003) A family of additive online algorithms for category ranking. J Mach Learn Res 3(6):1025–1058MATHMathSciNetGoogle Scholar
  11. de Carvalho ACPLF, Freitas AA (2009) A tutorial on multi-label classification techniques. Foundations of Computational Intelligence. A. Abraham, A.-E. Hassanien and V. Snásel, Springer-Verlag, 5Google Scholar
  12. Demsar J (2006) Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res 7:1–30MATHMathSciNetGoogle Scholar
  13. Di Lucca GA, Di Penta M, Gradara S (2002) An approach to classify software maintenance requests. IEEE International Conference on Software Maintenance (ICSM’02), Montréal, Québec, Canada, pp 93–102Google Scholar
  14. Dit B, Guerrouj L, Poshyvanyk D, Antoniol G (2011) Can better identifier splitting techniques help feature location? 19th IEEE International Conference on Program Comprehension (ICPC’11), Kingston, Ontario, Canada, pp 11–20Google Scholar
  15. Dumitru H, Gibiec M, Hariri N, Cleland-Huang J, Mobasher B, Castro-Herrera C, Mirakhorli M (2011) On-demand feature recommendations derived from mining public product descriptions. 33rd IEEE/ACM International Conference on Software Engineering (ICSE’11), Honolulu, Hawaii, USA, pp 181–190Google Scholar
  16. Feng C-XJ, Yu Z-GS, Emanuel JT, Li P-G, Shao X-Y, Wang Z-H (2008) Threefold versus fivefold cross-validation and individual versus average data in predictive regression modelling of machining experimental data. Int J Comput Integrated Manuf 21(6):702–714CrossRefGoogle Scholar
  17. Frakes W, Prieto-Diaz R, Fox C (1998) DARE: domain analysis and reuse environment. Ann Software Eng 5:125–141CrossRefGoogle Scholar
  18. Grechanik M, Csallner C, Fu C, Xie Q (2010) Is data privacy always good for software testing? 21st IEEE International Symposium on Software Reliability Engineering (ISSRE’10), San Jose, California, USA, pp 368–377Google Scholar
  19. Grechanik M, Fu C, Xie Q, McMillan C, Poshyvanyk D, Cumby C (2010) A search engine for finding highly relevant applications. 32nd ACM/IEEE International Conference on Software Engineering (ICSE’10), Cape Town, South Africa, pp 475–484Google Scholar
  20. Grechanik M, McMillan C, DeFerrari L, Comi M, Crespi S, Poshyvanyk D, Fu C, Xie Q, Ghezzi C (2010) An empirical investigation into a large-scale java open source code repository. 4th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM ’10), Bolzano-Bozen, ItalyGoogle Scholar
  21. Grissom RJ, Kim JJ (2012) Effect sizes for research: univariate and multivariate applications, 2nd edn. Taylor & Francis, New YorkGoogle Scholar
  22. Guyon I, Elisseeff A (2003) An introduction to variable and feature selection. J Mach Learn Res 3:1157–1182MATHGoogle Scholar
  23. Hindle A, Germán DM, Godfrey MW, Holt RC (2009) Automatic Classification of Large Changes into Maintenance Categories. 17th IEEE International Conference on Program Comprehension (ICPC’09), Vancouver, Canada, pp 30–39Google Scholar
  24. Hsu C, Lin C (2002) A comparison of methods for multiclass support vector machines. IEEE Trans Neural Network 13(2):415–425CrossRefGoogle Scholar
  25. Ji S, Tang L, Yu S, Ye J (2008) Extracting shared subspace for multi-label classification. 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD’08), Las Vegas, Nevada, USA, pp 381–389Google Scholar
  26. Jones C (2010) Software engineering best practices. McGraw-Hill, New YorkGoogle Scholar
  27. Kang KC, Cohen S, Hess J, Novak W, Peterson A (1990) Feature-oriented domain analysis (FODA) feasibility study Pittsburgh, Pennsylvania, USA, Carnegie Mellon University, Software Engineering InstituteGoogle Scholar
  28. Kawaguchi S, Garg PK, Matsushita M, Inoue K (2003) Automatic categorization algorithm for evolvable software archive. 6th International Workshop on Principles of Software Evolution (IWPSE’03), pp 195–200Google Scholar
  29. Kawaguchi S, Garg PK, Matsushita M, Inoue K (2006) MUDABlue: an automatic categorization system for open source repositories. J Syst Software 79(7):939–953CrossRefGoogle Scholar
  30. Kelly MB, Alexander JS, Adams B, Hassan AE (2011) Recovering a balanced overview of topics in a software domain. 11th IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM’11), Williamsburg, VA, USA, to appearGoogle Scholar
  31. Leopold E, Kindermann J (2002) Text categorization with support vector machines. How to represent texts in input space? Mach Learn 46(1):423–444CrossRefMATHGoogle Scholar
  32. Lorena AC, De Carvalho ACPLF (2004) Comparing techniques for multiclass classification using binary SVM predictors. Third Mexican International Conference on Artificial Intelligence (MICAI’04), Mexico City, Mexico, Springer, pp 272–281Google Scholar
  33. McMillan C, Grechanik M, Poshyvanyk D, Xie Q, Fu C (2011) Portfolio: finding relevant functions and their usages. 33rd IEEE/ACM International Conference on Software Engineering (ICSE’11), Honolulu, Hawaii, USA, pp 111–120Google Scholar
  34. McMillan C, Linares-Vásquez M, Poshyvanyk D, Grechanik M (2011) Categorizing software applications for maintenance. 27th IEEE International Conference on Software Maintenance (ICSM’11), Williamsburg, Virginia, USA, pp 343–352Google Scholar
  35. Menzies T, Marcus A (2008) Automated severity assessment of software defect reports. IEEE International Conference on Software Maintenance (ICSM’08), Beijing, China, pp 346–355Google Scholar
  36. Poshyvanyk D, Grechanik M (2009) Creating and evolving software by searching, selecting and synthesizing relevant source code. 31st IEEE/ACM International Conference on Software Engineering (ICSE’09), Vancouver, British Columbia, Canada, pp 283–286Google Scholar
  37. Prieto-Diaz R (1990) Domain analysis: an introduction. ACM SIGSOFT Software Eng Notes 15(2):47–54CrossRefGoogle Scholar
  38. Ratiu D, Deissenboeck F (2006) How programs represent reality (and How They Don’t). 13th Working Conference on Reverse Engineering (WCRE’06), pp 83–92Google Scholar
  39. Ratiu D, Deissenboeck F (2007) From reality to programs and (not quite) back again. 15th IEEE International Conference on Program Comprehension (ICPC’07), Banff, Alberta, Canada, pp 91–102Google Scholar
  40. Sandhu PS, Singh J, Singh H (2007) Approaches for categorization of reusable software components. J Comput Sci 3(5):266–273CrossRefGoogle Scholar
  41. Schuler D, Dallmeir V, Lindig C (2007) A dynamic birthmark for java. Twenty-second IEEE/ACM International Conference on Automated software Engineering (ASE 2007), Atlanta, Georgia, USA, pp 274–283Google Scholar
  42. Sebastiani, F. (2002) Machine learning in automated text categorization. ACM Comput Surv 34(1):1–47Google Scholar
  43. Sim SE, Umarji M, Ratanotayanon S, Lopes CV (2011) How well do search engines support code retrieval on the web? ACM Trans Software Eng Meth (TOSEM) 21(1)Google Scholar
  44. Tian K, Revelle M, Poshyvanyk D (2009) Using latent Dirichlet allocation for automatic categorization of software. 6th IEEE Working Conference on Mining Software Repositories (MSR’09), Vancouver, British Columbia, Canada, pp 163–166Google Scholar
  45. Ugurel S, Krovetz R, Giles CL (2002) What’s the code ? Automatic classification of source code archives. Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD-2002), Edmontong, Alberta, Canada, pp 632–638Google Scholar
  46. Újházi B, Ferenc R, Poshyvanyk D, Gyimóthy T (2010) New conceptual coupling and cohesion metrics for object-oriented systems. 10th IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM’10), Timişoara, Romania, pp 33–42Google Scholar
  47. Weiss C, Premraj R, Zimmermann T, Zeller A (2007) How long will it take to fix this bug? 4th IEEE International Workshop on Mining Software Repositories (MSR’07), Minneapolis, MN, pp 1–8Google Scholar
  48. Zhang M-L, Zhou Z-H (2005) A k-nearest neighbor based algorithm for multi-label classification. IEEE International Conference on Granular Computing, Beijing, China, pp 718–721Google Scholar
  49. Zhang M-L, Zhou Z-H (2006) Multi-label neural networks with applications to functional genomics and text categorization. IEEE Trans Knowl Data Eng 18(10):1338–1351CrossRefGoogle Scholar
  50. Zimmermann T, Nagappan N, Gall H, Giger E, Murphy B (2009) Cross-project defect prediction: a large scale experiment on data vs. domain vs. process. ESEC/SIGSOFT FSE 2009, Amsterdam, The Netherlands, pp 91–100Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2012

Authors and Affiliations

  • Mario Linares-Vásquez
    • 1
  • Collin McMillan
    • 2
  • Denys Poshyvanyk
    • 1
  • Mark Grechanik
    • 3
  1. 1.The College of William and MaryWilliamsburgUSA
  2. 2.Universitry of Notre DameNotre DameUSA
  3. 3.University of Illinois at ChicagoChicagoUSA

Personalised recommendations