Empirical Software Engineering

, Volume 21, Issue 1, pp 17–42 | Cite as

An empirical study of the textual similarity between source code and source code summaries

  • Paul W. McBurney
  • Collin McMillan


Source code documentation often contains summaries of source code written by authors. Recently, automatic source code summarization tools have emerged that generate summaries without requiring author intervention. These summaries are designed for readers to be able to understand the high-level concepts of the source code. Unfortunately, there is no agreed upon understanding of what makes up a “good summary.” This paper presents an empirical study examining summaries of source code written by authors, readers, and automatic source code summarization tools. This empirical study examines the textual similarity between source code and summaries of source code using Short Text Semantic Similarity metrics. We found that readers use source code in their summaries more than authors do. Additionally, this study finds that accuracy of a human written summary can be estimated by the textual similarity of that summary to the source code.


Source code summarization Documentation Textual similarity Automatic documentation generation 



The authors would like to thank David Croft for furnishing an LSS implementation. We would also like to thank the 23 participants of the case studies on which this paper is based for their time and efforts.


  1. Biggersta TJ, Mitbander BG, Webster D (1993) The concept assignment problem in program understanding. In: Proceedings of the 15th international conference on soft-ware engineering, ICSE ’93. IEEE Computer Society Press, Los Alamitos, pp 482–498.
  2. Brandt J, Dontcheva M, Weskamp M, Klemmer SR (2010) Example-centric programming: integrating web search into the development environment. In: Proceedings ofthe 28th international conference on human factors in computing systems, CHI ’10. ACM, New York, pp 513–522. doi: 10.1145/1753326.1753402
  3. Burden H, Heldal R (2011) Natural language generation from class diagrams. In: Proceedings of the 8th international workshop on model-driven engineering, verication and validation, MoDeVVa. ACM, New York, pp 8:1–8:8. doi: 10.1145/2095654.2095665
  4. Buse RP, Weimer WR (2008) Automatic documentation inference for exceptions. In: Proceedings of the 2008 international symposium on software testing and analysis, ISSTA ’08. ACM, New York, pp 273–282. doi: 10.1145/1390630.1390664
  5. Buse RP, Weimer WR (2010) Automatically documenting program changes. In: Proceedings of the IEEE/ACM international conference on Automated software engineering, ASE ’10. ACM, New York, pp 33–42. doi: 10.1145/1858996.1859005
  6. Croft D, Coupland S, Shell J, Brown S (2013) A fast and efficient semantic short text similarity metric. In: Proceedings of the 13th UK workshop on computational intelligence, UKCI ’13, pp 221–227Google Scholar
  7. Davison JW, Mancl DM, Opdyke WF (2000) Understanding and addressing the essential costs of evolving systems. Bell Labs Tech J 5(2):44–54Google Scholar
  8. de Souza SCB, Anquetil N, de Oliveira KM (2005) A study of the documentation essential to software maintenance. In: Proceedings of the 23rd annual international conference on design of communication: documenting & designing for pervasive information, SIG-DOC ’05. ACM, New York, pp 68–75. doi: 10.1145/1085313.1085331
  9. Eddy B, Robinson J, Kraft N, Carver J (2013) Evaluating source code summarization techniques: replication and expansion. In: Proceedings of the 21st international conference on program comprehension, ICPC ’13Google Scholar
  10. Fluri B, Wursch M, Gall HC (2007) Do code and comments co-evolve? On the relation between source code and comment changes. In: Proceedings of the 14th working conference on reverse engineering, WCRE ’07, IEEE Computer Society, Washington, DC, pp 70–79. doi: 10.1109/WCRE.2007.21
  11. Forward A, Lethbridge TC (2002) The relevance of software documentation, tools and technologies: a survey. In: Proceedings of the 2002 ACM symposium on document engineering, DocEng ’02. ACM, New York, pp 26–33. doi: 10.1145/585058.585065
  12. Guzzi A (2012) Documenting and sharing knowledge about code. In: Proceedings of the 2012 international conference on software engineering, ICSE 2012. IEEE Press, Piscataway, pp 1535–1538.
  13. Haiduc S, Aponte J, Moreno L, Marcus A (2010) On the use of automated text summarization techniques for summarizing source code. In: Proceedings of the 2010 17th working conference on reverse engineering, WCRE ’10. IEEE Computer Society, Washington, DC, pp 35–44. doi: 10.1109/WCRE.2010.13
  14. Holmes R, Walker RJ (2013) Systematizing pragmatic software reuse. ACM Trans Softw Eng Methodol 21(4):20:1–20:44. doi: 10.1145/2377656.2377657 Google Scholar
  15. Ibrahim WM, Bettenburg N, Adams B, Hassan AE (2012) Controversy corner: on the relationship between comment update practices and software bugs. J Syst Softw 85(10):2293–2304. doi: 10.1016/j.jss.2011.09.019 CrossRefGoogle Scholar
  16. Kajko-Mattsson M (2005) A survey of documentation practice within corrective maintenance. Empir Softw Eng 10(1):31–55. doi: 10.1023/ CrossRefGoogle Scholar
  17. Khamis N, Witte R, Rilling J (2010) Automatic quality assessment of source code comments: the JavadocMiner. In: Hopfe CJ, Rezgui Y, Métais EM, Preece AD, Li H (eds) 15th international conference on applications of natural language to information systems (NLDB 2010). Lecture Notes in Computer Science (LNCS), vol 6177/2010. Springer, Cardi, pp 68–79. doi: 10.1007/978-3-642-13881_27.
  18. Ko AJ, Myers BA (2005) A framework and methodology for studying the causes of software errors in programming systems. J Vis Lang Comput 16(12):41–84.  10.1016/j.jvlc.2004.08.003. CrossRefGoogle Scholar
  19. Ko AJ, Myers BA, Coblenz MJ, Aung HH (2006) An exploratory study of how developers seek, relate, and collect relevant information during software maintenance tasks. IEEE Trans Softw Eng 32(12):971–987. doi: 10.1109/TSE.2006.116 CrossRefGoogle Scholar
  20. Kotonya G, Lock S, Mariani J (2008) Opportunistic reuse: lessons from scrapheap software development. In: Proceedings of the 11th international symposium on component-based software engineering, CBSE ’08. Springer, Berlin, pp 302–309. doi: 10.1007/978-3-540-87891-9_20
  21. Kramer D (1999) Api documentation from source code comments: a case study of javadoc. In: Proceedings of the 17th annual international conference on computer documentation, SIGDOC ’99. ACM, New York, pp 147–153. doi: 10.1145/318372.318577
  22. Lakhotia A (1993) Understanding someone else’s code: analysis of experiences. J Syst Softw 23(3):269–275. doi: 10.1016/0164-1212(93)90101-3 CrossRefMathSciNetGoogle Scholar
  23. Lethbridge T, Singer J, Forward A (2003) How software engineers use documentation: the state of the practice. IEEE Softw 20(6):35–39. doi: 10.1109/MS.2003.1241364 CrossRefGoogle Scholar
  24. Li Y, Bandar ZA, McLean D (2003) An approach for measuring semantic similarity between words using multiple information sources. IEEE Trans Knowl Data Eng 15(4):871–882. doi: 10.1109/TKDE.2003.1209005 CrossRefGoogle Scholar
  25. Li Y, McLean D, Bandar ZA, O’Shea JD, Crockett K (2006) Sentence similarity based on semantic nets and corpus statistics. IEEE Trans Knowl Data Eng 18(8):1138–1150. doi: 10.1109/TKDE.2006.130 CrossRefGoogle Scholar
  26. Liblit B, Begel A, Sweeser E (2006) Cognitive perspectives on the role of naming in computer programs. In: Proceedings of the 18th annual psychology of programming workshop. Psychology of Programming Interest Group, SussexGoogle Scholar
  27. Littman DC, Pinto J, Letovsky S, Soloway E (1987) Mental models and software maintenance. J Syst Softw 7(4):341–355. doi: 10.1016/0164-1212(87)90033-1. CrossRefGoogle Scholar
  28. Mani S, Catherine R, Sinha VS, Dubey A (2012) Ausum: approach for unsupervised bug report summarization. In: Proceedings of the ACM SIGSOFT 20th international symposium on the foundations of software engineering, FSE ’12. ACM, New York, pp 11:1–11:11. doi: 10.1145/2393596.2393607
  29. Mann H, Whitney D (1947) On a test of whether one of two random variables is stochastically larger than the other. Ann Math Stat 50–60Google Scholar
  30. McBurney PW, McMillan C (2014) Automatic documentation generation via source code summarization of method context. To appear in proceedings of the 22nd international conference on program comprehension, ICPC ’14. New York, pp 1–10Google Scholar
  31. Miller GA (1995) Wordnet: a lexical database for english. Commun ACM 38:39–41CrossRefGoogle Scholar
  32. Moreno L, Aponte J, Giriprasad S, Marcus A, Pollock L, Vijay-Shanker K (2013) Automatic generation of natural language summaries for java classes. In: Proceedings of the 21st international conference on program comprehension, ICPC ’13Google Scholar
  33. Roehm T, Tiarks R, Koschke R, Maalej W (2012) How do professional developers comprehend software?. In: Proceedings of the 2012 international conference on software engineering, ICSE 2012. IEEE Press, Piscataway, pp 255–265.
  34. Singer J, Lethbridge T, Vinson N, Anquetil N (1997) An examination of software engineering work practices. In: Proceedings of the 1997 conference of the centre for advanced studies on collaborative research, CASCON ’97. IBM Press, p 21.
  35. Sridhara G, Hill E, Muppaneni D, Pollock L, Vijay-Shanker K (2010) Towards automatically generating summary comments for java methods. In: Proceedings of the IEEE/ACM international conference on automated software engineering, ASE ’10. ACM, New York, pp 43–52. doi: 10.1145/1858996.1859006
  36. Sridhara G, Pollock L, Vijay-Shanker K (2011a) Automatically detecting and describing high level actions within methods. In: Proceedings of the 33rd international conference on software engineering, ICSE ’11. ACM, New York, pp 101–110. doi: 10.1145/1985793.1985808
  37. Sridhara G, Pollock L, Vijay-Shanker K (2011b) Generating parameter comments and integrating with method summaries. In: Proceedings of the 2011 IEEE 19th international conference on program comprehension, ICPC ’11. IEEE Computer Society, Washington, DC, pp 71–80. doi: 10.1109/ICPC.2011.28
  38. Steidl D, Hummel B, Juergens E (2013) Quality analysis of source code comments. In: Proceedings of the 21st international conference on program comprehension, ICPC ’13Google Scholar
  39. Stylos J, Myers BA (2006) Mica: a web-search tool for finding api components and examples. In: Proceedings of the visual languages and human-centric computing, VL-HCC ’06. IEEE Computer Society, Washington, DC, pp 195–202. doi: 10.1109/VLHCC.2006.32

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  1. 1.Department of Computer Science and EngineeringUniversity of Notre DameNotre DameUSA

Personalised recommendations