Requirements Engineering

, Volume 20, Issue 3, pp 281–300 | Cite as

On the role of semantics in automated requirements tracing

Original Article


In this paper, we investigate the potential benefits of utilizing natural language semantics in automated traceability link retrieval. In particular, we evaluate the performance of a wide spectrum of semantically enabled information retrieval methods in capturing and presenting requirements traceability links in software systems. Our objectives are to gain more operational insights into these methods and to provide practical guidelines for the design and development of effective requirements tracing and management tools. To achieve our research objectives, we conduct an experimental analysis using three datasets from various application domains. Results show that considering more semantic relations in traceability link retrieval does not necessarily lead to higher quality results. Instead, a more focused semantic support, that targets specific semantic relations, is expected to have a greater impact on the overall performance of tracing tools. In addition, our analysis shows that explicit semantic methods, that exploit local or domain-specific sources of knowledge, often achieve a more satisfactory performance than latent methods, or methods that derive semantics from external or general-purpose knowledge sources.


Information retrieval Traceability Semantics 


  1. 1.
    Abebe S, Tonella P (2010) Natural language parsing of program element names for concept extraction. In: International conference on program comprehension, pp 156–159Google Scholar
  2. 2.
    Ahn D, Jijkoun V, Mishne G, Mller K, de Rijke M, Schlobach S (2004) Using Wikipedia at the TREC QA track. In: Interactive poster and demonstration sessions, pp 73–76Google Scholar
  3. 3.
    Andrzejewski D, Mulhern A, Ben BL, Zhu X (2007) Statistical debugging using latent topic models. In: European conference on machine learning, pp 6–17Google Scholar
  4. 4.
    Anquetil N, Fourrier C, Lethbridge T (1999) Experiments with clustering as a software remodularization method. In: Working conference on reverse engineering, pp 235–255Google Scholar
  5. 5.
    Anquetil N, Lethbridge T (1998) Assessing the relevance of identifier names in a legacy software system. In: Conference of the centre for advanced studies on collaborative research, pp 4–14Google Scholar
  6. 6.
    Antoniol G, Caprile B, Potrich A, Tonella P (2000) Design-code traceability for object-oriented systems. Ann Softw Eng 9(1–4):35–58CrossRefGoogle Scholar
  7. 7.
    Aslam J, Yilmaz E, Pavlu V (2005) A geometric interpretation of r-precision and its correlation with average precision. In: Annual international ACM SIGIR conference on research and development in information retrieval, pp 573–574Google Scholar
  8. 8.
    Asuncion H, Asuncion A, Taylor R (2010) Software traceability with topic modeling. In: International conference on software engineering, pp 95–104Google Scholar
  9. 9.
    Baeza-Yates R, Ribeiro-Neto B (1999) Modern information retrieval. Addison-Wesley, Reading, MAGoogle Scholar
  10. 10.
    Ben Charrada E, Koziolek A, Glinz M (2012) Identifying outdated requirements based on source code changes. In: International requirements engineering conference, pp 61 –70Google Scholar
  11. 11.
    Biggers L, Bocovich C, Capshaw R, Eddy B, Etzkorn L, Kraft N (2012) Configuring latent Dirichlet allocation based feature location. Empir Softw Eng 1–36Google Scholar
  12. 12.
    Biggerstaff T, Mitbander B, Webster D (1994) Program understanding and the concept assignment problem. Commun ACM 37(5):72–82CrossRefGoogle Scholar
  13. 13.
    Binkley D, Lawrie D (2010) Information retrieval applications in software development. In: Computer technologies and information sciences, chap 37Google Scholar
  14. 14.
    Binkley D, Lawrie D (2011) Maintenance and evolution: information retrieval applications. In: Laplante PA (ed) Encyclopedia of software engineering, pp 454–463Google Scholar
  15. 15.
    Blei D, Ng A, Jordan M (2003) Latent Dirichlet allocation. J Mach Learn Res 3:993–1022MATHGoogle Scholar
  16. 16.
    Budanitsky A, Hirst G (2006) Evaluating wordnet-based measures of lexical semantic relatedness. Comput Linguist 32(1):13–47CrossRefMATHGoogle Scholar
  17. 17.
    Capobianco G, De Lucia A, Oliveto R, Panichella A, Panichella S (2013) Improving IR-based traceability recovery via noun-based indexing of software artifacts. J Softw Maint Evolut Res Pract 25(7):743–762CrossRefGoogle Scholar
  18. 18.
    Chen P, Lin S (2010) Automatic keyword prediction using Google similarity distance. Expert Syst Appl 37(3):1928–1938CrossRefGoogle Scholar
  19. 19.
    Chowdhury A, McCabe M (1998) Improving information retrieval systems using part of speech tagging. Technical report, ISR, Institute for Systems ResearchGoogle Scholar
  20. 20.
    Cilibrasi R, Vitanyi P (2007) The google similarity distance. IEEE Trans Knowl Data Eng 19(3):370–383CrossRefGoogle Scholar
  21. 21.
    Cleland-Huang J, Heimdahl M, Huffman-Hayes J, Lutz R, Mäder P (2012) Trace queries for safety requirements in high assurance systems. In: International conference on requirements engineering: foundation for software quality, pp 179–193Google Scholar
  22. 22.
    Cleland-Huang J, Settimi R, Duan C, Zou X (2005) Utilizing supporting evidence to improve dynamic requirements traceability. In: International conference on requirements engineering, pp 135–144Google Scholar
  23. 23.
    Cleland-Huang J, Settimi R, Romanova E (2007) Best practices for automated traceability. Computer 40(6):27–35CrossRefGoogle Scholar
  24. 24.
    De Lucia A, Fasano F, Oliveto R, Tortora G (2007) Recovering traceability links in software artifact management systems using information retrieval methods. ACM Trans Softw Eng Methodol 16(4):13–50CrossRefGoogle Scholar
  25. 25.
    De Lucia A, Oliveto R, Zurolo F., Di Penta M (2006) Improving comprehensibility of source code via traceability information: a controlled experiment. In: International conference on program comprehension, pp 317–326Google Scholar
  26. 26.
    Dean A, Voss D (1999) Design and analysis of experiments. Springer, BerlinCrossRefMATHGoogle Scholar
  27. 27.
    Deerwester S, Dumais S, Furnas G, Landauer T, Harshman R (1990) Indexing by latent semantic analysis. J Am Soc Inf Sci 41(6):391–407CrossRefGoogle Scholar
  28. 28.
    Deiβenböck F, Pizka M (2005) Concise and consistent naming. In: International workshop on program comprehension, pp 97–106Google Scholar
  29. 29.
    Dekhtyar A, Huffman-Hayes J, Antoniol G (2007) Benchmarks for traceability? In: International symposium on grand challenges in traceabilityGoogle Scholar
  30. 30.
    Demmel J, Kahan W (1990) Accurate singular values of bidiagonal matrices. J Sci Stat Comput 11(5):873–912MathSciNetCrossRefMATHGoogle Scholar
  31. 31.
    Demšar J (2006) Statistical comparisons of classifiers over multiple data sets. J Mach Learn Res 7:1–30MathSciNetMATHGoogle Scholar
  32. 32.
    Dumais S (1993) Lsi meets trec: a status report. In: Harman D (ed) The first text retrieval conference (TREC1), National Institute of Standards and Technology Special Publication, pp 137–152Google Scholar
  33. 33.
    Etzkorn L, Davis C (1995) An approach to object-oriented program understanding. In: IEEE workshop on program comprehension, pp 14–15Google Scholar
  34. 34.
    Evans D, Zhai C (1996) Noun-phrase analysis in unrestricted text for information retrieval. In: Annual meeting on association for computational linguistics, pp 17–24Google Scholar
  35. 35.
    Falleri J, Huchard M, Lafourcade M, Nebut C, Prince V, Dao M (2010) Automatic extraction of a wordnet-like identifier network from software. In: International conference on program comprehension, pp 4–13Google Scholar
  36. 36.
    Fang J, Guo L (2011) Calculation of relatedness by using search results. In: International workshop on intelligent systems and applications, pp 1–4Google Scholar
  37. 37.
    Feilkas M, Ratiu D, Jurgens E (2009) The loss of architectural knowledge during system evolution: an industrial case study. In: International conference on program comprehension, pp 188–197Google Scholar
  38. 38.
    Fellbaum C (1998) WordNet: an electronic lexical database. MIT Press, Cambridge, MAMATHGoogle Scholar
  39. 39.
    Finkelstein L, Gabrilovich E, Matias Y, Rivlin E, Solan Z, Wolfman G, Ruppin E (2002) Placing search in context: the concept revisited. ACM Trans Inf Syst 20(1):116–131CrossRefGoogle Scholar
  40. 40.
    Furnas G, Deerwester S, Dumais S, Landauer T, Xarshman R, Streeter L, Lochbaum K (1988) Information retrieval using a singular value decomposition model of latent semantic structure. In: Annual international ACM SIGIR conference on research and development in information retrieval, pp 465–480Google Scholar
  41. 41.
    Gabrilovich E, Markovitch S (2007) Computing semantic relatedness using Wikipedia-based explicit semantic analysis. In: International joint conference on artificial intelligence, pp 1606–1611Google Scholar
  42. 42.
    Gibiec M, Czauderna A, Cleland-Huang J (2010) Towards mining replacement queries for hard-to-retrieve traces. In: International conference on automated software engineering, pp 245–254Google Scholar
  43. 43.
    Gligorov R, Aleksovski Z, Kate W, Harmelen F (2007) Using Google distance to weight approximate ontology matches. In: International conference on world wide web, pp 767–776Google Scholar
  44. 44.
    Gotel O, Finkelstein A (1994) An analysis of the requirements traceability problem. In: International conference on requirements engineering, pp 94–101Google Scholar
  45. 45.
    Gotel O, Finkelstein A (1995) Contribution structures. In: International symposium on requirements engineering, pp 100–107Google Scholar
  46. 46.
    Gotel O, Morris S (2011) Out of the labyrinth: leveraging other disciplines for requirements traceability. In: IEEE international requirements engineering conference, pp 121–130Google Scholar
  47. 47.
    Grant S, Cordy J (2010) Estimating the optimal number of latent concepts in source code analysis. In: International working conference on source code analysis and manipulation, pp 65–74Google Scholar
  48. 48.
    Griffiths T (2004) Steyvers M finding scientific topics. In: The National Academy of Sciences, pp 5228–5235Google Scholar
  49. 49.
    Grzywaczewski A, Iqbal R (2012) Task-specific information retrieval systems for software engineers. J Comput Syst Sci 78(4):1204–1218MathSciNetCrossRefGoogle Scholar
  50. 50.
    Guo Y, Yang M, Wang J, Yang P, Li F (2009) An ontology-based approach for traceability recovery. In: International symposium on knowledge acquisition and modeling, pp 160–163Google Scholar
  51. 51.
    Han E, Karypis G (2000) Centroid-based document classification: Analysis and experimental results. In: European conference on principles of data mining and knowledge discovery, pp 424–431Google Scholar
  52. 52.
    Hata M, Homae F, Hagiwara H (2009) Semantic relatedness between words in each individual brain: an event-related potential study. Neurosci Lett 501(2):72–77CrossRefGoogle Scholar
  53. 53.
    Hazen T (2010) Direct and latent modeling techniques for computing spoken document similarity. In: Spoken language technology workshop, pp 366–371Google Scholar
  54. 54.
    Hindle A, Bird C, Zimmermann T, Nagappan N (2012) Relating requirements to implementation via topic analysis: do topics extracted from requirements make sense to managers and developers? In: International conference on software maintenance, pp 243–252Google Scholar
  55. 55.
    Huffman-Hayes J, Dekhtyar A, Osborne J (2003) Improving requirements tracing via information retrieval. In: International conference on requirements engineering, pp 138–147Google Scholar
  56. 56.
    Huffman-Hayes J, Dekhtyar A, Sundaram S (2006) Advancing candidate link generation for requirements tracing: the study of methods. IEEE Trans Softw Eng 32(1):4–19CrossRefGoogle Scholar
  57. 57.
    Jurafsky D, Martin J (2000) Speech and language processing. Prentice Hall, Englewood Cliffs NJGoogle Scholar
  58. 58.
    Kit L, Man C, Baniassad E (2006) On finding duplication and near-duplication in large software systems. In: Annual ACM SIGPLAN conference on object-oriented programming systems, languages, and applications, pp 383–396Google Scholar
  59. 59.
    von Knethen A (2002) Automatic change support based on a trace model. In: International workshop on traceability in emerging forms of software engineeringGoogle Scholar
  60. 60.
    Kuhn A, Ducasse S, Gírba T (2007) Semantic clustering: Identifying topics in source code. Inf Softw Technol 49(3):230–243CrossRefGoogle Scholar
  61. 61.
    Lawrie D, Feild H, Binkley D (2007) Extracting meaning from abbreviated identifiers. In: International working conference on source code analysis and manipulation, pp 213–222Google Scholar
  62. 62.
    Lehman M (1984) On understanding laws, evolution, and conservation in the large-program life cycle. J Syst Softw 1(3):213–221Google Scholar
  63. 63.
    Linstead E, Rigor P, Bajracharya S, Lopes C, Baldi P (2007) Mining concepts from code with probabilistic topic models. In: International conference on automated software engineering, pp 461–464Google Scholar
  64. 64.
    Lioma C, Blanco R (2009) Part of speech based term weighting for information retrieval. In: Advances in information retrieval, pp 412–423Google Scholar
  65. 65.
    Liu Y, Poshyvanyk D, Ferenc R, Gyimóthy T, Chrisochoides N (2009) Modelling class cohesion as mixtures of latent topics. In: International conference on software maintenance, pp 233–242Google Scholar
  66. 66.
    Lormans M (2006) Can lsi help reconstructing requirements traceability in design and test. In: European conference on software maintenance and reengineering, pp 47–56Google Scholar
  67. 67.
    Luisa M, Mariangela F, Pierluigi I (2004) Market research for requirements analysis using linguistic tools. Requir Eng 9(1):40–56CrossRefGoogle Scholar
  68. 68.
    Mahmoud A, Niu N (2010) Using semantics-enabled information retrieval in requirements tracing: An ongoing experimental investigation. In: Annual computer software and applications conference, pp 246–247Google Scholar
  69. 69.
    Mahmoud A, Niu N (2011) Source code indexing for automated tracing. In: International workshop on traceability in emerging forms of software engineering, pp 3–9Google Scholar
  70. 70.
    Mahmoud A, Niu N, Xu S (2012) A semantic relatedness approach for traceability link recovery. In: International conference on program comprehension, pp 183–192Google Scholar
  71. 71.
    Maletic J, Marcus A (2000) Using latent semantic analysis to identify similarities in source code to support program understanding. In: International conference on tools with artificial intelligence, pp 46–53Google Scholar
  72. 72.
    Marcus A, Maletic J (2001) Identification of high-level concept clones in source code. In: International conference on automated software engineering, pp 107–114Google Scholar
  73. 73.
    Marcus A, Maletic J (2003) Recovering documentation-to-source-code traceability links using latent semantic indexing. In: International conference on software engineering, pp 125–135Google Scholar
  74. 74.
    Maskeri G, Sarkar S, Heafield K (2008) Mining business topics in source code using Latent Dirichlet allocation. In: ISEC, pp 113–120Google Scholar
  75. 75.
    Meneely A, Smith B, Williams L (2012) iTrust electronic health care system: a case study, chap. software and systems traceability. Springer, BerlinGoogle Scholar
  76. 76.
    Milne D, Witten I (2008) An effective, low-cost measure of semantic relatedness obtained from wikipedia links. In: AAAI workshop on wikipedia and artificial intelligence, pp 25–30Google Scholar
  77. 77.
    Nallapati R, Cohen W, Lafferty J (2007) Parallelized variational EM for Latent Dirichlet allocation: an experimental evaluation of speed and scalability. In: International conference on data mining workshops, pp 349–354Google Scholar
  78. 78.
    Niu N, Easterbrook S (2008) Extracting and modeling product line functional requirements. In: International requirements engineering conference, pp 155–164Google Scholar
  79. 79.
    Niu N, Mahmoud A (2012) Enhancing candidate link generation for requirements tracing: the cluster hypothesis revisited. In: IEEE international requirements engineering conference, pp 81–90Google Scholar
  80. 80.
    Nuseibeh B, Easterbrook S (2000) Requirements engineering: a roadmap. In: Conference on the future of software engineering, pp 35–46Google Scholar
  81. 81.
    Oliveto R, Gethers M, Poshyvanyk D, De Lucia A (2010) On the equivalence of information retrieval methods for automated traceability link recovery. In: International conference on program comprehension, pp 68–71Google Scholar
  82. 82.
    Panichella A, Dit B, Oliveto R, Di Penta M, Poshyvanyk D, De Lucia A (2013) How to effectively use topic models for software engineering tasks? An approach based on genetic algorithms. In: International conference on software engineering, pp 522–531Google Scholar
  83. 83.
    Patwardhan S, Banerjee S, Pedersen T (2005) Senserelate::targetword: a generalized framework for word sense disambiguation. In: Interactive poster and demonstration sessions, pp 73–76Google Scholar
  84. 84.
    Pedersen J, Silverstein C, Vogt C (2000) Verity at trec-6: out-of-the-box and beyond. Inf Process Manage 36(1):187–204CrossRefGoogle Scholar
  85. 85.
    Porteous I, Newman D, Ihler A, Asuncion A, Smyth P, Welling M (2008) Fast collapsed gibbs sampling for latent Dirichlet allocation. In: ACM SIGKDD international conference on knowledge discovery and data mining, pp 569–577Google Scholar
  86. 86.
    Porter F (1997) An algorithm for suffix stripping. Morgan Kaufmann Publishers Inc., Los Altos, CA, pp 313–316Google Scholar
  87. 87.
    Poshyvanyk D, gal Guhneuc Y, Marcus A (2007) Feature location using probabilistic ranking of methods based on execution scenarios and information retrieval. IEEE Trans Softw Eng 33(6):420–432CrossRefGoogle Scholar
  88. 88.
    Pucher M (2007) Wordnet-based semantic relatedness measures in automatic speech recognition for meetings. In: Annual meeting of the ACL on interactive poster and demonstration sessions, pp 129–132Google Scholar
  89. 89.
    Ramesh B, Jarke M (2001) Towards reference models for requirements traceability. IEEE Trans Softw Eng 27(1):58–93CrossRefGoogle Scholar
  90. 90.
    Rao A, Lu A, Meier E, Ahmed S, Pliske D (2000) Query processing in trec-6. Inf Process Manage 36(1):179–186CrossRefGoogle Scholar
  91. 91.
    Rosario B (2000) Latent semantic indexing: an overview. INFOSYS 240 Spring Paper, University of California, BerkeleyGoogle Scholar
  92. 92.
    Salton G, Wong A, Yang C (1975) A vector space model for automatic indexing. Commun ACM 18(11):613–620CrossRefMATHGoogle Scholar
  93. 93.
    Sarukkai R (2002) Foundations of web technology. The Springer International Series in Engineering and Computer Science, New York, pp 106–108CrossRefGoogle Scholar
  94. 94.
    Schmid H (1994) Probabilistic part-of-speech tagging using decision trees. In: International conference on new methods in language processing, pp 44–49Google Scholar
  95. 95.
    Settimi R, Cleland-Huang J, Ben Khadra O, Mody J, Lukasik W, DePalma C (2004) Supporting software evolution through dynamically retrieving traces to uml artifacts. In: International workshop on the principles of software evolution, pp 49–54Google Scholar
  96. 96.
    Shepherd D, Fry Z, Hill E, Pollock L, Vijay-Shanker K (2007) Using natural language program analysis to locate and understand action-oriented concerns. In: International conference on aspect-oriented software development, pp 212–224Google Scholar
  97. 97.
    Sorg P, Cimiano P (2012) Exploiting wikipedia for cross-lingual and multilingual information retrieval. Data Knowl Eng 74(0):26–45CrossRefGoogle Scholar
  98. 98.
    Spanoudakis G, Zisman A (2004) Software traceability: a roadmap. Handb Softw Eng Knowl Eng 3:395–428Google Scholar
  99. 99.
    Strube M, Ponzetto S (2006) Wikirelate! computing semantic relatedness using wikipedia. In: National conference on artificial intelligence, pp 1419–1424Google Scholar
  100. 100.
    Sundaram S, Huffman-Hayes J, Dekhtyar A, Holbrook E (2010) Assessing traceability of software engineering artifacts. Requir Eng J 15(3):313–335CrossRefGoogle Scholar
  101. 101.
    Teh Y, Jordan M, Beal M, Blei D (2006) Hierarchical Dirichlet processes. J Am Stat Assoc 101(476):1566–1581MathSciNetCrossRefMATHGoogle Scholar
  102. 102.
    Teufel S (2007) An overview of evaluation methods in trec ad hoc information retrieval and trec question answering. In: Dybkjaer L, Hemsen H, Minker W (eds) Evaluation of text and speech systems, pp 163–186Google Scholar
  103. 103.
    Thomas S, Adams B, Hassan A, Blostein D (2010) Validating the use of topic models for software evolution. In: IEEE working conference on source code analysis and manipulation, pp 55–64Google Scholar
  104. 104.
    Tsatsaronis G, Varlamis I, Vazirgiannis M (2010) Text relatedness based on a word thesaurus. Commun ACM 37(1):1–40MATHGoogle Scholar
  105. 105.
    Wei X, Croft B (2006) LDA-based document models for ad-hoc retrieval. In: ACM SIGIR, pp 178–185Google Scholar
  106. 106.
    Wong S, Ziarko W, Raghavan V, Wong P (2012) On modeling of information retrieval concepts in vector spaces. In: ACM transactions database systems, pp 299–321Google Scholar
  107. 107.
    Zhai K, Boyd-Graber J, Asadi N, Alkhouja M (2012) Mr. LDA: a flexible large scale topic modeling package using variational inference in MapReduce. In: International conference on world wide web, pp 879–888Google Scholar
  108. 108.
    Zhang Y, Witte R, Rilling J, Haarslev V (2006) An ontology-based approach for traceability recovery. In: International workshop on metamodels, schemas, grammars, and ontologies for reverse engineering, pp 36–43Google Scholar

Copyright information

© Springer-Verlag London 2014

Authors and Affiliations

  1. 1.Department of Computer Science and EngineeringMississippi State UniversityMississippi StateUSA

Personalised recommendations