Experimenting with information retrieval methods in the recovery of feature-code SPL traces

  • Tassio ValeEmail author
  • Eduardo Santana de Almeida



The information retrieval research provides alternatives to recover traces from existing software information. In the software product line (SPL) engineering context, further research on information retrieval methods is required to explore the existence of products’ source code and support the SPL adoption by providing traceability information.


This work evaluates the methods’ retrieval quality targeting the extraction of feature-code trace links in a set of variant-rich software development projects.


We propose a research method comprising two experiments with five information retrieval methods (Classic vector, Latent semantic indexing, Neural network, Extended boolean and BM25), applied to forty-one projects. The SPLTrac suite automates the research operation, using the information retrieval methods results as input to perform a five-step quantitative data analysis procedure based on parametric and non-parametric statistical techniques. The quality measurement is expressed by four dependent variables: precision, recall, F-measure and execution time.


With a homogeneous result for execution time between methods, there are discrepancies for the other metrics. While Extended boolean presents the best results for precision and F-measure, BM25 provides the greatest recall results.


Such evidence indicates there is not a dominant method in terms of retrieval quality, since they need further improvements to achieve better performance in all quality perspectives.


Software product lines Software traceability Information retrieval Controlled-experiment 



  1. Apel S, Beyer D (2011) Feature cohesion in software product lines: an exploratory study. In: Proceedings of the international conference on software engineering, ser. ICSE ’11. ACM, New York, pp 421–430Google Scholar
  2. Apel S, Kastner C, Lengauer C (2013) Language-independent and automated software composition: the featurehouse experience. IEEE Trans Softw Eng 39(1):63–79CrossRefGoogle Scholar
  3. Baeza-Yates RA, Ribeiro-Neto BA (2011) Modern information retrieval - the concepts and technology behind search, 2nd edn. Pearson Education Ltd, HarlowGoogle Scholar
  4. Basili VR, Rombach HD (1988) The tame project: towards improvement-oriented software environments. IEEE Trans Softw Eng 14(6):758–773CrossRefGoogle Scholar
  5. Batory D, Sarvela JN, Rauschmayer A (2004) Scaling step-wise refinement. IEEE Trans Softw Eng 30(6):355–371CrossRefGoogle Scholar
  6. Bayer J, Widen T (2002) Introducing traceability to product lines. In: Revised Papers from the 4th international workshop on software product-family engineering, ser. PFE ’01. Springer, London, pp 409–416CrossRefGoogle Scholar
  7. Berger T, Rublack R, Nair D, Atlee JM, Becker M, Czarnecki K, Wasowski A (2013) A survey of variability modeling in industrial practice. In: Proceedings of the international workshop on variability modelling of software-intensive systems, ser. VaMoS ’13. ACM, New York, pp 7:1–7:8Google Scholar
  8. Borg M, Runeson P (2013) IR in software traceability: from a bird’s eye view. In: Proceedings of the IIEEE international symposium on empirical software engineering and measurement, ser. ESEM ’13, pp 243–246Google Scholar
  9. Borg M, Runeson P, Ardö A (2014) Recovering from a decade: a systematic mapping of information retrieval approaches to software traceability. Empir Softw Eng 19(6):1565–1616CrossRefGoogle Scholar
  10. Capilla R, Bosch J, Kang K (2013) Systems and software variability management, concepts, tools and experiences. Springer-Verlag Berlin Heidelberg Inc, BerlinCrossRefGoogle Scholar
  11. Cleland-Huang J, Gotel O, Zisman A (2012) Software and systems traceability. Springer Publishing Company, Berlin. IncorporatedCrossRefGoogle Scholar
  12. Cleland-Huang J, Gotel OCZ, Huffman Hayes J, Mäder P., Zisman A (2014) Software traceability: trends and future directions. In: Proceedings of the on future of software engineering, ser. FOSE ’14. ACM, New York, pp 55–69Google Scholar
  13. Clements P, Northrop L (2001) Software product lines: practices and patterns. Addison-Wesley Longman Publishing Co. Inc, BostonGoogle Scholar
  14. Deelstra S, Sinnema M, Bosch J (2009) Variability assessment in software product families. Inf Softw Technol 51(1):195–218CrossRefGoogle Scholar
  15. DeGroot MH, Schervish MJ (2002) Probability and statistics, 3rd edn. Addison-Wesley, ReadingGoogle Scholar
  16. Furnas GW, Deerwester S, Dumais ST, Landauer TK, Harshman RA, Streeter LA, Lochbaum KE (1988) Information retrieval using a singular value decomposition model of latent semantic structure. In: Proceedings of the annual international ACM SIGIR conference on research and development in information retrieval, ser. SIGIR ’88. ACM, New York, pp 465–480Google Scholar
  17. Gelman A (2005) Analysis of variance? Why it is more important than ever. Ann Stat 33(1):1–53MathSciNetCrossRefGoogle Scholar
  18. Ossher H, Tarr P (2002) Multi-dimensional separation of concerns and the hyperspace approach. In: Akşit M. (ed) Software architectures and component technology. Springer, Boston, pp 293–323CrossRefGoogle Scholar
  19. Hayes JH, Dekhtyar A, Sundaram SK (2006) Advancing candidate link generation for requirements tracing: the study of methods. IEEE Trans Softw Eng 32 (1):4–19CrossRefGoogle Scholar
  20. Hedges L, Olkin I, Hedges L (1985) Statistical methods for meta-analysis. Academic PressGoogle Scholar
  21. Hochberg Y (1988) A sharper bonferroni procedure for multiple tests of significance. Biometrika 75:800–802MathSciNetCrossRefGoogle Scholar
  22. Jirapanthong W, Zisman A (2005) Supporting product line development through traceability. In: Proceedings of the Asia-Pacific software engineering conference, ser. APSEC ’05. IEEE Computer Society, Washington, pp 506–514Google Scholar
  23. Kampenes VB, Dybå T, Hannay JE, Sjøberg DIK (2007) Systematic review: a systematic review of effect size in software engineering experiments. Inf Softw Technol 49(11–12):1073–1086CrossRefGoogle Scholar
  24. Kang K, Cohen SG, Hess JA, Novak WE, Peterson AS (1990) Feature-oriented domain analysis (foda) feasibility study, Carnegie-Mellon University Software Engineering Institute, Tech. Rep.Google Scholar
  25. Kastner C (2007) CIDE: decomposing legacy applications into features. In: Proceedings of the international software product line conference. Second Volume (Workshops), ser, SPLC ’07, pp 149–150Google Scholar
  26. Keenan E, Czauderna A, Leach G, Cleland-Huang J, Shin Y, Moritz E, Gethers M, Poshyvanyk D, Maletic J, Hayes JH, Dekhtyar A, Manukian D, Hossein S, Hearn D (2012) TraceLab: an experimental workbench for equipping researchers to innovate, synthesize, and comparatively evaluate traceability solutions. In: 34th International Conference on Software Engineering, ser. ICSE ’12Google Scholar
  27. Kernighan BW, Ritchie DM (1988) The C programming language. Prentice Hall Press, Upper Saddle RiverzbMATHGoogle Scholar
  28. Klock S, Gethers M, Dit B, Poshyvanyk D (2011) Traceclipse: an eclipse plug-in for traceability link recovery and management. In: Proceedings of the international workshop on traceability in emerging forms of software engineering, ser. TEFSE ’11. ACM, New York, pp 24–30Google Scholar
  29. Kolesnikov S, Roth J, Apel S (2014) On the relation between internal and external feature interactions in feature-oriented product lines: a case study. In: Proceedings of the 6th international workshop on feature-oriented software development, ser. FOSD ’14. ACM, New York, pp 1–8Google Scholar
  30. Kong L, Li J, Li Y, Yang Y, Wang Q (2009) A requirement traceability refinement method based on relevance feedback. In: Proceedings of the international conference on software engineering and knowledge engineering, ser. SEKE ’09Google Scholar
  31. Kruskal WH, Wallis WA (1952) Use of ranks in one-criterion variance analysis. J Am Stat Assoc 47(260):583–621CrossRefGoogle Scholar
  32. Liebig J, Apel S, Lengauer C, Kästner C., Schulze M (2010) An analysis of the variability in forty preprocessor-based software product lines. In: Proceedings of the 32Nd ACM/IEEE international conference on software engineering - volume 1, ser. ICSE ’10. ACM, New York, pp 105–114Google Scholar
  33. Lindvall M, Sandahl K (1996) Practical implications of traceability. Software: Practice and Experience 26(10):1161–1180Google Scholar
  34. Lopez-Herrejon RE, Linsbauer L, Galindo JA, Parejo JA, Benavides D, Segura S, Egyed A (2015) An assessment of search-based techniques for reverse engineering feature models. J Syst Softw 103:353– 369CrossRefGoogle Scholar
  35. Mann HB, Whitney DR (1947) On a test of whether one of two random variables is stochastically larger than the other. The Annals of Mathematical Statistics 18(1):50–60MathSciNetCrossRefGoogle Scholar
  36. Metzger A, Pohl K (2014) Software product line engineering and variability management: achievements and challenges. In: Proceedings of the on future of software engineering, ser. FOSE ’14. ACM, New York, pp 70–84Google Scholar
  37. Montgomery DC, Peck EA, Vining GG (2007) Introduction to linear regression analysis, solutions manual (wiley series in probability and statistics). Wiley-Interscience, New YorkGoogle Scholar
  38. Parvathy AG, Vasudevan BG, Balakrishnan R (2008) A comparative study of document correlation techniques for traceability analysis. In: Proceedings of the international conference on enterprise information systems, ser. ICEIS ’08, pp 64–69Google Scholar
  39. Pohl K, Böckle G., van der Linden FJ (2005) Software product line engineering: foundations, principles and techniques. Springer-Verlag New York Inc., SecaucusCrossRefGoogle Scholar
  40. Riebisch M, Philippow I (2001) Evolution of product lines using traceability. In: Proceedings of the workshop on engineering complex object-oriented systems for evolutionGoogle Scholar
  41. Robertson S, Zaragoza H, Taylor M (2004) Simple bm25 extension to multiple weighted fields. In: Proceedings of the ACM international conference on information and knowledge management, ser. CIKM ’04. ACM, New York, pp 42–49Google Scholar
  42. Robertson SE, Walker S (1994) Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval. In: Proceedings of the annual international ACM SIGIR conference on research and development in information retrieval, ser. SIGIR ’94. Springer-Verlag New York, Inc, New York, pp 232–241CrossRefGoogle Scholar
  43. Robertson SE, Walker S, Jones S, Hancock-Beaulieu M, Gatford M (1994) Okapi at TREC-3. In: Proceedings of the text retrieval conference, ser. TREC-3. pp 109–126Google Scholar
  44. Salton G, Fox EA, Wu H (1983) Extended boolean information retrieval. Commun ACM 26(11):1022–1036MathSciNetCrossRefGoogle Scholar
  45. Salton G, Wong A, Yang CS (1975) A vector space model for automatic indexing. Commun ACM 18(11):613–620CrossRefGoogle Scholar
  46. Salton G, Yang CS (1973) On the specification of term values in automatic indexing. J Doc 29(4):351–372CrossRefGoogle Scholar
  47. Shapiro SS, Wilk MB (1965) An analysis of variance test for normality (complete samples). Biometrika 52(3/4):591–611MathSciNetCrossRefGoogle Scholar
  48. Shull FJ, Carver JC, Vegas S, Juristo N (2008) The role of replications in empirical software engineering. Empir Softw Eng 13(2):211–218CrossRefGoogle Scholar
  49. Sobernig S, Apel S, Kolesnikov S, Siegmund N (2016) Quantifying structural attributes of system decompositions in 28 feature-oriented software product lines. Empir Softw Eng 21(4):1670–1705CrossRefGoogle Scholar
  50. Spanoudakis G, Zisman A (2004) Software traceability: a roadmap. In: Handbook of software engineering and knowledge engineering. World Scientific Publishing, pp 395–428Google Scholar
  51. Tukey JW (1949) Comparing individual means in the analysis of variance. Biometrics 5(2):99–114MathSciNetCrossRefGoogle Scholar
  52. Vale T, de Almeida ES, Alves V, Kulesza U, Niu N, de Lima R (2017) Software product lines traceability: a systematic mapping study. Inf Softw Technol 84 (C):1–18CrossRefGoogle Scholar
  53. Welch BL (1947) The generalization of ‘student’s problem when several different population variances are involved. Biometrika 34(1/2):28–35MathSciNetCrossRefGoogle Scholar
  54. Wilcoxon F (1945) Individual comparisons by ranking methods. Biom Bull 1 (6):80–83CrossRefGoogle Scholar
  55. Wilkinson R, Hingston P (1991) Using the cosine measure in a neural network for document retrieval. In: Proceedings of the annual international ACM SIGIR conference on research and development in information retrieval, ser. SIGIR ’91. ACM, New York, pp 202–210Google Scholar
  56. Wohlin C, Runeson P, Höst M, Ohlsson MC, Regnell B, Wesslén A (2012) Experimentation in software engineering. Springer Publishing Company, Berlin. IncorporatedCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Center of Exact Sciences and TechnologyFederal University of Recôncavo da BahiaSalvadorBrazil
  2. 2.Computer Science DepartmentFederal University of BahiaSalvadorBrazil

Personalised recommendations