Empirical Software Engineering

, Volume 22, Issue 3, pp 1063–1102 | Cite as

Eye movements in software traceability link recovery

  • Bonita Sharif
  • John Meinken
  • Timothy Shaffer
  • Huzefa Kagdi


Information Retrieval (IR) approaches, such as Latent Semantic Indexing (LSI) and Vector Space Model (VSM), are commonly applied to recover software traceability links. Recently, an approach based on developers’ eye gazes was proposed to retrieve traceability links. This paper presents a comparative study on IR and eye-gaze based approaches. In addition, it reports on the possibility of using eye gaze links as an alternative benchmark in comparison to commits. The study conducted asked developers to perform bug-localization tasks on the open source subject system JabRef. The iTrace environment, which is an eye tracking enabled Eclipse plugin, was used to collect eye gaze data. During the data collection phase, an eye tracker was used to gather the source code entities (SCE’s), developers looked at while solving these tasks. We present an algorithm that uses the collected gaze dataset to produce candidate traceability links related to the tasks. In the evaluation phase, we compared the results of our algorithm with the results of an IR technique, in two different contexts. In the first context, precision and recall metric values are reported for both IR and eye gaze approaches based on commits. In the second context, another set of developers were asked to rate the candidate links from each of the two techniques in terms of how useful they were in fixing the bugs. The eye gaze approach outperforms standard LSI and VSM approaches and reports a 55 % precision and 67 % recall on average for all tasks when compared to how the developers actually fixed the bug. In the second context, the usefulness results show that links generated by our algorithm were considered to be significantly more useful (to fix the bug) than those of the IR technique in a majority of tasks. We discuss the implications of this radically different method of deriving traceability links. Techniques for feature location/bug localization are commonly evaluated on benchmarks formed from commits as is done in the evaluation phase of this study. Although, commits are a reasonable source, they only capture entities that were eventually changed to fix a bug or resolve a feature. We investigate another type of benchmark based on eye tracking data, namely links generated from the bug-localization tasks given to the developers in the data collection phase. The source code entities relevant to subjected bugs recommended from IR methods are evaluated on both commits and links generated from eye gaze. The results of the benchmarking phase show that the use of eye tracking could form an effective (complementary) benchmark and add another interesting perspective in the evaluation of bug-localization techniques.


Eye-tracking Software traceability link recovery Continuous traceability Eye-gaze benchmark 


  1. Alhindawi N, Dragan N, Collard M, Maletic J (2013) Improving feature location by enhancing source code with stereotypes. In: 29Th IEEE international conference on software maintenance, ICSM 2013Google Scholar
  2. Ali N, Sharafi Z, Guéhéneuc Y. G., Antoniol G (2012) An empirical study on requirements traceability using eye-tracking. In: ICSM, pp 191 – 200Google Scholar
  3. Antoniol G, Canfora G, Casazza G, De Lucia A, Merlo E (2002) Recovering traceability links between code and documentation. IEEE Trans Softw Eng 28(10):970–983CrossRefGoogle Scholar
  4. Asuncion H, Asuncion A, Taylor RN (2010) Software traceability with topic modeling. In: 32Nd ACM/IEEE international conference on software engineering, vol. 1, pp. 95–104. ACMGoogle Scholar
  5. Asuncion H, Francois F, Taylor RN (2007) An end-to-end industrial software traceability tool. In: 6Th ESEC/FSE, pp 115–124Google Scholar
  6. Bavota G, Lucia AD, Oliveto R, Panichella A, Ricci F, Tortora G (2013) The role of artefact corpus in lsi-based traceability recovery. 2013 7th International Workshop on Traceability in Emerging Forms of Software Engineering (TEFSE) 0, 83–89. doi: 10.1109/TEFSE.2013.6620160
  7. Binkley D, Lawrie D (2011) Information retrieval applications in software maintenance and evolution. In: Encyclopedia of software engineeringGoogle Scholar
  8. Capobianco G, De Lucia A, Oliveto R, Panichella A, Panichella S (2009) On the role of nouns in ir-based traceability recovery. In: 17Th IEEE international conference on program comprehension, pp 148–157Google Scholar
  9. Cleland-Huang J, Czauderna A, Dekhtyar A, Gotel O, Hayes JH, Keenan E, Leach G, Maletic JI, Poshyvanyk D, Shin Y, Zisman A, Antoniol G, Berenbach B, Egyed A, Maeder P (2011) Grand challenges, benchmarks, and tracelab: developing infrastructure for the software traceability community. In: 6th TEFSE. Panel, pp 17–23Google Scholar
  10. Cleland-Huang J, Gotel O, Zisman A (2012) Software and systems traceability Spring-VerlagGoogle Scholar
  11. (2007) Center of excellence for software traceability.
  12. Davis APH (1993) Software requirements: Objects, Functions and States. Prentice HallGoogle Scholar
  13. De Lucia A, Fasano F, Oliveto R, Tortora G (2006) Can information retrieval techniques effectively support traceability link recovery?. In: 14Th IEEE international conference on program comprehension (ICPC’06), pp 307–316Google Scholar
  14. De Lucia A, Fasano F, Oliveto R, Tortora G (2007) Recovering traceability links in software artefact management systems using information retrieval methods. ACM (TOSEM) 16(4):13CrossRefGoogle Scholar
  15. De Lucia A, Oliveto R, Tortora G (2009) Assessing ir-based traceability recovery tools through controlled experiments. Empirical Softw Engg 14(1):57–92. doi: 10.1007/s10664-008-9090-8 CrossRefGoogle Scholar
  16. Deerwester S, Dumais ST, Furnas GW, Landauer TK, Harshman R (1990) Indexing by latent semantic analysis. J Am Soc Inf Sci 41(6):391–407CrossRefGoogle Scholar
  17. Dit B, Holtzhauer A, Poshyvanyk D, Kagdi H (2013) A dataset from change history to support evaluation of software maintenance tasks. In: Proceedings of the 10th Working Conference on Mining Software Repositories, MSR ’13, pp. 131–134. IEEE Press, Piscataway, NJ, USA.
  18. Dit B, Moritz E, Poshyvanyk D (2012) A tracelab-based solution for creating, conducting, and sharing feature location experiments. In: Program Comprehension (ICPC), 2012 IEEE 20th International Conference on. doi: 10.1109/ICPC.2012.6240489, pp 203–208
  19. Dit B., Revelle M., Gethers M., Poshyvanyk D. (2013) Feature location in source code: a taxonomy and survey. J Softw: Evol Process 25(1):53–95. doi: 10.1002/smr.567 Google Scholar
  20. Dit B, Revelle M, Gethers M, Poshyvanyk D (2013) Feature location in source code: a taxonomy and survey. J Soft Maint Evo.: Res Pract 25(1):53–95CrossRefGoogle Scholar
  21. Duchowski AT (2003) Eye tracking methodology: Theory and practice. Springer-Verlag, LondonCrossRefzbMATHGoogle Scholar
  22. Fritz T, Begel A, Müller S. C., Yigit-Elliott S, Züger M. (2014) Using psycho-physiological measures to assess task difficulty in software development. In: Proceedings of the 36th international conference on software engineering, ICSE 2014. ACM, New York, pp 402–413Google Scholar
  23. Gethers M, Oliveto R, Poshyvanyk D, De Lucia A (2011) On integrating orthogonal information retrieval methods to improve traceability link recovery. In: 27Th IEEE international conference on software maintenance (ICSM’11), pp 133–142Google Scholar
  24. Gotel O, Cleland-Huang J, Hayes JH, Zisman A, Egyed A, Grünbacher P., Antoniol G (2012) The quest for ubiquity: A roadmap for software and systems traceability research. In: 2012 20th IEEE International Requirements Engineering Conference (RE). doi: 10.1109/RE.2012.6345841, pp 71–80
  25. Hill E, Bacchelli A, Binkley D, Dit B, Lawrie D, Oliveto R (2013) Which feature location technique is better?. In: Proceedings of the 2013 IEEE international conference on software maintenance, ICSM’13. IEEE Computer Society, Washington, pp 408–411Google Scholar
  26. Jinshui Wang Xin Peng ZX, Zhao W (2011) An exploratory study of feature location process: Distinct phases, recurring patterns, and elementary actions. In: Proceedings of the 27th IEEE international conference on software maintenance, ICSM’11, pp 213–222Google Scholar
  27. Keenan E, Czauderna A, Leach G, Cleland-Huang J, Shin Y (2012) Tracelab: An experimental workbench for equipping researchers to innovate, synthesize, and comparatively evaluate traceability solutions. In: ICSE, p 4Google Scholar
  28. Kersten M, Murphy G (2005) Mylar: a degree-of-interest model for ides. In: 4th international conference on aspect-oriented software development, pp 159–168Google Scholar
  29. Kevic K, Walters B, Shaffer T, Sharif B, Fritz T, Shepherd DC (2015) Tracing software developers eyes and interactions for change tasks Proceedings of the 10th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software EngineeringGoogle Scholar
  30. Lin J, Lin CC, Cleland-Huang J, Settimi R, Amaya J, Bedford G, Berenbach B, Khadra OB, Duan C, Zou X (2006) Poirot: A distributed tool supporting enterprise-wide automated traceability. 2014 IEEE 22nd International Requirements Engineering Conference (RE) 0, 363–364. doi: 10.1109/RE.2006.48
  31. Lucia AD, Oliveto R, Sgueglia P (2006) Incremental approach and user feedbacks: a silver bullet for traceability recovery. In: 2006 22nd IEEE International Conference on Software Maintenance. doi: 10.1109/ICSM.2006.32, pp 299–309
  32. Lukins SK, Kraft NA, Etzkorn LH (2010) Bug localization using latent dirichlet allocation. Inf Softw Technol 52(9):972–990CrossRefGoogle Scholar
  33. Mader P, Gotel O, Philippow I (2009) Motivation matters in the traceability trenches. In: RE 2009, pp 143–148Google Scholar
  34. Manning C, Raghavan P, Schütze H. (2008) Introduction to information retrieval cambridge university pressGoogle Scholar
  35. Marcus A, Maletic JI (2003) Recovering documentation-to-source-code traceability links using latent semantic indexing. In: 25Th IEEE/ACM international conference on software engineering (ICSE’03), pp 125–137Google Scholar
  36. Oliveto R, Gethers M, Poshyvanyk D, De Lucia A (2010) On the equivalence of information retrieval methods for automated traceability link recovery. In: 18Th IEEE international conference on program comprehension (ICPC), pp 68–71Google Scholar
  37. Qusef A, Oliveto R, De Lucia A (2010) Recovering traceability links between unit tests and classes under test: an improved method. In: IEEE International conference on software maintenance, pp 1–10Google Scholar
  38. Ramesh B, Jarke M (2001) Towards reference models for requirements traceability. IEEE TSE 27(1):58–93Google Scholar
  39. Rao S, Kak A (2011) Retrieval from software libraries for bug localization: a comparative study of generic and composite text models. In: Proceedings of the 8th working conference on mining software repositories, MSR ’11. ACM, New York, pp 43–52Google Scholar
  40. Rodeghero P, McMillan C, McBurney PW, Bosch N, D’Mello S (2014) Improving automated source code summarization via an eye-tracking study of programmers. In: ICSE 2014. ACM, New York, pp 390–401Google Scholar
  41. Scanniello G, Marcus A (2011) Clustering support for static concept location in source code. In: Program Comprehension (ICPC), 2011 IEEE 19th International Conference on. doi: 10.1109/ICPC.2011.13, pp 1–10
  42. Scanniello G, Marcus A, Pascale D (2015) Link analysis algorithms for static concept location: An empirical assessment. Empir Softw Eng 20(6):1666–1720. doi: 10.1007/s10664-014-9327-7 CrossRefGoogle Scholar
  43. Shaffer T, Wise JL, Walters B, Müller S.C., Falcone M, Sharif B (2015) itrace: Enabling eye tracking on software artifacts within the ide to support software engineering tasks. In: Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2015, pp. 954–957. ACM, New York, NY, USA. doi: 10.1145/2786805.2803188
  44. Sharafi Z, Soh Z, Guéhéneuc Y. G. (2015) A systematic literature review on the usage of eye-tracking in software engineering Elsevier Journal of Information and Software Technology (IST)Google Scholar
  45. Sharif B, Kagdi H (2011) On the use of eye tracking in software traceability. In: 6Th TEFSE, pp 67–70Google Scholar
  46. Walters B, Falcone M, Shibble A, Sharif B (2013) Towards an eye-tracking enabled ide for software traceability tasks. In: 7Th TEFSE, pp 51–54Google Scholar
  47. Walters B, Shaffer T, Sharif B, Kagdi H (2014) Capturing software traceability links from developers’ eye gazes. In: Proceedings of the 22Nd International Conference on Program Comprehension, ICPC 2014, pp. 201–204. ACM, New York, NY, USA. doi: 10.1145/2597008.2597795
  48. Wang J, Peng X, Xing Z, Zhao W (2013) Improving feature location practice with multi-faceted interactive exploration. In: Proceedings of the 2013 International Conference on Software Engineering, ICSE ’13, pp. 762–771. IEEE Press, Piscataway, NJ, USA.
  49. Xing Z, Xue Y, Jarzabek S (2013) A large scale linux-kernel based benchmark for feature location research. In: Proceedings of the 2013 International Conference on Software Engineering, ICSE ’13, pp. 1311–1314. IEEE Press, Piscataway, NJ, USA.
  50. Zhou J, Zhang H, Lo D (2012) Where should the bugs be fixed? - more accurate information retrieval-based bug localization based on bug reports. In: Proceedings of the 34th international conference on software engineering, ICSE ’12. IEEE Press, Piscataway, pp 14–24Google Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  1. 1.Department of Computer Science and Information SystemsYoungstown State UniversityYoungstownUSA
  2. 2.Department of Electrical Engineering and Computer ScienceWichita State UniversityWichitaUSA

Personalised recommendations