Mining reading patterns from eye-tracking data: method and demonstration

  • Constantina Ioannou
  • Indira NurdianiEmail author
  • Andrea Burattin
  • Barbara Weber
Special Section Paper


Understanding how developers interact with different software artifacts when performing comprehension tasks has a potential to improve developers’ productivity. In this paper, we propose a method to analyze eye-tracking data using process mining to find distinct reading patterns of how developers interacted with the different artifacts. To validate our approach, we conducted an exploratory study using eye-tracking involving 11 participants. We applied our method to investigate how developers interact with different artifacts during domain and code understanding tasks. To contextualize the reading patterns and to better understand the perceived benefits and challenges participants associated with the different artifacts and their choice of reading patterns, we complemented the eye-tracking data with the data obtained from think aloud. The study used behavior-driven development, a development practice that is increasingly used in Agile software development contexts, as a setting. The study shows that our method can be used to explore developers’ behavior at an aggregated level and identify behavioral patterns at varying levels of granularity.


Process mining Eye-tracking Reading patterns Source code Behavior-driven development 


Supplementary material


  1. 1.
    Abrahamsson, P., Salo, O., Ronkainen, J., Warsta, J.: Agile software development methods—review and analysis, Technical Report 478. VTT Publications (2002)Google Scholar
  2. 2.
    Binamungu, L.P., Embury, S.M., Konstantinou, N.: Maintaining behaviour driven development specifications: challenges and opportunities. In: IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER), pp. 175–184 (2018)Google Scholar
  3. 3.
    Blinman, S., Cockburn, A.: Program comprehension: investigating the effects of naming style and documentation. In: Proceedings of the 6th Australasian Conference on User Interface, vol. 40, AUIC ’05, pp. 73–78. Australian Computer Society, Inc, Darlinghurst (2005)Google Scholar
  4. 4.
    Boehm, B.: Software engineering economics. IEEE Trans. Softw. Eng. 10, 4–21 (1984)CrossRefGoogle Scholar
  5. 5.
    Brooks, R.: Towards a theory of the comprehension of computer programs. Int. J. Man-Mach. Stud. 18(6), 543–554 (1983)CrossRefGoogle Scholar
  6. 6.
    Busjahn, T., Schulte, C., Busjahn, A.: Analysis of code reading to gain more insight in program comprehension. In: Proceedings of the 11th Koli Calling International Conference on Computing Education Research, Koli Calling ’11, pp. 1–9. ACM, New York (2011)Google Scholar
  7. 7.
    Cios, K.J., Pedrycz, W., Swiniarski, R.W.: Data Mining Methods for Knowledge Discovery. Springer, Boston (1998)CrossRefGoogle Scholar
  8. 8.
    Crosby, M.E., Stelovsky, J.: How do we read algorithms? A case study. Computer 23(1), 25–35 (1990)CrossRefGoogle Scholar
  9. 9.
    de Souza, P.L., do Prado, A.F., de Souza, W.L., dos Santos Forghieri Pereira, S.M., Pires, L.F.: Combining behaviour-driven development with scrum for software development in the education domain. In: ICEIS (2), pp. 449–458. SciTePress (2017)Google Scholar
  10. 10.
    Fritz, T., Begel, A., Müller, S.C., Yigit-Elliott, S., Züger, M.: Using psycho-physiological measures to assess task difficulty in software development. In: Proceedings of the 36th International Conference on Software Engineering, pp. 402–413. ACM, New York (2014)Google Scholar
  11. 11.
    Gemino, A., Wand, Y.: A framework for empirical evaluation of conceptual modeling techniques. Requir. Eng. 9(4), 248–260 (2004)CrossRefGoogle Scholar
  12. 12.
    Goldberg, J.H., Kotval, X.P.: Computer interface evaluation using eye movements: methods and constructs. Int. J. Ind. Ergon. 24(6), 631–645 (1999)CrossRefGoogle Scholar
  13. 13.
    Graziotin, D., Wang, X., Abrahamsson, P.: Do feelings matter? On the correlation of affects and the self-assessed productivity in software engineering. J. Softw. Evol. Process 27(7), 467–487 (2015)CrossRefGoogle Scholar
  14. 14.
    Gulden, J., Burattin, A., Andaloussi, A.A., Weber, B.: From analytical purposes to data visualizations: a decision process guided by a conceptual framework and eye tracking. Softw. Syst. Model. (2019)Google Scholar
  15. 15.
    Holmqvist, K., Nyström, M., Andersson, R., Dewhurst, R., Jarodzka, H., Van de Weijer, J.: Eye Tracking: A Comprehensive Guide to Methods and Measures. OUP, Oxford (2011)Google Scholar
  16. 16.
    IEEE Task Force on Process Mining.: Process Mining Manifesto. Business Process Management Workshops, pp. 169–194 (2011)Google Scholar
  17. 17.
    Ioannou, C., Burattin, A., Weber, B.: Mining developers’ workflows from IDE usage. In: Matulevičius, R., Dijkman, R. (eds.) Advanced Information Systems Engineering Workshops, pp. 167–179. Springer, Cham (2018)CrossRefGoogle Scholar
  18. 18.
    Johnny, S.: The Coding Manual for Qualitative Researchers. SAGE Publications Limited, Thousand Oaks (2012)Google Scholar
  19. 19.
    Kevic, K., Walters, B.M., Shaffer, T.R., Sharif, B., Shepherd, D.C., Fritz, T.: Tracing software developers’ eyes and interactions for change tasks. In Proceedings of the 10th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2015, pp. 202–213. ACM, New York (2015)Google Scholar
  20. 20.
    Kim, J., Hahn, J., Hahn, H.: How do we understand a system with (so) many diagrams? Cognitive integration processes in diagrammatic reasoning. Inf. Syst. Res. 11(3), 284–303 (2000)CrossRefGoogle Scholar
  21. 21.
    Kindler, E., Rubin, V., Schäfer, W.: Incremental workflow mining based on document versioning information. In: Software Process Workshop, volume 3840 of LNCS, pp. 287–301. Springer, Berlin (2005)CrossRefGoogle Scholar
  22. 22.
    King, T.M., Nunez, G., Santiago, D., Cando, A., Mack, C.: Legend: An Agile DSL Toolset for Web Acceptance Testing. pp. 409–412 (2014)Google Scholar
  23. 23.
    Lawrie, D., Morrell, C., Feild, H., Binkley, D.: Effective identifier names for comprehension and memory. Innov. Syst. Softw. Eng. 3(4), 303–318 (2007)CrossRefGoogle Scholar
  24. 24.
    Leemans, M., van der Aalst, W.M.P.: Process mining in software systems: discovering real-life business transactions and process models from distributed systems. In: Proceedings of MODELS, pp. 44–53 (2015)Google Scholar
  25. 25.
    Leemans, M., van der Aalst, W.M.P., van den Brand, M.G..J.: Recursion aware modeling and discovery for hierarchical software event log analysis (extended). CoRR, abs/1710.09323 (2017)Google Scholar
  26. 26.
    Leemans, Sander J. J., Fahland, Dirk, van der Aalst, Wil M. P.: Discovering block-structured process models from event logs - a constructive approach. In Application and Theory of Petri Nets and Concurrency. Springer, (2013)Google Scholar
  27. 27.
    Lenberg, P., Feldt, R., Wallgren, L.: Behavioral software engineering: a definition and systematic literature review. J. Syst. Softw. 107, 15–37 (2015)CrossRefGoogle Scholar
  28. 28.
    Letovsky, S.: Cognitive processes in program comprehension. J. Syst. Softw. 7(4), 325–339 (1987)CrossRefGoogle Scholar
  29. 29.
    Liu, C., van Dongen, B.F., Assy, N., Aalst, W.M.P.: Software architectural model discovery from execution data. In: 13th International Conference on Evaluation of Novel Approaches to Software Engineering, 03 (2018)Google Scholar
  30. 30.
    Liu, C., van Dongen, B., Assy, N., van der Aalst, W.M.P.: Component behavior discovery from software execution data. In: IEEE Symposium Series on Computational Intelligence (SSCI), pp. 1–8, (2016)Google Scholar
  31. 31.
    Mller, S.C., Fritz, T.: Stakeholders’ information needs for artifacts and their dependencies in a real world context. In: IEEE International Conference on Software Maintenance, pp 290–299 (2013)Google Scholar
  32. 32.
    Nancy, P.: Empirical studies of programmers: Second workshop. In: Comprehension Strategies in Programming, pp. 100–113. Ablex Publishing Corp., Norwood (1987)Google Scholar
  33. 33.
    North, D.: Introducing BDD (2006). Last Accessed October 2018
  34. 34.
    Rubin, V., Günther, C.W., van der Aalst, W.M.P., Kindler, E., van Dongen, B.F., Schäfer, W.: Process mining framework for software processes. In Proceedings of ICSP 2007, pp. 169–181. Springer (2007)Google Scholar
  35. 35.
    Nurdiani, I., Börstler, J., Fricker, S.: The impacts of agile and lean practices on project constraints: a tertiary study. J. Syst. Softw. 119, 162–183 (2016)CrossRefGoogle Scholar
  36. 36.
    Oliveira, G., Marczak, S.: On the understanding of bdd scenarios’ quality: Preliminary practitioners’ opinions. In: Kamsties, E., Horkoff, J., Dalpiaz, F. (eds.) Requirements Engineering: Foundation for Software Quality, pp. 290–296. Springer, Berlin (2018)CrossRefGoogle Scholar
  37. 37.
    Salviulo, F., Scanniello, G.: Dealing with identifiers and comments in source code comprehension and maintenance: results from an ethnographically-informed study with students and professionals. In: Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering, EASE ’14, pp. 48:1–48:10. ACM, New York (2014)Google Scholar
  38. 38.
    Schröter, I., Krüger, J., Siegmund, J., Leich, T.: Comprehending studies on program comprehension. In: IEEE/ACM 25th International Conference on Program Comprehension (ICPC), pp. 308–311 (2017)Google Scholar
  39. 39.
    Shaffer, T.R., Wise, J.L., Walters, B.M., Müller, S.C., Falcone, M., Sharif, B.: itrace: Enabling eye tracking on software artifacts within the ide to support software engineering tasks. In: Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2015, pp. 954–957. ACM, New York (2015)Google Scholar
  40. 40.
    Sharafi, Z., Soh, Z., Guéhéneuc, Y., Antoniol, G.: Women and men—different but equal: on the impact of identifier style on source code reading. In: 20th IEEE International Conference on Program Comprehension (ICPC), pp. 27–36 (2012)Google Scholar
  41. 41.
    Sharafi, Z., Soh, Z., Guéhéneuc, Y.-G.: A systematic literature review on the usage of eye-tracking in software engineering. Inf. Softw. Technol. 67, 79–107 (2015)CrossRefGoogle Scholar
  42. 42.
    Siau, K.Leng: Empirical studies in information modeling: Interpretation of the object relationship. PhD thesis, University of British Columbia (1996)Google Scholar
  43. 43.
    Smart, J.F.: BDD in Action. Manning Publications, New York (2014)Google Scholar
  44. 44.
    IEEE Computer Society, Bourque, P., Fairley, R.E.: Guide to the Software Engineering Body of Knowledge (SWEBOK(R)): Version 3.0., 3rd edn. IEEE Computer Society Press, Los Alamitos (2014)Google Scholar
  45. 45.
    Solis, C., Wang, X.: A study of the characteristics of behaviour driven development. In: 37th EUROMICRO Conference on Software Engineering and Advanced Applications, pp. 383–387 (2011)Google Scholar
  46. 46.
    Soloway, E., Adelson, B., Ehrlich, K.: Knowledge and processes in the comprehension of computer programs. Nat. Expert. 129, 152 (1988)Google Scholar
  47. 47.
    van der Aalst, W.: Big software on the run: In vivo software analytics based on process mining (keynote). In: Proceedings of ICSSP 2015, pp. 1–5. ACM (2015)Google Scholar
  48. 48.
    van der Aalst, W.M.P.: Process Mining: Data Science in Action. Springer, Berlin (2016)CrossRefGoogle Scholar
  49. 49.
    Vans, A.M., von Mayrhauser, A.: Program comprehension during software maintenance and evolution. Computer 28(8), 44–55 (1995)CrossRefGoogle Scholar
  50. 50.
    Version One.: 12th Annual State of Agile™ Report, 2017. Last Accessed December 2018Google Scholar
  51. 51.
    Vessey, I., Conger, S.A.: Requirements Specification: Learning Object, Process, and Data Methodologies. Communications of the ACM (1994)Google Scholar
  52. 52.
    Walters, B., Shaffer, T., Sharif, B., Kagdi, H.: Capturing software traceability links from developers’ eye gazes. In: Proceedings of the 22nd International Conference on Program Comprehension, ICPC 2014, pp. 201–204. ACM, New York (2014)Google Scholar
  53. 53.
    Wang, Y., Wagner, S.: Combining stpa and bdd for safety analysis and verification in agile development: a controlled experiment. In: Garbajosa, J., Wang, X., Aguiar, A. (eds.) Agile Processes in Software Engineering and Extreme Programming, pp. 37–53. Springer, Cham (2018)CrossRefGoogle Scholar
  54. 54.
    Wieringa, R.J.: Design Science Methodology for Information Systems and Software Engineering. Springer, Berlin (2014). CrossRefGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  1. 1.DTU Compute, Software and Process EngineeringTechnical University of DenmarkKgs. LyngbyDenmark
  2. 2.SDU Software Engineering, The Mærsk Mc-Kinney Møller InstituteUniversity of Southern DenmarkOdense MDenmark
  3. 3.University of St. GallenSt. GallenSwitzerland

Personalised recommendations