Skip to main content
SpringerLink
Log in

Mining reading patterns from eye-tracking data: method and demonstration

  • Special Section Paper
  • Published:
Software and Systems Modeling Aims and scope Submit manuscript

Abstract

Understanding how developers interact with different software artifacts when performing comprehension tasks has a potential to improve developers’ productivity. In this paper, we propose a method to analyze eye-tracking data using process mining to find distinct reading patterns of how developers interacted with the different artifacts. To validate our approach, we conducted an exploratory study using eye-tracking involving 11 participants. We applied our method to investigate how developers interact with different artifacts during domain and code understanding tasks. To contextualize the reading patterns and to better understand the perceived benefits and challenges participants associated with the different artifacts and their choice of reading patterns, we complemented the eye-tracking data with the data obtained from think aloud. The study used behavior-driven development, a development practice that is increasingly used in Agile software development contexts, as a setting. The study shows that our method can be used to explore developers’ behavior at an aggregated level and identify behavioral patterns at varying levels of granularity.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Notes

  1. See http://www.i-trace.org/.

  2. See http://www.eclipse.org/.

  3. See http://fluxicon.com/disco/.

  4. See https://tobiigaming.com/product/tobii-eye-tracker-4c/.

  5. The new version of the iTrace plugin is available on GitHub, at https://github.com/CIoann/cucitraceSetup.

  6. See https://atlasti.com/.

References

  1. Abrahamsson, P., Salo, O., Ronkainen, J., Warsta, J.: Agile software development methods—review and analysis, Technical Report 478. VTT Publications (2002)

  2. Binamungu, L.P., Embury, S.M., Konstantinou, N.: Maintaining behaviour driven development specifications: challenges and opportunities. In: IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER), pp. 175–184 (2018)

  3. Blinman, S., Cockburn, A.: Program comprehension: investigating the effects of naming style and documentation. In: Proceedings of the 6th Australasian Conference on User Interface, vol. 40, AUIC ’05, pp. 73–78. Australian Computer Society, Inc, Darlinghurst (2005)

  4. Boehm, B.: Software engineering economics. IEEE Trans. Softw. Eng. 10, 4–21 (1984)

    Article  Google Scholar 

  5. Brooks, R.: Towards a theory of the comprehension of computer programs. Int. J. Man-Mach. Stud. 18(6), 543–554 (1983)

    Article  Google Scholar 

  6. Busjahn, T., Schulte, C., Busjahn, A.: Analysis of code reading to gain more insight in program comprehension. In: Proceedings of the 11th Koli Calling International Conference on Computing Education Research, Koli Calling ’11, pp. 1–9. ACM, New York (2011)

  7. Cios, K.J., Pedrycz, W., Swiniarski, R.W.: Data Mining Methods for Knowledge Discovery. Springer, Boston (1998)

    Book  Google Scholar 

  8. Crosby, M.E., Stelovsky, J.: How do we read algorithms? A case study. Computer 23(1), 25–35 (1990)

    Article  Google Scholar 

  9. de Souza, P.L., do Prado, A.F., de Souza, W.L., dos Santos Forghieri Pereira, S.M., Pires, L.F.: Combining behaviour-driven development with scrum for software development in the education domain. In: ICEIS (2), pp. 449–458. SciTePress (2017)

  10. Fritz, T., Begel, A., Müller, S.C., Yigit-Elliott, S., Züger, M.: Using psycho-physiological measures to assess task difficulty in software development. In: Proceedings of the 36th International Conference on Software Engineering, pp. 402–413. ACM, New York (2014)

  11. Gemino, A., Wand, Y.: A framework for empirical evaluation of conceptual modeling techniques. Requir. Eng. 9(4), 248–260 (2004)

    Article  Google Scholar 

  12. Goldberg, J.H., Kotval, X.P.: Computer interface evaluation using eye movements: methods and constructs. Int. J. Ind. Ergon. 24(6), 631–645 (1999)

    Article  Google Scholar 

  13. Graziotin, D., Wang, X., Abrahamsson, P.: Do feelings matter? On the correlation of affects and the self-assessed productivity in software engineering. J. Softw. Evol. Process 27(7), 467–487 (2015)

    Article  Google Scholar 

  14. Gulden, J., Burattin, A., Andaloussi, A.A., Weber, B.: From analytical purposes to data visualizations: a decision process guided by a conceptual framework and eye tracking. Softw. Syst. Model. (2019)

  15. Holmqvist, K., Nyström, M., Andersson, R., Dewhurst, R., Jarodzka, H., Van de Weijer, J.: Eye Tracking: A Comprehensive Guide to Methods and Measures. OUP, Oxford (2011)

    Google Scholar 

  16. IEEE Task Force on Process Mining.: Process Mining Manifesto. Business Process Management Workshops, pp. 169–194 (2011)

  17. Ioannou, C., Burattin, A., Weber, B.: Mining developers’ workflows from IDE usage. In: Matulevičius, R., Dijkman, R. (eds.) Advanced Information Systems Engineering Workshops, pp. 167–179. Springer, Cham (2018)

    Chapter  Google Scholar 

  18. Johnny, S.: The Coding Manual for Qualitative Researchers. SAGE Publications Limited, Thousand Oaks (2012)

    Google Scholar 

  19. Kevic, K., Walters, B.M., Shaffer, T.R., Sharif, B., Shepherd, D.C., Fritz, T.: Tracing software developers’ eyes and interactions for change tasks. In Proceedings of the 10th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2015, pp. 202–213. ACM, New York (2015)

  20. Kim, J., Hahn, J., Hahn, H.: How do we understand a system with (so) many diagrams? Cognitive integration processes in diagrammatic reasoning. Inf. Syst. Res. 11(3), 284–303 (2000)

    Article  Google Scholar 

  21. Kindler, E., Rubin, V., Schäfer, W.: Incremental workflow mining based on document versioning information. In: Software Process Workshop, volume 3840 of LNCS, pp. 287–301. Springer, Berlin (2005)

    Chapter  Google Scholar 

  22. King, T.M., Nunez, G., Santiago, D., Cando, A., Mack, C.: Legend: An Agile DSL Toolset for Web Acceptance Testing. pp. 409–412 (2014)

  23. Lawrie, D., Morrell, C., Feild, H., Binkley, D.: Effective identifier names for comprehension and memory. Innov. Syst. Softw. Eng. 3(4), 303–318 (2007)

    Article  Google Scholar 

  24. Leemans, M., van der Aalst, W.M.P.: Process mining in software systems: discovering real-life business transactions and process models from distributed systems. In: Proceedings of MODELS, pp. 44–53 (2015)

  25. Leemans, M., van der Aalst, W.M.P., van den Brand, M.G..J.: Recursion aware modeling and discovery for hierarchical software event log analysis (extended). CoRR, abs/1710.09323 (2017)

  26. Leemans, Sander J. J., Fahland, Dirk, van der Aalst, Wil M. P.: Discovering block-structured process models from event logs - a constructive approach. In Application and Theory of Petri Nets and Concurrency. Springer, (2013)

  27. Lenberg, P., Feldt, R., Wallgren, L.: Behavioral software engineering: a definition and systematic literature review. J. Syst. Softw. 107, 15–37 (2015)

    Article  Google Scholar 

  28. Letovsky, S.: Cognitive processes in program comprehension. J. Syst. Softw. 7(4), 325–339 (1987)

    Article  Google Scholar 

  29. Liu, C., van Dongen, B.F., Assy, N., Aalst, W.M.P.: Software architectural model discovery from execution data. In: 13th International Conference on Evaluation of Novel Approaches to Software Engineering, 03 (2018)

  30. Liu, C., van Dongen, B., Assy, N., van der Aalst, W.M.P.: Component behavior discovery from software execution data. In: IEEE Symposium Series on Computational Intelligence (SSCI), pp. 1–8, (2016)

  31. Mller, S.C., Fritz, T.: Stakeholders’ information needs for artifacts and their dependencies in a real world context. In: IEEE International Conference on Software Maintenance, pp 290–299 (2013)

  32. Nancy, P.: Empirical studies of programmers: Second workshop. In: Comprehension Strategies in Programming, pp. 100–113. Ablex Publishing Corp., Norwood (1987)

  33. North, D.: Introducing BDD (2006). http://dannorth.net/introducing-bdd/. Last Accessed October 2018

  34. Rubin, V., Günther, C.W., van der Aalst, W.M.P., Kindler, E., van Dongen, B.F., Schäfer, W.: Process mining framework for software processes. In Proceedings of ICSP 2007, pp. 169–181. Springer (2007)

  35. Nurdiani, I., Börstler, J., Fricker, S.: The impacts of agile and lean practices on project constraints: a tertiary study. J. Syst. Softw. 119, 162–183 (2016)

    Article  Google Scholar 

  36. Oliveira, G., Marczak, S.: On the understanding of bdd scenarios’ quality: Preliminary practitioners’ opinions. In: Kamsties, E., Horkoff, J., Dalpiaz, F. (eds.) Requirements Engineering: Foundation for Software Quality, pp. 290–296. Springer, Berlin (2018)

    Chapter  Google Scholar 

  37. Salviulo, F., Scanniello, G.: Dealing with identifiers and comments in source code comprehension and maintenance: results from an ethnographically-informed study with students and professionals. In: Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering, EASE ’14, pp. 48:1–48:10. ACM, New York (2014)

  38. Schröter, I., Krüger, J., Siegmund, J., Leich, T.: Comprehending studies on program comprehension. In: IEEE/ACM 25th International Conference on Program Comprehension (ICPC), pp. 308–311 (2017)

  39. Shaffer, T.R., Wise, J.L., Walters, B.M., Müller, S.C., Falcone, M., Sharif, B.: itrace: Enabling eye tracking on software artifacts within the ide to support software engineering tasks. In: Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2015, pp. 954–957. ACM, New York (2015)

  40. Sharafi, Z., Soh, Z., Guéhéneuc, Y., Antoniol, G.: Women and men—different but equal: on the impact of identifier style on source code reading. In: 20th IEEE International Conference on Program Comprehension (ICPC), pp. 27–36 (2012)

  41. Sharafi, Z., Soh, Z., Guéhéneuc, Y.-G.: A systematic literature review on the usage of eye-tracking in software engineering. Inf. Softw. Technol. 67, 79–107 (2015)

    Article  Google Scholar 

  42. Siau, K.Leng: Empirical studies in information modeling: Interpretation of the object relationship. PhD thesis, University of British Columbia (1996)

  43. Smart, J.F.: BDD in Action. Manning Publications, New York (2014)

    Google Scholar 

  44. IEEE Computer Society, Bourque, P., Fairley, R.E.: Guide to the Software Engineering Body of Knowledge (SWEBOK(R)): Version 3.0., 3rd edn. IEEE Computer Society Press, Los Alamitos (2014)

  45. Solis, C., Wang, X.: A study of the characteristics of behaviour driven development. In: 37th EUROMICRO Conference on Software Engineering and Advanced Applications, pp. 383–387 (2011)

  46. Soloway, E., Adelson, B., Ehrlich, K.: Knowledge and processes in the comprehension of computer programs. Nat. Expert. 129, 152 (1988)

    Google Scholar 

  47. van der Aalst, W.: Big software on the run: In vivo software analytics based on process mining (keynote). In: Proceedings of ICSSP 2015, pp. 1–5. ACM (2015)

  48. van der Aalst, W.M.P.: Process Mining: Data Science in Action. Springer, Berlin (2016)

    Book  Google Scholar 

  49. Vans, A.M., von Mayrhauser, A.: Program comprehension during software maintenance and evolution. Computer 28(8), 44–55 (1995)

    Article  Google Scholar 

  50. Version One.: 12th Annual State of Agile™ Report, 2017. Last Accessed December 2018

  51. Vessey, I., Conger, S.A.: Requirements Specification: Learning Object, Process, and Data Methodologies. Communications of the ACM (1994)

  52. Walters, B., Shaffer, T., Sharif, B., Kagdi, H.: Capturing software traceability links from developers’ eye gazes. In: Proceedings of the 22nd International Conference on Program Comprehension, ICPC 2014, pp. 201–204. ACM, New York (2014)

  53. Wang, Y., Wagner, S.: Combining stpa and bdd for safety analysis and verification in agile development: a controlled experiment. In: Garbajosa, J., Wang, X., Aguiar, A. (eds.) Agile Processes in Software Engineering and Extreme Programming, pp. 37–53. Springer, Cham (2018)

    Chapter  Google Scholar 

  54. Wieringa, R.J.: Design Science Methodology for Information Systems and Software Engineering. Springer, Berlin (2014). https://doi.org/10.1007/978-3-662-43839-8

    Book  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. DTU Compute, Software and Process Engineering, Technical University of Denmark, 2800, Kgs. Lyngby, Denmark

    Constantina Ioannou, Andrea Burattin & Barbara Weber

  2. SDU Software Engineering, The Mærsk Mc-Kinney Møller Institute, University of Southern Denmark, 5230, Odense M, Denmark

    Indira Nurdiani

  3. University of St. Gallen, 9000, St. Gallen, Switzerland

    Barbara Weber

Authors
  1. Constantina Ioannou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Indira Nurdiani
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Andrea Burattin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Barbara Weber
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Indira Nurdiani.

Additional information

Communicated by Dr. Iris Reinhartz-Berger and Sérgio Guerreiro.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Task description

Task description

Fig. 13
figure 13

The code understanding question for Task 1.2.0 and the answer Task 1.2.1 files are shown

Full size image

Tasks consist of answering questions regarding domain understanding and code understanding. In this appendix, we present two screenshots with question and corresponding answer files along with the structure of the software artifacts. We choose to present Task 1.1.0 and Task 1.2.0 as example. Task 1.1.0 is a domain understanding question (see Fig. 12), whereas Task 1.2.0 is a code understanding question (see Fig. 13). The remaining tasks are available at https://github.com/CIoann/cucitrace.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ioannou, C., Nurdiani, I., Burattin, A. et al. Mining reading patterns from eye-tracking data: method and demonstration. Softw Syst Model 19, 345–369 (2020). https://doi.org/10.1007/s10270-019-00759-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10270-019-00759-4

Keywords

Search

Navigation