Abstract
Understanding how developers interact with different software artifacts when performing comprehension tasks has a potential to improve developers’ productivity. In this paper, we propose a method to analyze eye-tracking data using process mining to find distinct reading patterns of how developers interacted with the different artifacts. To validate our approach, we conducted an exploratory study using eye-tracking involving 11 participants. We applied our method to investigate how developers interact with different artifacts during domain and code understanding tasks. To contextualize the reading patterns and to better understand the perceived benefits and challenges participants associated with the different artifacts and their choice of reading patterns, we complemented the eye-tracking data with the data obtained from think aloud. The study used behavior-driven development, a development practice that is increasingly used in Agile software development contexts, as a setting. The study shows that our method can be used to explore developers’ behavior at an aggregated level and identify behavioral patterns at varying levels of granularity.
Similar content being viewed by others
Notes
The new version of the iTrace plugin is available on GitHub, at https://github.com/CIoann/cucitraceSetup.
See https://atlasti.com/.
References
Abrahamsson, P., Salo, O., Ronkainen, J., Warsta, J.: Agile software development methods—review and analysis, Technical Report 478. VTT Publications (2002)
Binamungu, L.P., Embury, S.M., Konstantinou, N.: Maintaining behaviour driven development specifications: challenges and opportunities. In: IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER), pp. 175–184 (2018)
Blinman, S., Cockburn, A.: Program comprehension: investigating the effects of naming style and documentation. In: Proceedings of the 6th Australasian Conference on User Interface, vol. 40, AUIC ’05, pp. 73–78. Australian Computer Society, Inc, Darlinghurst (2005)
Boehm, B.: Software engineering economics. IEEE Trans. Softw. Eng. 10, 4–21 (1984)
Brooks, R.: Towards a theory of the comprehension of computer programs. Int. J. Man-Mach. Stud. 18(6), 543–554 (1983)
Busjahn, T., Schulte, C., Busjahn, A.: Analysis of code reading to gain more insight in program comprehension. In: Proceedings of the 11th Koli Calling International Conference on Computing Education Research, Koli Calling ’11, pp. 1–9. ACM, New York (2011)
Cios, K.J., Pedrycz, W., Swiniarski, R.W.: Data Mining Methods for Knowledge Discovery. Springer, Boston (1998)
Crosby, M.E., Stelovsky, J.: How do we read algorithms? A case study. Computer 23(1), 25–35 (1990)
de Souza, P.L., do Prado, A.F., de Souza, W.L., dos Santos Forghieri Pereira, S.M., Pires, L.F.: Combining behaviour-driven development with scrum for software development in the education domain. In: ICEIS (2), pp. 449–458. SciTePress (2017)
Fritz, T., Begel, A., Müller, S.C., Yigit-Elliott, S., Züger, M.: Using psycho-physiological measures to assess task difficulty in software development. In: Proceedings of the 36th International Conference on Software Engineering, pp. 402–413. ACM, New York (2014)
Gemino, A., Wand, Y.: A framework for empirical evaluation of conceptual modeling techniques. Requir. Eng. 9(4), 248–260 (2004)
Goldberg, J.H., Kotval, X.P.: Computer interface evaluation using eye movements: methods and constructs. Int. J. Ind. Ergon. 24(6), 631–645 (1999)
Graziotin, D., Wang, X., Abrahamsson, P.: Do feelings matter? On the correlation of affects and the self-assessed productivity in software engineering. J. Softw. Evol. Process 27(7), 467–487 (2015)
Gulden, J., Burattin, A., Andaloussi, A.A., Weber, B.: From analytical purposes to data visualizations: a decision process guided by a conceptual framework and eye tracking. Softw. Syst. Model. (2019)
Holmqvist, K., Nyström, M., Andersson, R., Dewhurst, R., Jarodzka, H., Van de Weijer, J.: Eye Tracking: A Comprehensive Guide to Methods and Measures. OUP, Oxford (2011)
IEEE Task Force on Process Mining.: Process Mining Manifesto. Business Process Management Workshops, pp. 169–194 (2011)
Ioannou, C., Burattin, A., Weber, B.: Mining developers’ workflows from IDE usage. In: Matulevičius, R., Dijkman, R. (eds.) Advanced Information Systems Engineering Workshops, pp. 167–179. Springer, Cham (2018)
Johnny, S.: The Coding Manual for Qualitative Researchers. SAGE Publications Limited, Thousand Oaks (2012)
Kevic, K., Walters, B.M., Shaffer, T.R., Sharif, B., Shepherd, D.C., Fritz, T.: Tracing software developers’ eyes and interactions for change tasks. In Proceedings of the 10th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2015, pp. 202–213. ACM, New York (2015)
Kim, J., Hahn, J., Hahn, H.: How do we understand a system with (so) many diagrams? Cognitive integration processes in diagrammatic reasoning. Inf. Syst. Res. 11(3), 284–303 (2000)
Kindler, E., Rubin, V., Schäfer, W.: Incremental workflow mining based on document versioning information. In: Software Process Workshop, volume 3840 of LNCS, pp. 287–301. Springer, Berlin (2005)
King, T.M., Nunez, G., Santiago, D., Cando, A., Mack, C.: Legend: An Agile DSL Toolset for Web Acceptance Testing. pp. 409–412 (2014)
Lawrie, D., Morrell, C., Feild, H., Binkley, D.: Effective identifier names for comprehension and memory. Innov. Syst. Softw. Eng. 3(4), 303–318 (2007)
Leemans, M., van der Aalst, W.M.P.: Process mining in software systems: discovering real-life business transactions and process models from distributed systems. In: Proceedings of MODELS, pp. 44–53 (2015)
Leemans, M., van der Aalst, W.M.P., van den Brand, M.G..J.: Recursion aware modeling and discovery for hierarchical software event log analysis (extended). CoRR, abs/1710.09323 (2017)
Leemans, Sander J. J., Fahland, Dirk, van der Aalst, Wil M. P.: Discovering block-structured process models from event logs - a constructive approach. In Application and Theory of Petri Nets and Concurrency. Springer, (2013)
Lenberg, P., Feldt, R., Wallgren, L.: Behavioral software engineering: a definition and systematic literature review. J. Syst. Softw. 107, 15–37 (2015)
Letovsky, S.: Cognitive processes in program comprehension. J. Syst. Softw. 7(4), 325–339 (1987)
Liu, C., van Dongen, B.F., Assy, N., Aalst, W.M.P.: Software architectural model discovery from execution data. In: 13th International Conference on Evaluation of Novel Approaches to Software Engineering, 03 (2018)
Liu, C., van Dongen, B., Assy, N., van der Aalst, W.M.P.: Component behavior discovery from software execution data. In: IEEE Symposium Series on Computational Intelligence (SSCI), pp. 1–8, (2016)
Mller, S.C., Fritz, T.: Stakeholders’ information needs for artifacts and their dependencies in a real world context. In: IEEE International Conference on Software Maintenance, pp 290–299 (2013)
Nancy, P.: Empirical studies of programmers: Second workshop. In: Comprehension Strategies in Programming, pp. 100–113. Ablex Publishing Corp., Norwood (1987)
North, D.: Introducing BDD (2006). http://dannorth.net/introducing-bdd/. Last Accessed October 2018
Rubin, V., Günther, C.W., van der Aalst, W.M.P., Kindler, E., van Dongen, B.F., Schäfer, W.: Process mining framework for software processes. In Proceedings of ICSP 2007, pp. 169–181. Springer (2007)
Nurdiani, I., Börstler, J., Fricker, S.: The impacts of agile and lean practices on project constraints: a tertiary study. J. Syst. Softw. 119, 162–183 (2016)
Oliveira, G., Marczak, S.: On the understanding of bdd scenarios’ quality: Preliminary practitioners’ opinions. In: Kamsties, E., Horkoff, J., Dalpiaz, F. (eds.) Requirements Engineering: Foundation for Software Quality, pp. 290–296. Springer, Berlin (2018)
Salviulo, F., Scanniello, G.: Dealing with identifiers and comments in source code comprehension and maintenance: results from an ethnographically-informed study with students and professionals. In: Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering, EASE ’14, pp. 48:1–48:10. ACM, New York (2014)
Schröter, I., Krüger, J., Siegmund, J., Leich, T.: Comprehending studies on program comprehension. In: IEEE/ACM 25th International Conference on Program Comprehension (ICPC), pp. 308–311 (2017)
Shaffer, T.R., Wise, J.L., Walters, B.M., Müller, S.C., Falcone, M., Sharif, B.: itrace: Enabling eye tracking on software artifacts within the ide to support software engineering tasks. In: Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2015, pp. 954–957. ACM, New York (2015)
Sharafi, Z., Soh, Z., Guéhéneuc, Y., Antoniol, G.: Women and men—different but equal: on the impact of identifier style on source code reading. In: 20th IEEE International Conference on Program Comprehension (ICPC), pp. 27–36 (2012)
Sharafi, Z., Soh, Z., Guéhéneuc, Y.-G.: A systematic literature review on the usage of eye-tracking in software engineering. Inf. Softw. Technol. 67, 79–107 (2015)
Siau, K.Leng: Empirical studies in information modeling: Interpretation of the object relationship. PhD thesis, University of British Columbia (1996)
Smart, J.F.: BDD in Action. Manning Publications, New York (2014)
IEEE Computer Society, Bourque, P., Fairley, R.E.: Guide to the Software Engineering Body of Knowledge (SWEBOK(R)): Version 3.0., 3rd edn. IEEE Computer Society Press, Los Alamitos (2014)
Solis, C., Wang, X.: A study of the characteristics of behaviour driven development. In: 37th EUROMICRO Conference on Software Engineering and Advanced Applications, pp. 383–387 (2011)
Soloway, E., Adelson, B., Ehrlich, K.: Knowledge and processes in the comprehension of computer programs. Nat. Expert. 129, 152 (1988)
van der Aalst, W.: Big software on the run: In vivo software analytics based on process mining (keynote). In: Proceedings of ICSSP 2015, pp. 1–5. ACM (2015)
van der Aalst, W.M.P.: Process Mining: Data Science in Action. Springer, Berlin (2016)
Vans, A.M., von Mayrhauser, A.: Program comprehension during software maintenance and evolution. Computer 28(8), 44–55 (1995)
Version One.: 12th Annual State of Agile™ Report, 2017. Last Accessed December 2018
Vessey, I., Conger, S.A.: Requirements Specification: Learning Object, Process, and Data Methodologies. Communications of the ACM (1994)
Walters, B., Shaffer, T., Sharif, B., Kagdi, H.: Capturing software traceability links from developers’ eye gazes. In: Proceedings of the 22nd International Conference on Program Comprehension, ICPC 2014, pp. 201–204. ACM, New York (2014)
Wang, Y., Wagner, S.: Combining stpa and bdd for safety analysis and verification in agile development: a controlled experiment. In: Garbajosa, J., Wang, X., Aguiar, A. (eds.) Agile Processes in Software Engineering and Extreme Programming, pp. 37–53. Springer, Cham (2018)
Wieringa, R.J.: Design Science Methodology for Information Systems and Software Engineering. Springer, Berlin (2014). https://doi.org/10.1007/978-3-662-43839-8
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by Dr. Iris Reinhartz-Berger and Sérgio Guerreiro.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Task description
Task description
Tasks consist of answering questions regarding domain understanding and code understanding. In this appendix, we present two screenshots with question and corresponding answer files along with the structure of the software artifacts. We choose to present Task 1.1.0 and Task 1.2.0 as example. Task 1.1.0 is a domain understanding question (see Fig. 12), whereas Task 1.2.0 is a code understanding question (see Fig. 13). The remaining tasks are available at https://github.com/CIoann/cucitrace.
Rights and permissions
About this article
Cite this article
Ioannou, C., Nurdiani, I., Burattin, A. et al. Mining reading patterns from eye-tracking data: method and demonstration. Softw Syst Model 19, 345–369 (2020). https://doi.org/10.1007/s10270-019-00759-4
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10270-019-00759-4