Skip to main content
SpringerLink
Log in

Do attention and memory explain the performance of software developers?

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

Writing and modifying source code are core activities in software development and evolution. The outcome of a coding task in terms of quality may depend on several aspects, such as the difficulty of the task or the complexity of the system. Besides, it is well known that individual characteristics of developers, like the programming experience, play a lead role in this. Recent work started exploring the influence that cognitive human aspects have on the ability of developers to acquire information from the source code (e.g., finding security blind spots). However, it is still unknown to what extent such aspects influence their ability of completing coding tasks. In this paper, we theorize that two cognitive human aspects, attention and memory, play a role in predicting the outcome of a coding task. We conducted a controlled experiment involving 32 participants (18 bachelor students, 9 master students, 2 Ph.D. students. and 3 practitioners), in which we asked them to complete two bug-fixing and two feature implementation tasks. We measured, for each of them, three attention-related factors (i.e., alerting, orienting, and executive control) and two memory-related ones (i.e., working memory and immediate recall) through well-established psychometric tests. Finally, we investigated to what extent these factors can explain the correctness, the readability and the time taken to complete a task in function of such factors. Our results show that all the attention- and memory-related factors achieved very low correlation with correctness and time. Indeed, the number of years of programming experience is far more important than all the other variables we considered for explaining the correctness and the time required to complete a task. Moreover, we found a significant relationship between orienting (an attention-related factor) and code readability.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Notes

  1. ANT is also known as “Flanker Inhibitory Control and Attention Test” (e.g., in the NIH Toolbox Cognition Battery))

  2. https://leetcode.com/

  3. We used the tool available at https://www.statskingdom.com/sample_size_regression.html

  4. See, for example: https://github.com/FantacyWorld/libMSVL/blob/master/dyn_array.c#L307.

References

  • Allan J (2013) Cognitions, Springer New York, pp 441–441. https://doi.org/10.1007/978-1-4419-1005-9_1114

  • Baddeley AD (1983) Working memory. Philos Trans R Soc Lond of London B, Biol Sci 302(1110):311–324

    Article  Google Scholar 

  • Benedict RH, DeLuca J, Phillips G, LaRocca N, Hudson LD, Rudick R, Consortium MSOA (2017) Validity of the symbol digit modalities test as a cognition performance outcome measure for multiple sclerosis. Mult Scler J 23(5):721–733

    Article  Google Scholar 

  • Benjafield JG, Smilek D, Kingstone A (2010) Cognition, 4th edn. Oxford University Press, New York

    Google Scholar 

  • Benjamini Y, Hochberg Y (1995) Controlling the false discovery rate: a practical and powerful approach to multiple testing. J R Stat Soc Ser B (Methodol) 57(1):289–300

    MathSciNet  MATH  Google Scholar 

  • Bhattacharya P, Neamtiu I (2011) Bug-fix time prediction models: can we do better? In: Proceedings of the 8th working conference on mining software repositories, pp 207–210

  • Bialystok E, DePape AM (2009) Musical expertise, bilingualism, and executive functioning. J Exp Psychol Hum Percept Perform 35(2):565

    Article  Google Scholar 

  • Blackburn JD, Scudder GD, Van Wassenhove LN (1996) Improving speed and productivity of software development: a global survey of software developers. IEEE Trans Soft Eng 22(12):875–885

    Article  Google Scholar 

  • Blank A, Frush Holt R, Pisoni DB, Kronenberger WG (2020) Associations between parenting stress, language comprehension, and inhibitory control in children with hearing loss. J Speech Hear Res 63(1):321–333

    Article  Google Scholar 

  • Bowrin AR, King J (2010) Time pressure, task complexity, and audit effectiveness. Managerial auditing journal

  • Brearly TW, Rowland JA, Martindale SL, Shura RD, Curry D, Taber KH (2018) Comparability of ipad and web-based nih toolbox cognitive battery administration in veterans. Arch Clin Neuropsychol 34(4):524–530

    Article  Google Scholar 

  • Brun Y, Lin T, Somerville JE, Myers E, Ebner NC (2021) Blindspots in python and java apis result in vulnerable code. arXiv:2103.06091

  • Cairncross M, Gindwani H, Rita Egbert A, Torres IJ, Hutchison JS, Dams O’Connor K, Panenka WJ, Brubacher JR, Meddings L, Kwan L, et al (2022) Criterion validity of the brief test of adult cognition by telephone (btact) for mild traumatic brain injury. Brain Injury pp 1–9

  • Cappos J, Zhuang Y, Oliveira D, Rosenthal M, Yeh KC (2014) Vulnerabilities as blind spots in developer’s heuristic-based decision-making processes. In: Proceedings of the 2014 new security paradigms workshop, pp 53–62

  • Cliff N (1993) Dominance statistics: Ordinal analyses to answer ordinal questions. Psychol Bull 114(3):494

    Article  Google Scholar 

  • Colom R, Flores-Mendoza C, Quiroga MÁ, Privado J (2005) Working memory and general intelligence: The role of short-term storage. Pers Indiv Diff 39(5):1005–1014

    Article  Google Scholar 

  • Conway A, Jarrold C, Miyake A (2008) Variation in working memory. Oxford University Press

    Book  Google Scholar 

  • Coulter NS (1983) Software science and cognitive psychology. IEEE Trans Softw Eng 2:166–171

    Article  Google Scholar 

  • Crawford JR (1998) Introduction to the assessment of attention and executive functioning. Neuropsychol Rehabil 8(3):209–211

    Article  Google Scholar 

  • Di Nucci D, Palomba F, De Rosa G, Bavota G, Oliveto R, De Lucia A (2017) A developer centered bug prediction model. IEEE Trans Softw Eng 44(1):5–24

    Article  Google Scholar 

  • Ebisch SJ, Mantini D, Romanelli R, Tommasi M, Perrucci MG, Romani GL, Colom R, Saggino A (2013) Long-range functional interactions of anterior insula and medial frontal cortex are differently modulated by visuospatial and inductive reasoning tasks. Neuroimage 78:426–438

    Article  Google Scholar 

  • Efroymson MA (1960) Multiple regression analysis. Mathematical Methods for Digital Computers pp 191–203

  • Eriksen BA, Eriksen CW (1974) Effects of noise letters upon the identification of a target letter in a nonsearch task. Percept Psychophys 16(1):143–149

    Article  Google Scholar 

  • Fan J, McCandliss BD, Sommer T, Raz A, Posner MI (2002) Testing the efficiency and independence of attentional networks. Journal of cognitive neuroscience 14(3):340–347

    Article  Google Scholar 

  • Fellows RP, Schmitter-Edgecombe M (2020) Symbol digit modalities test: Regression-based normative data and clinical utility. Arch Clin Neuropsychol 35(1):105–115

    Article  Google Scholar 

  • Fisher RA (1922) On the interpretation of \(\chi \) 2 from contingency tables, and the calculation of p. J R Stat Soc 85(1):87–94

    Article  Google Scholar 

  • Floyd B, Santander T, Weimer W (2017) Decoding the representation of code in the brain: An fmri study of code review and expertise. In: 2017 IEEE/ACM 39th International conference on software engineering (ICSE), IEEE, pp 175–186

  • Forn C, Belloch V, Bustamante JC, Garbin G, Parcet-Ibars MÀ, Sanjuan A, Ventura N, Ávila C (2009) A symbol digit modalities test version suitable for functional mri studies. Neurosci Lett 456(1):11--14

    Article  Google Scholar 

  • Forn C, Rocca MA, Boscá I, Casanova B, Sanjuan A, Filippi M (2013) Analysis of “task-positive” and “task-negative” functional networks during the performance of the symbol digit modalities test in patients at presentation with clinically isolated syndrome suggestive of multiple sclerosis. Exp Brain Res 225(3):399–407

  • Friedman NP, Miyake A (2004) The relations among inhibition and interference control functions: a latent-variable analysis. J Exp Psychol: General 133(1):101

    Article  Google Scholar 

  • Gellman MD, Turner JR (eds) (2013) Cognition, Springer New York, pp 441–441. https://doi.org/10.1007/978-1-4419-1005-9_100314

  • Genova HM, Hillary FG, Wylie G, Rypma B, Deluca J (2009) Examination of processing speed deficits in multiple sclerosis using functional magnetic resonance imaging. J Int Neuropsychol Soc 15(3):383–393

    Article  Google Scholar 

  • Gershon RC, Wagster MV, Hendrie HC, Fox NA, Cook KF, Nowinski CJ (2013) Nih toolbox for assessment of neurological and behavioral function. Neurology 80(11 Supplement 3):S2–S6

  • Green CS, Bavelier D (2007) Action-video-game experience alters the spatial resolution of vision. Psychol Sci 18(1):88–94

    Article  Google Scholar 

  • Habib M, Besson M (2009) What do music training and musical experience teach us about brain plasticity? Music Percept 26(3):279–285

    Article  Google Scholar 

  • Halberda J, Mazzocco MM, Feigenson L (2008) Individual differences in non-verbal number acuity correlate with maths achievement. Nature 455(7213):665–668

    Article  Google Scholar 

  • He Z, Shu F, Yang Y, Li M, Wang Q (2012) An investigation on the feasibility of cross-project defect prediction. Autom Softw Eng 19(2):167–199

    Article  Google Scholar 

  • Howard SJ, Johnson J, Pascual-Leone J (2014) Clarifying inhibitory control: Diversity and development of attentional inhibition. Cogn Dev 31:1–21

    Article  Google Scholar 

  • Huang Y, Liu X, Krueger R, Santander T, Hu X, Leach K, Weimer W (2019) Distilling neural representations of data structure manipulation using fmri and fnirs. In: 2019 IEEE/ACM 41st international conference on software engineering (ICSE), IEEE, pp 396–407

  • Hughes DJ (2018) Psychometric validity: Establishing the accuracy and appropriateness of psychometric measures. A multidisciplinary reference on survey, scale and test development, The Wiley handbook of psychometric testing, pp 751–779

    Google Scholar 

  • Hutchins M, Foster H, Goradia T, Ostrand T (1994) Experiments on the effectiveness of dataflow-and control-flow-based test adequacy criteria. In: Proceedings of 16th international conference on software engineering, IEEE, pp 191–200

  • Juristo N, Moreno AM (2013) Basics of software engineering experimentation. Springer Science & Business Media

  • Karas Z, Jahn A, Weimer W, Huang Y (2021) Connecting the dots: rethinking the relationship between code and prose writing with functional connectivity. In: Proceedings of the 29th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering, pp 767–779

  • Koen JD, Aly M, Wang WC, Yonelinas AP (2013) Examining the causes of memory strength variability: Recollection, attention failure, or encoding variability? J Exp Psychol Learn Mem Cogn 39(6):1726

    Article  Google Scholar 

  • Krueger R, Huang Y, Liu X, Santander T, Weimer W, Leach K (2020) Neurological divide: an fmri study of prose and code writing. In: 2020 IEEE/ACM 42nd international conference on software engineering (ICSE), IEEE, pp 678–690

  • Kuutila M, Mäntylä MV, Claes M, Elovainio M (2017) Reviewing literature on time pressure in software engineering and related professions: computer assisted interdisciplinary literature review. In: 2017 IEEE/ACM 2nd international workshop on emotion awareness in software engineering (SEmotion), IEEE, pp 54–59

  • Lachman ME, Agrigoroaei S, Tun PA, Weaver SL (2014) Monitoring cognitive functioning: psychometric properties of the brief test of adult cognition by telephone. Assessment 21(4):404–417

    Article  Google Scholar 

  • LeetCode (2020a) Leetcode problem: Duplicate zeros. https://leetcode.com/problems/duplicate-zeros/

  • LeetCode (2020b) Leetcode problem: Find and replace pattern. https://leetcode.com/problems/find-and-replace-pattern/

  • LeetCode (2020c) Leetcode problem: Regular expression matching. https://leetcode.com/problems/regular-expression-matching/

  • LeetCode (2020d) Leetcode problem: Sort the matrix diagonally. https://leetcode.com/problems/sort-the-matrix-diagonally/

  • Lesage E, Sutherland MT, Ross TJ, Salmeron BJ, Stein EA (2020) Nicotine dependence (trait) and acute nicotinic stimulation (state) modulate attention but not inhibitory control: converging fmri evidence from go-nogo and flanker tasks. Neuropsychopharmacology 45(5):857--865

    Article  Google Scholar 

  • Li Z, Jing XY, Zhu X (2018) Progress on approaches to software defect prediction. Iet Software 12(3):161–175

    Article  Google Scholar 

  • Lilienthal L, Tamez E, Shelton JT, Myerson J, Hale S (2013) Dual n-back training increases the capacity of the focus of attention. Psychon Bull Rev 20(1):135–141

  • Mann HB, Whitney DR (1947) On a test of whether one of two random variables is stochastically larger than the other. The annals of mathematical statistics pp 50–60

  • Martínez K, Burgaleta M, Román FJ, Escorial S, Shih PC, Quiroga MÁ, Colom R (2011) Can fluid intelligence be reduced to ’simple’short-term storage? Intelligence 39(6):473–480

    Article  Google Scholar 

  • Matchock RL, Mordkoff JT (2009) Chronotype and time-of-day influences on the alerting, orienting, and executive components of attention. Exp brain Res 192(2):189–198

    Article  Google Scholar 

  • Mundy E, Gilmore CK (2009) Children’s mapping between symbolic and nonsymbolic representations of number. J Exp Child Psycholo 103(4):490–502

    Article  Google Scholar 

  • Musso M, Kyndt E, Cascallar E, Dochy F (2012) Predicting mathematical performance: The effect of cognitive processes and self-regulation factors. Education Research International 2012

  • Nour S, Struys E, Stengers H (2019) Attention network in interpreters: The role of training and experience. Behav Sci 9(4):43

    Article  Google Scholar 

  • Oded Y (2011) Biofeedback-based mental training in the military–the “mental gym™” project. Biofeedback 39(3):112–118

  • Oliveira D, Rosenthal M, Morin N, Yeh KC, Cappos J, Zhuang Y (2014) It’s the psychology stupid: how heuristics explain software vulnerabilities and how priming can illuminate developer’s blind spots. In: Proceedings of the 30th annual computer security applications conference, pp 296–305

  • Oliveira DS, Lin T, Rahman MS, Akefirad R, Ellis D, Perez E, Bobhate R, DeLong LA, Cappos J, Brun Y (2018) API blindspots: Why experienced developers write vulnerable code. In: Fourteenth symposium on usable privacy and security (\(\{\)SOUPS\(\}\) 2018), pp 315–328

  • Parmenter B, Weinstock-Guttman B, Garg N, Munschauer F, Benedict RH (2007) Screening for cognitive impairment in multiple sclerosis using the symbol digit modalities test. Multiple Sclerosis J 13(1):52–57

    Article  Google Scholar 

  • Passolunghi MC, Vercelloni B, Schadee H (2007) The precursors of mathematics learning: Working memory, phonological ability and numerical competence. Cogn Dev 22(2):165–184

    Article  Google Scholar 

  • Peitek N, Siegmund J, Apel S, Kästner C, Parnin C, Bethmann A, Leich T, Saake G, Brechmann A (2018) A look into programmers’ heads. IEEE Trans Softw Eng 46(4):442–462

    Article  Google Scholar 

  • Piantadosi V, Scalabrino S, Serebrenik A, Novielli N, Oliveto R (2021) Replication package of "do attention and memory explain the performance of software developers?". https://figshare.com/s/f3cf009d98ac60530ec6

  • Pinto G, Rebouças M, Castor F (2017) Inadequate testing, time pressure, and (over) confidence: a tale of continuous integration users. In: 2017 IEEE/ACM 10th international workshop on cooperative and human aspects of software engineering (CHASE), IEEE, pp 74–77

  • Posner MI (1980) Orienting of attention. Q J Exp Psychol 32(1):3–25

    Article  Google Scholar 

  • Posner MI, Petersen SE (1990) The attention system of the human brain. Ann Rev Neurosci 13(1):25–42

    Article  Google Scholar 

  • Posnett D, D’Souza R, Devanbu P, Filkov V (2013) Dual ecological measures of focus in software development. In: 2013 35th international conference on software engineering (ICSE), IEEE, pp 452–461

  • Rahman F, Posnett D, Devanbu P (2012) Recalling the" imprecision" of cross-project defect prediction. In: Proceedings of the ACM SIGSOFT 20th international symposium on the foundations of software engineering, pp 1–11

  • Rapport MD, Kofler MJ, Alderson RM, Timko TM Jr, DuPaul GJ (2009) Variability of attention processes in adhd: Observations from the classroom. J Atten Disord 12(6):563–573

    Article  Google Scholar 

  • Rasch RH, Tosi HL (1992) Factors affecting software developers’ performance: An integrated approach. MIS quarterly pp 395–413

  • Ricca F, Di Penta M, Torchiano M, Tonella P, Ceccato M (2007) The role of experience and ability in comprehension tasks supported by uml stereotypes. In: 29th International conference on software engineering (ICSE’07), IEEE, pp 375–384

  • Ricca F, Di Penta M, Torchiano M, Tonella P, Ceccato M (2009) How developers’ experience and ability influence web application comprehension tasks supported by uml stereotypes: A series of four experiments. IEEE Trans Softw Eng 36(1):96–118

    Article  Google Scholar 

  • Roy E (2013) Cognitive Function, Springer New York, New York, NY, pp 448–449. https://doi.org/10.1007/978-1-4419-1005-9_1117

  • Scalabrino S, Linares-Vásquez M, Oliveto R, Poshyvanyk D (2018) A comprehensive model for code readability. J Softw: Evol Process 30(6):e1958

    Google Scholar 

  • Scalabrino S, Bavota G, Vendome C, Poshyvanyk D, Oliveto R, et al (2019) Automatically assessing code understandability. IEEE Trans Softw Eng

  • Schacter DL, Wagner AD, Buckner RL (2000) Memory systems of 1999. Oxford University Press

    Google Scholar 

  • Schellenberg EG (2004) Music lessons enhance iq. Psychol Sci 15(8):511–514

    Article  Google Scholar 

  • Sharafi Z, Huang Y, Leach K, Weimer W (2021) Toward an objective measure of developers’ cognitive activities. ACM Trans Softw Eng Methodol (TOSEM) 30(3):1–40

    Article  Google Scholar 

  • Shneiderman B, Mayer R (1979) Syntactic/semantic interactions in programmer behavior: A model and experimental results. Int J Comput Sci Inf Sci 8(3):219–238

    Article  MATH  Google Scholar 

  • Shorey C, Friedman E (2018) Multimorbidity and cognitive decline in a national sample of aging adults. Innovation in Aging 2(Suppl 1):505

    Article  Google Scholar 

  • Siegmund J, Kästner C, Liebig J, Apel S, Hanenberg S (2014) Measuring and modeling programming experience. Empir Softw Eng 19(5):1299–1334

    Article  Google Scholar 

  • Siegmund J, Peitek N, Parnin C, Apel S, Hofmeister J, Kästner C, Begel A, Bethmann A, Brechmann A (2017) Measuring neural efficiency of program comprehension. In: Proceedings of the 2017 11th joint meeting on foundations of software engineering, pp 140–150

  • Silva P, Spedo C, Barreira AA, Leoni RF (2018) Symbol digit modalities test adaptation for magnetic resonance imaging environment: A systematic review and meta-analysis. Mult. Sclerand Relat Disord 20:136–143

    Article  Google Scholar 

  • Sloan L (2015) Learn about Spearman’s Rank-order Correlation Coefficient in SPSS with Data from the General Social Survey (2012). SAGE Publications

  • Song MK, Ward SE, Bair E, Weiner LJ, Bridgman JC, Hladik GA, Gilet CA (2015) Patient-reported cognitive functioning and daily functioning in chronic dialysis patients. Hemodial Int 19(1):90–99

    Article  Google Scholar 

  • Spearman C (1961) The proof and measurement of association between two things. Am J Psychol

  • Thota MK, Shajin FH, Rajesh P et al (2020) Survey on software defect prediction techniques. International Journal of Applied Science and Engineering 17(4):331–344

    Google Scholar 

  • Tiego J, Testa R, Bellgrove MA, Pantelis C, Whittle S (2018) A hierarchical model of inhibitory control. Front Psychol 9:1339

    Article  Google Scholar 

  • Tun PA, Lachman ME (2006) Telephone assessment of cognitive function in adulthood: the brief test of adult cognition by telephone. Age and Ageing 35(6):629–632

    Article  Google Scholar 

  • Wang H, Fan J, Yang Y (2004) Toward a multilevel analysis of human attentional networks. In: Proceedings of the annual meeting of the cognitive science society, vol 26

  • Weaver B, Bedard M, McAuliffe J, Parkkari M (2009) Using the attention network test to predict driving test scores. Accid Anal Prev 41(1):76–83

    Article  Google Scholar 

  • Weaver B, Bédard M, McAuliffe J (2013) Evaluation of a 10-minute version of the attention network test. The Clinical Neuropsychologist 27(8):1281–1299

    Article  Google Scholar 

  • Wei W, Yuan H, Chen C, Zhou X (2012) Cognitive correlates of performance in advanced mathematics. Br J Educ Psychol 82(1):157–181

    Article  Google Scholar 

  • Wong WE, Horgan JR, London S, Mathur AP (1998) Effect of test set minimization on fault detection effectiveness. Softw: Pract Experience 28(4):347–369

  • Wong WE, Debroy V, Gao R, Li Y (2013) The dstar method for effective software fault localization. IEEE Trans Reliab 63(1):290–308

    Article  Google Scholar 

  • Woumans E, Ceuleers E, Van der Linden L, Szmalec A, Duyck W (2015) Verbal and nonverbal cognitive control in bilinguals and interpreters. J Exp Psychol: Learn Mem Cogn 41(5):1579

    Google Scholar 

  • Zelazo PD, Anderson JE, Richler J, Wallner-Allen K, Beaumont JL, Weintraub S (2013) Ii. nih toolbox cognition battery (cb): Measuring executive function and attention. Monogr Soc Res Child Dev 78(4):16–33

  • Zhang H, Gong L, Versteeg S (2013) Predicting bug-fixing time: an empirical study of commercial software projects. In: 2013 35th international conference on software engineering (ICSE), IEEE, pp 1042–1051

  • Zimmermann T, Nagappan N, Gall H, Giger E, Murphy B (2009) Cross-project defect prediction: a large scale experiment on data vs. domain vs. process. In: Proceedings of the 7th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering, pp 91–100

Download references

Author information

Authors and Affiliations

  1. University of Molise, Campobasso, Italy

    Valentina Piantadosi, Simone Scalabrino & Rocco Oliveto

  2. Eindhoven University of Technology, Eindhoven, The Netherlands

    Alexander Serebrenik

  3. University of Bari, Bari, Italy

    Nicole Novielli

Authors
  1. Valentina Piantadosi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Simone Scalabrino
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Alexander Serebrenik
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nicole Novielli
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Rocco Oliveto
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Valentina Piantadosi.

Ethics declarations

Conflicts of Interest

The authors declare that they have no conflict of interest.

Additional information

Communicated by: Communicated by: Rafael Prikladnicki

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Piantadosi, V., Scalabrino, S., Serebrenik, A. et al. Do attention and memory explain the performance of software developers?. Empir Software Eng 28, 112 (2023). https://doi.org/10.1007/s10664-023-10316-9

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10664-023-10316-9

Keywords

Search

Navigation