Empirical evaluation of the effects of experience on code quality and programmer productivity: an exploratory study
There is a widespread belief in both SE and other branches of science that experience helps professionals to improve their performance. However, cases have been reported where experience not only does not have a positive influence but sometimes even degrades the performance of professionals.
Determine whether years of experience influence programmer performance.
We have analysed 10 quasi-experiments executed both in academia with graduate and postgraduate students and in industry with professionals. The experimental task was to apply ITLD on two experimental problems and then measure external code quality and programmer productivity.
Programming experience gained in industry does not appear to have any effect whatsoever on quality and productivity. Overall programming experience gained in academia does tend to have a positive influence on programmer performance. These two findings may be related to the fact that, as opposed to deliberate practice, routine practice does not appear to lead to improved performance. Experience in the use of productivity tools, such as testing frameworks and IDE also has positive effects.
Years of experience are a poor predictor of programmer performance. Academic background and specialized knowledge of task-related aspects appear to be rather good predictors.
KeywordsExperience Industry Academy Programming Iterative test-last development External quality Productivity Performance
We would like to acknowledge Dr.Hakan Erdogmus who contributed to the design of one of the tasks used in the study (BSK) and the corresponding test cases. We also wish to acknowledge Mr. Timo Raty for his participation in the creation of the code templates for C++, and the training given in one of the quasi-experiments. We wish also acknowledge Mr. Adrian Santos for his support in the collection of the subjects’ data.
- Adelson B (1984) When novices surpass experts: the difficulty of a task may increase with expertise. J Exp Psychol: Learn Mem Cogn 10(3):483Google Scholar
- Aranda A, Dieste O, Juristo N (2014) Evidence of the presence of bias in subjective metrics: analysis within a family of experiments. Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering (EASE 2014). London, UK, pp 24–27Google Scholar
- Askar P, Davenport D (2009) An investigation of factors related to self-efficacy for java programming among engineering students. Turk Online J Educ Technol 8(1):26–32Google Scholar
- Belsley DA (1991) Conditioning diagnostics: collinearity and weak data in regression. WileyGoogle Scholar
- Bob U (2005) The bowling game kata. Retrieved from http://butunclebob.com/ArticleS.UncleBob.TheBowlingGameKata
- Burkhardt J, Détienne F, Wiedenbeck S (1997) Mental representations constructed by experts and novices in object-oriented program comprehension. In: Howard S, Hammond J, Lindgaard G (eds) Springer US, pp 339–346Google Scholar
- Camerer CF, Johnson EJ (1997) 10 the process-performance paradox in expert judgment: How can experts know so much and predict so badly? Research on Judgment and Decision Making: Currents, Connections, and Controversies. 342Google Scholar
- Chase WG, Simon HA (1973) The mind’s eye in chessGoogle Scholar
- Chulis K (2012) Optimal segmentation approach and application. clustering vs. classification trees. Retrieved from http://www.ibm.com/developerworks/library/ba-optimal-segmentation/
- Colvin G (2008) Talent is overrated: What really separates world-class performers from Everybody Else. Penguin Publishing GroupGoogle Scholar
- Crosby M, Scholtz J, Widenbeck S (2002) The roles beacons play in comprehension for novice and expert programmers. 14th Workshop of the Psychology of Programming Interest Group, Brunel University. pp 58–73Google Scholar
- Curtis B (1984) Fifteen years of psychology in software engineering: individual differences and cognitive science. IEEE Press, OrlandoGoogle Scholar
- Darcy DP, Ma M (2005) Exploring individual characteristics and programming performance: Implications for programmer selection. Proceedings of the 38th Annual Hawaii International Conference on System Sciences, 314a.Google Scholar
- Daun M, Salmon A, Weyer T, Pohl K (2015) The impact of students’ skills and experiences on empirical results: A controlled experiment with undergraduate and graduate students. Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering, Art. No. 29.Google Scholar
- De Groot AD (1978) Thought and choice in chess. Walter de GruyterGoogle Scholar
- Ericsson KA (2006a) The influence of experience and deliberate practice on the development of superior expert performance. The Cambridge Handbook of Expertise and Expert Performance, pp 683–703Google Scholar
- Ericsson KA (2006b) An introduction to cambridge handbook of expertise and expert performance: Its development, organization, and content. In: Ericsson KA, Charness N, Hoffman RR, Feltovich PJ (eds) The cambridge handbook of expertise and expert performance. Cambridge University Press, pp 3–19Google Scholar
- Experience (2015) from http://www.merriam-webster.com/dictionary/experience. Retrieved 7 Oct 2015
- Fenton N, Bieman J (2014) Software metrics: a rigorous and practical approach, third edition. CRC Press.Google Scholar
- Field A, Miles J, Field Z (2012) Discovering statistics using R. SAGE PublicationsGoogle Scholar
- Glenwick DS (2016) Handbook of methodological approaches to community-based research: Qualitative, quantitative, and mixed methods. Oxford University PressGoogle Scholar
- Hedges LV, Olkin I (1985) Statistical methods for meta-analysis. Academic PressGoogle Scholar
- ISO I (2011) IEC25010: 2011 systems and software engineering–systems and software quality requirements and evaluation (SQuaRE)–System and software quality models. Int Organ StandGoogle Scholar
- Jeffries R, Turner AA, Polson PG, Atwood ME (1981) The processes involved in designing software. Cogn Skills Acquis 255:283Google Scholar
- MacCallum R, Zhang S, Preacher K, Rucker D (2002) On the practice of dichotomization of quantitative variables. 7:10–40Google Scholar
- Madeyski L (2005) Preliminary analysis of the effects of pair programming and test-driven development on the external code quality. Proceedings of the 2005 Conference on Software Engineering: Evolution and Emerging Technologies. pp. 113–123Google Scholar
- Mayer RE (1997) From novice to expert. In: Helander M, Landauer TK, Prabhu P (eds) Handbook of human-computer interaction, 2nd edn. Elsevier Science B.V, pp. 781–795Google Scholar
- Miles J, Shevlin M (2001) Applying regression and correlation: A guide for students and researchers. SAGE PublicationsGoogle Scholar
- Muller MM, Padberg F (2004) An empirical study about the feelgood factor in pair programming. Proceedings 10th International Symposium on Software Metrics. pp 151–158Google Scholar
- Nisbet R, Elder J, Miner G (2009) Handbook of statistical analysis and data mining applications. Academic PressGoogle Scholar
- Ricca F, Di Penta M, Torchiano M, Tonella P, Ceccato M (2007) The role of experience and ability in comprehension tasks supported by UML stereotypes. 29th International Conference on Software Engineering. pp 375–384Google Scholar
- Riley RD, Lambert PC, Abo-Zaid G (2010) Meta-analysis of individual participant data: Rationale, conduct, and reporting. BMJ 340. doi: 10.1136/bmj.c221
- Runeson P (2003) Using students as experiment subjects – an analysis on graduate and freshmen student data. Proceedings 7Th International conference on empirical assessment & evaluation in software engineering. pp 95–102Google Scholar
- Sim SE, Ratanotayanon S, Aiyelokun O, Morris E (2006) An initial study to develop an empirical test for software engineering expertise. Institute for Software Research, University of California, Irvine, CA, USA, Technical Report# UCI-ISR-06-6Google Scholar
- Soloway E, Bonar J, Ehrlich K (1983) Cognitive strategies and looping constructs: an empirical study. Commun ACM 26(11):853–860Google Scholar
- Votta LG (1994) By the way, has anyone studied any real programmers, yet? Software Process Workshop, 1994. Proceedings., Ninth International. pp 93–95Google Scholar
- Weisberg S (2005). Applied Linear Regression, third edition. John Wiley & Sons, Inc., Hoboken, NJGoogle Scholar
- Williams L, Kudrjavets G, Nagappan N (2009) On the effectiveness of unit test automation at microsoft. software reliability engineering, 2009. ISSRE ‘09. 20th International symposium on. pp 81–89Google Scholar