Skip to main content
SpringerLink
Log in

Measuring and modeling programming experience

  • Published:
Empirical Software Engineering Aims and scope Submit manuscript

Abstract

Programming experience is an important confounding parameter in controlled experiments regarding program comprehension. In literature, ways to measure or control programming experience vary. Often, researchers neglect it or do not specify how they controlled for it. We set out to find a well-defined understanding of programming experience and a way to measure it. From published comprehension experiments, we extracted questions that assess programming experience. In a controlled experiment, we compare the answers of computer-science students to these questions with their performance in solving program-comprehension tasks. We found that self estimation seems to be a reliable way to measure programming experience. Furthermore, we applied exploratory and confirmatory factor analyses to extract and evaluate a model of programming experience. With our analysis, we initiate a path toward validly and reliably measuring and describing programming experience to better understand and control its influence in program-comprehension experiments.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Notes

  1. ICPC was a workshop until 2005 (IWPC), which we also included.

  2. ESEM first took place in 2007.

  3. http://www.infosun.fim.uni-passau.de/spl/janet/prophet/

  4. The German system allows students to take courses in a flexible order and timing.

  5. Small: ±0.1 to ±0.3; medium: ±0.3 to ±0.5; strong: ±0.5 to ±1 (Cohen 1988).

  6. At German universities, semester starts mid October.

  7. http://www-01.ibm.com/software/analytics/spss/products/statistics/amos/. The script of the analysis is available at the project’s website.

  8. There is a heated ongoing debate about fit indices. Since there is not the best index, it is common to present different indices that focus on different aspects (Bagozzi and Yi 2012; Hu and Bentler 199; Hu and Bentler 1998).

References

  • Anderson T, Finn J (1996) The new statistical analysis of data. Springer

  • Anderson T, Rubin H (1956) Statistical inference in factor analysis. In: Proc. Berkeley symposium on mathematical statistics and probability, vol 5. University of California Press, pp 111–150

  • Arisholm E, Gallis H, Dybå T, Sjøberg D (2007) Evaluating pair programming with respect to system complexity and programmer expertise. IEEE Trans Softw Eng 33(2):65–86

    Article  Google Scholar 

  • Askar P, Davenport D (2009) An investigation of factors related to self-efficacy for Java programming among engineering students. TOJET 8(1):26–32

    Google Scholar 

  • Bagozzi R, Yi Y (2012) Specification, evaluation, and interpretation of structural equation models. J Acad Mark Sci 40(1):8–34

    Article  Google Scholar 

  • Bartholomew D, Martin Knott IM (2011) Latent variable models and factor analysis: a unified approach. Wiley, New York

  • Biffl S, Grossmann W (2001) Evaluating the accuracy of defect estimation models based on inspection data from two inspection cycles. In: Proc. int’l conf. software engineering (ICSE). IEEE CS, pp 145–154

  • Bornat R, Dehnadi S, Simon (2008) Mental models, consistency and programming aptitude. In: Proc. conf. on Australasian computing education, vol 78. Australian Computer Society, Inc., pp 53–61

  • Bunse C (2006) Using patterns for the refinement and translation of UML models: a controlled experiment. Empir Software Eng 11(2):227–267

    Article  Google Scholar 

  • Cohen J (1988) Statistical power analysis for the behavioral sciences, 2nd edn. Routledge Academic Press

  • Cohen J, Cohen P (1983) Applied multiple regression: correlation analysis for the behavioral sciences, 2nd edn. Addison-Wesley, Reading, MA

    Google Scholar 

  • Costello A, Osborne J (2005) Best practices in exploratory factor analysis: four recommendations for getting the most from your analysis. Practical Assessment, Research & Evaluation (PARE) 10(7):173–178

    Google Scholar 

  • Easterbrook S, Singer J, Storey M-A, Damian D (2008) Selecting empirical methods for software engineering research. In: Guide to advanced empirical software engineering. Springer, pp 285–311

  • Erdogmus H, Morisio M, Torchiano M (2005) On the effectiveness of the test-first approach to programming. IEEE Trans Softw Eng 31(3):226–237

    Article  Google Scholar 

  • Feigenspan J, Apel S, Liebig J, Kästner C (2011a) Exploring software measures to assess program comprehension. In: Proc. int’l symposium empirical software engineering and measurement (ESEM). IEEE CS, paper 3, pp 1–10

  • Feigenspan J, Schulze M, Papendieck M, Kästner C, Dachselt R, Köppen, V, Frisch M (2011b) Using background colors to support program comprehension in software product lines. In: Proc. int’l conf. evaluation and assessment in software engineering (EASE). Institution of Engineering and Technology, pp 66–75

  • Feigenspan J, Siegmund N (2012) Supporting comprehension experiments with human subjects. In: Proc. int’l conf. program comprehension (ICPC). IEEE CS, Tool demo, pp 244–246

  • Feigenspan J, Kästner C, Liebig J, Apel S, Hanenberg S (2012) Measuring programming experience. In: Proc. int’l conf. program comprehension (ICPC). IEEE CS, pp 73–82

  • Feigenspan J, Kästner C, Apel S, Liebig J, Schulze M, Dachselt R, Papendieck M, Leich T, Saake G (2013) Do background colors improve program comprehension in the #ifdef hell?. Empir Software Eng 18(4):699–745

    Article  Google Scholar 

  • Figueiredo E, Cacho N, Monteiro M, Kulesza U, Garcia R, Soares S, Ferrari F, Khan S, Filho F, Dantas F (2008) Evolving software product lines with aspects: an empirical study on design stability. In: Proc. int’l conf. software engineering (ICSE). ACM Press, pp 261–270

  • Goodwin J (1999) Research in psychology: methods and design, 2nd edn. Wiley, New York

    Google Scholar 

  • Hanenberg S, Kleinschmager S, Josupeit-Walter M (2009) Does aspect-oriented programming increase the development speed for crosscutting code? An empirical study. In: Proc. int’l symposium empirical software engineering and measurement (ESEM). IEEE CS, pp 156–167

  • Hannay J, Arisholm E, Engvik H, Sjøberg D (2010) Effects of personality on pair programming. IEEE Trans Softw Eng 36(1):61–80

    Article  Google Scholar 

  • Höst M, Regnell B, Wohlin C (2000) Using students as subjects: a comparative study of students and professionals in lead-time impact assessment. Empir Software Eng 5(3):201–214

    Article  MATH  Google Scholar 

  • Hu L, Bentler PM (1999) Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Struct Equ Model 6(1):1–55

    Article  Google Scholar 

  • Hu L, Bentler PM (1998) Fit indexes in covariance structure modeling: sensitivity to underparameterized model misspecification. Psychol Methods 3(4):424–453

    Article  Google Scholar 

  • Jedlitschka A, Ciolkowski M, Pfahl D (2008) Reporting experiments in software engineering. In: Guide to advanced empirical software engineering. Springer, pp 201–228

  • Kitchenham B, Charters S (2007) Guidelines for performing systematic literature reviews in software engineering. Technical Report EBSE 2007-001, Keele University and Durham University Joint Report

  • Kleinschmager S, Hanenberg S (2011) How to rate programming skills in programming experiments? A preliminary, exploratory, study based on university marks, pretests, and self-estimation. In: Proc. ACM SIGPLAN workshop on evaluation and usability of programming languages and tools. ACM Press, pp 15–24

  • Lewis-Beck M (1980) Applied regression: an introduction. Sage Pubications

  • Müller M (2004) Are reviews an alternative to pair programming? Empir Software Eng 9(4):335–351

    Article  Google Scholar 

  • Peterson R (2000) Constructing effective questionnaires. Sage Publications

  • Ricca F, Di Penta M, Torchiano M, Tonella P, Ceccato M (2007) The role of experience and ability in comprehension tasks supported by UML stereotypes. In: Proc. int’l conf. software engineering (ICSE). IEEE CS, pp 375–384

  • Siegmund J, Kästner C, Liebig J, Apel S (2012) Comparing program comprehension of physically and virtually separated concerns. In: Proc. int’l workshop on feature-oriented software development (FOSD). ACM Press, pp 17–24

  • Sillito J, Murphy G, De Volder K (2008) Asking and answering questions during a programming change task. IEEE Trans Softw Eng 34(4):434–451

    Article  Google Scholar 

  • Steiger JH, Lind J (1980) Statistically-based tests for the number of common factors. Presented at the annual meeting of the Psychometric Society

  • Tichy W (2000) Hints for reviewing empirical work in software engineering. Empir Software Eng 5(4):309–312

    Article  MathSciNet  Google Scholar 

  • von Mayrhauser A, Vans M (1995) Program comprehension during software maintenance and evolution. Computer 28(8):44–55

    Article  Google Scholar 

  • Westland C (2010) Lower bounds on sample size in structural equation modeling. Electron Commer Res Appl 9(6):476–487

    Article  Google Scholar 

Download references

Acknowledgements

We thank all the reviewers for their constructive feedback. We thank Jana Schumann for her support in the literature study and all experimenters for their support in setting up and conducting the experiment. Thanks to Veit Köppen for his support in the analysis of data. Siegmund’s work is supported by BMBF project 01IM10002B, Kästner’s work by ERC grant #203099, and Apel’s work by DFG projects AP 206/2, AP 206/4, and AP 206/5.

Author information

Authors and Affiliations

  1. University of Passau, Passau, Germany

    Janet Siegmund (born Feigenspan), Jörg Liebig & Sven Apel

  2. Carnegie Mellon University, Pittsburgh, PA, 15213, USA

    Christian Kästner

  3. University of Duisburg-Essen, Essen, Germany

    Stefan Hanenberg

Authors
  1. Janet Siegmund
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Christian Kästner
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jörg Liebig
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sven Apel
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Stefan Hanenberg
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Janet Siegmund.

Additional information

Communicated by: Michael Godfrey and Arie van Deursen

Appendix

Appendix

We show the results of an exploratory factor analysis, including question s.PE, which loads on factor experience with mainstream languages.

Table 13 Follow-up study: factor loadings of variables in questionnaire (including s.PE)
Full size table

Rights and permissions

Reprints and permissions

About this article

Cite this article

Siegmund, J., Kästner, C., Liebig, J. et al. Measuring and modeling programming experience . Empir Software Eng 19, 1299–1334 (2014). https://doi.org/10.1007/s10664-013-9286-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10664-013-9286-4

Keywords

Search

Navigation