Empirical Software Engineering

, Volume 15, Issue 6, pp 599–617 | Cite as

Measuring fidelity to extreme programming: a psychometric approach

  • George MichaelidesEmail author
  • Chris Thomson
  • Stephen Wood


This study assesses the Shodan survey as an instrument for measuring an individual’s or a team’s adherence to the extreme programming (XP) methodology. Specifically, we hypothesize that the adherence to the XP methodology is not a uni-dimensional construct as presented by the Shodan survey but a multidimensional one reflecting dimensions that are theoretically grounded in the XP literature. Using data from software engineers in the University of Sheffield’s Software Engineering Observatory, two different models were thus tested and compared using confirmatory factor analysis: a uni-dimensional model and a four-dimensional model. We also present an exploratory analysis of how these four dimensions affect students’ grades. The results indicate that the four-dimensional model fits the data better than the uni-dimensional one. Nevertheless, the analysis also uncovered flaws with the Shodan survey in terms of the reliability of the different dimensions. The exploratory analysis revealed that some of the XP dimensions had linear or curvilinear relationship with grades. Through validating the four-dimensional model of the Shodan survey this study highlights how psychometric techniques can be used to develop software engineering metrics of fidelity to agile or other software engineering methods.


Psychometrics Confirmatory factor analysis Extreme Programming (XP) Fidelity Adherence Shodan survey 



This work was supported by an EPSRC grant: EP/D031516 - the Sheffield Software Engineering Observatory.


  1. Abrahamsson P (2003) Extreme programming: first results from a controlled case study. Euromicro Conference 2003. Proceeding, 29th, pp 259–266Google Scholar
  2. Abrahamsson P, Koskela J (2004) Extreme programming: a survey of empirical data from a controlled case study. Proceedings 2004 International Symposium on Empirical Software Engineering, pp 73–82Google Scholar
  3. Bahli B, Zeid ESA (2005) The role of knowledge creation in adopting extreme programming model: an empirical study. ITI 3rd International Conference on Information and Communications Technology 2005: Enabling Technologies for the New Knowledge SocietyGoogle Scholar
  4. Bartholomew DJ, Knott M (1999) Latent variable models and factor analysis. Oxford University Press, New YorkzbMATHGoogle Scholar
  5. Beck K, Andres C (2004) Extreme programming explained: embrace change, 2nd edn. Addison-Wesley Professional, BostonGoogle Scholar
  6. Cao L, Mohan K, Xu P, Ramesh B (2004) How extreme does extreme programming have to be? Adapting XP practices to large-scale projects. Proceedings of the 37th Annual Hawaii International Conference on System Sciences 2004Google Scholar
  7. Chong J (2005) Social behaviors on XP and non-XP teams: A comparative study. In Proc. Agile United ConferenceGoogle Scholar
  8. Cohen J (1988) Statistical power analysis for the behavioral sciences. Lawrence Erlbaum Associates, New YorkzbMATHGoogle Scholar
  9. Cronbach LJ (1951) Coefficient alpha and the internal structure of tests. Psychometrika 16:297–334CrossRefGoogle Scholar
  10. Fruhling A, Tyser L, De Vreede GJ (2005) Experiences with extreme programming in telehealth: Developing and implementing a biosecurity health care application. HICSS ′05. Proceedings of the 38th Annual Hawaii International Conference on System SciencesGoogle Scholar
  11. Gittings R, Hope S (2001) A study of human solutions in extreme programming. Proc. 13th Workshop of the Psychology of Programming Group, pp 41–51Google Scholar
  12. James LR, Demaree RG, Wolf G (1984) Estimating within-group interrater reliability with and without response bias. J Appl Psychol 69:85–98CrossRefGoogle Scholar
  13. Jokela T, Abrahamsson P (2004) Usability assessment of an extreme programming project: close co-operation with the customer does not equal to good usability. Product Focused Software Process Improvement, pp 397–407Google Scholar
  14. Koskela J, Abrahamsson P (2004) On-site customer in an XP project: empirical results from a case study. Softw Process Improv 3281:1–11CrossRefGoogle Scholar
  15. Krebs W (2002) Turning the knobs: a coaching pattern for XP through agile metrics. Presented at Extreme Programming/Agile Universe, Chicago IL, pp 60–69Google Scholar
  16. Layman L (2004) Empirical investigation of the impact of extreme programming practices on software projects. OOPSLA ′04: Companion to the 19th annual ACM SIGPLAN conference on Object-oriented programming systems, languages, and applications. Vancouver, Canada, ACMGoogle Scholar
  17. Layman L, Williams L, Cunningham L (2004) Exploring extreme programming in context: an industrial case study. Agile Development Conference 2004Google Scholar
  18. Layman L, Williams L, Cunningham L (2006) Motivations and measurements in an agile case study. J Systems Archit 52(11):654–667CrossRefGoogle Scholar
  19. Macias F (2004) Empirical assessment of extreme programming. Unpublished PhD thesis, University of SheffieldGoogle Scholar
  20. Mackenzie A, Monk S (2004) From cards to code: how extreme programming re-embodies programming as a collective practice. Comput Support Coop Work (CSCW) 13:91–117CrossRefGoogle Scholar
  21. Mannaro K, Melis M, Marchesi M (2004) Empirical analysis on the satisfaction of it employees comparing XP practices with other software development methodologies. Extreme Programming and Agile Processes in Software Engineering, Lecture Notes in Computer Science, 3092/2004: 166–174, Springer Berlin/HeidelbergGoogle Scholar
  22. Martin A, Biddle R, Noble J (2004a) When XP met outsourcing. Extreme programming and agile processes in software engineering, lecture notes in computer science, 3092/2004: 51–59, Springer Berlin / HeidelbergGoogle Scholar
  23. Martin A, Biddle R, Noble J (2004b) The XP customer role in practice: three studies. Agile Development Conference 2004Google Scholar
  24. Merisalo-Rantanen H, Tuure T, Matti R (2005) Is extreme programming just old wine in new bottles: a comparison of two cases. J Database Manage 16:41–61Google Scholar
  25. Moser R, Scotto M, Sillitti A, Succi G (2007) Does XP deliver quality and maintainable code? Agile processes in software engineering and extreme programming, lecture notes in computer science, 4536/2007: 105–114, Springer Berlin / HeidelbergGoogle Scholar
  26. Muller M, Tichy W (2001) Case study: extreme programming in a university environment. ICSE ′01: Proceedings of the 23rd International Conference on Software Engineering. Toronto, Canada, IEEE Computer SocietyGoogle Scholar
  27. Muthén BO (1994) Multilevel covariance structure analysis. Sociol Methods Res 22:376–398CrossRefGoogle Scholar
  28. Newkirk J, Martin R (2000) Extreme programming in practice. OOPSLA ′00: Addendum to the 2000 proceedings of the conference on Object-oriented programming, systems, languages, and applications (Addendum). Minneapolis, ACMGoogle Scholar
  29. Noll J, Atkinson D (2003) Comparing extreme programming to traditional development for student projects: a case study. Extreme programming and agile processes in software engineering, lecture notes in computer science, 2675/2003: 1013, Springer Berlin / HeidelbergGoogle Scholar
  30. Nunnaly JC, Bernstein IH (1994) Psychometric theory, 3rd edn. McGraw-Hill, New YorkGoogle Scholar
  31. R Development Core Team (2008) R: A language and environment for statistical computing. Series in psychology. R Foundation for Statistical Computing, Vienna, AustriaGoogle Scholar
  32. Robinson H, Sharp H (2004) The characteristics of XP teams. Extreme programming and agile processes in software engineering, lecture notes in computer science, 3092/2004: 139–147, Springer Berlin / HeidelbergGoogle Scholar
  33. Robinson H, Sharp H (2005a) Organisational culture and XP: Three case studies. Proceedings Agile Conference 2005Google Scholar
  34. Robinson H, Sharp H (2005b) The social side of technical practices. Extreme programming and agile processes in software engineering, lecture notes in computer science, 3556/2005: 100–108, Springer Berlin / HeidelbergGoogle Scholar
  35. Sfetsos P, Angelis L, Stalmenos I (2006) Investigating the extreme programming system: an empirical study. Empir Softw Eng 11:269–301CrossRefGoogle Scholar
  36. Sharp H, Robinson H (2004) An ethnographic study of XP practice. Empir Softw Eng 9:353–375CrossRefGoogle Scholar
  37. Stephens M, Rosenberg D (2003) Extreme programming refactored: the case against XP. APress, BerkeleyGoogle Scholar
  38. Syed-Abdullah S (2005) Empirical study on extreme programming. Unpublished PhD thesis, University of SheffieldGoogle Scholar
  39. Tessem BR (2003) Experiences in learning XP practices: A qualitative study. Extreme programming and agile processes in software engineering, lecture notes in computer science, 2675/2003:1012, Springer Berlin / HeidelbergGoogle Scholar
  40. Thomson C, Holcombe M (2009) The Sheffield software engineering observatory archive: six years of empirical data collected from 73 complete projects. CS-09-01. Department of Computer Science, University of SheffieldGoogle Scholar
  41. Williams L, Krebs W, Layman L (2004a) Extreme programming evaluation framework for object-oriented languages–version 1.4. Technical report, NCSUGoogle Scholar
  42. Williams L, Krebs W, Layman L (2004b) Extreme programming evaluation framework for object-oriented languages–version 1.3. Technical report, NCSUGoogle Scholar
  43. Young M, Edwards H, McDonald S, Thompson B (2005) Personality characteristics in an XP team: A repertory grid study. HSSE ′05: Proceedings of the 2005 workshop on human and social factors of software engineering. St. Louis MO, ACMGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2010

Authors and Affiliations

  • George Michaelides
    • 1
    Email author
  • Chris Thomson
    • 2
  • Stephen Wood
    • 3
  1. 1.Division of PsychologyNottingham Trent UniversityNottinghamUK
  2. 2.Business SchoolThe University of HullHullUK
  3. 3.School of ManagementUniversity of LeicesterLeicesterUK

Personalised recommendations