Empirical Software Engineering

, Volume 23, Issue 1, pp 452–489 | Cite as

Empirical software engineering experts on the use of students and professionals in experiments

  • Davide FalessiEmail author
  • Natalia Juristo
  • Claes Wohlin
  • Burak Turhan
  • Jürgen Münch
  • Andreas Jedlitschka
  • Markku Oivo


[Context] Controlled experiments are an important empirical method to generate and validate theories. Many software engineering experiments are conducted with students. It is often claimed that the use of students as participants in experiments comes at the cost of low external validity while using professionals does not. [Objective] We believe a deeper understanding is needed on the external validity of software engineering experiments conducted with students or with professionals. We aim to gain insight about the pros and cons of using students and professionals in experiments. [Method] We performed an unconventional, focus group approach and a follow-up survey. First, during a session at ISERN 2014, 65 empirical researchers, including the seven authors, argued and discussed the use of students in experiments with an open mind. Afterwards, we revisited the topic and elicited experts’ opinions to foster discussions. Then we derived 14 statements and asked the ISERN attendees excluding the authors, to provide their level of agreement with the statements. Finally, we analyzed the researchers’ opinions and used the findings to further discuss the statements. [Results] Our survey results showed that, in general, the respondents disagreed with us about the drawbacks of professionals. We, on the contrary, strongly believe that no population (students, professionals, or others) can be deemed better than another in absolute terms. [Conclusion] Using students as participants remains a valid simplification of reality needed in laboratory contexts. It is an effective way to advance software engineering theories and technologies but, like any other aspect of study settings, should be carefully considered during the design, execution, interpretation, and reporting of an experiment. The key is to understand which developer population portion is being represented by the participants in an experiment. Thus, a proposal for describing experimental participants is put forward.


Experimentation Threats to validity Generalization Subjects of experiments Participants in experiments 



We would like to thank all the ISERN 2014 participants for the inspiring and energetic discussions. We would like to thank both the anonymous experts and the following non-anonymous experts for participating in the survey: Paris Avgeriou, Teresa Baldassarre, Victor Basili, Giovanni Cantone, Jeff Carver, Tore Dybå, Hakan Erdogmus, Vladimir Mandic, Manuel Mastrofini, Daniel Mendez, Oscar Pastor, Guilherme Horta Travassos, Stephan Wagner, Qing Wang, Roel Wieringa, and Dietmar Winkler. We thank Sonnhild Namingha for proof reading the manuscript. This research is supported in part by the Academy of Finland Project 278354.


  1. Basili VR, Reiter RW (May 1981) A controlled experiment quantitatively comparing software development approaches. IEEE Trans Softw Eng SE-7(3):299–320CrossRefGoogle Scholar
  2. Basili VR, Selby RW (Dec. 1987) Comparing the effectiveness of software testing strategies. IEEE Trans Softw Eng SE-13(12):1278–1296CrossRefGoogle Scholar
  3. Basili, V. R. McGarry, F. E. Pajerski, R. and Zelkowitz, M. V. 2002 “Lessons learned from 25 years of process improvement: The rise and fall of the NASA software engineering laboratory,” in Proceedings - International Conference on Software Engineering, pp. 69–79.Google Scholar
  4. Beecham S, Hall T, Rainer A (Mar. 2003) Software process improvement problems in twelve software companies: an empirical analysis. Empir Softw Eng 8(1):7–42CrossRefzbMATHGoogle Scholar
  5. Berander, P. 2004 “Using students as subjects in requirements prioritization,” in Proceedings - 2004 International Symposium on Empirical Software Engineering, ISESE 2004, pp. 167–176.Google Scholar
  6. Berg K. E. and Latin, R. W. 2003 Essentials of research methods in health, Physical Education, Exercise Science, and Recreation. LWW.Google Scholar
  7. Bergersen GR, Sjoberg DIK, Dyba T (2014) Construction and validation of an instrument for measuring programming skill. IEEE Trans Softw Eng 40(12):1163–1184CrossRefGoogle Scholar
  8. Bratthall, L. Johansson, E. and Regnell, B. 2000 “Is a Design Rationale Vital when Predicting Change Impact? – A Controlled Experiment on Software,” in 2nd International Conference on Product Focused Software Process Improvement (PROFES 2000),pp. 126–139.Google Scholar
  9. Ciolkowski, M. 2009 “What do we know about perspective-based reading? An approach for quantitative aggregation in software engineering,” in 2009 3rd International Symposium on Empirical Software Engineering and Measurement, ESEM 2009, pp. 133–144.Google Scholar
  10. Easton G, Easton A, Belch M (Sep. 2003) An experimental investigation of electronic focus groups. Inf Manag 40(8):717–727CrossRefGoogle Scholar
  11. Fagerholm, F. Oza, N. and Munch, J. 2013 “A platform for teaching applied distributed software development: The ongoing journey of the Helsinki software factory,” in 2013 3rd International Workshop on Collaborative Teaching of Globally Distributed Software Development, CTGDSD 2013 - Proceedings,pp. 1–5.Google Scholar
  12. Gorschek T, Garre P, Larsson S, Wohlin C (2006) A model for technology transfer in practice. IEEE Softw 23(6):88–95CrossRefGoogle Scholar
  13. Höst, M. Regnell, B. and Wohlin, C. 2000 “Using Students as Subjects—A Comparative Study of Students and Professionals in Lead-Time Impact Assessment,” in Empir Softw Eng, vol. 5, pp. 201–214.Google Scholar
  14. Jedlitschka A. and Pfahl, D. 2005 “Reporting guidelines for controlled experiments in software engineering,” in 2005 International Symposium on Empirical Software Engineering, ISESE 2005, pp. 95–104.Google Scholar
  15. Juristo N. and Moreno, A. M. 2001 Basics of Software Engineering Experimentation, vol. 5/6.Google Scholar
  16. Jyrki, K. Bragge, J. and Lehtola, L. 2008 “The Focus Group Method as an Empirical Tool in Software Engineering,” in Guide to Advanced Empirical Software Engineering, pp. 93–116.Google Scholar
  17. Kontio, J. Lehtola, L. and Bragge, J. 2004 “Using the Focus Group Method in Software Engineering: Obtaining Practitioner and User Experiences,” in International Symposium on Empirical Software Engineering, pp. 271–280.Google Scholar
  18. Lethbridge TC, Sim SE, Singer J (Jul. 2005) Studying software engineers: data collection techniques for software field studies. Empir Softw Eng 10(3):311–341CrossRefGoogle Scholar
  19. Lindvall, M. Basili, V. R. Boehm, B. W. Costa, P. Dangle, K. Shull, F. Tesoriero, R. Williams, L. A. and Zelkowitz, M. V. 2002“Empirical Findings in Agile Methods,” in Second XP Universe and First Agile Universe Conference on Extreme Programming and Agile Methods - XP/Agile Universe 2002,pp. 197–207.Google Scholar
  20. M.H. and Wohlin, C. M. C. Gustavsson, A. 1996 “A Framework for Technology Introduction in Software Organizations,” in Software Process Improvement Conference,pp. 167–176.Google Scholar
  21. McMeekin, D. A. Von Konsky, B. R. Robey, M. and Cooper, D. J. A. 2009 “The significance of participant experience when evaluating software inspection techniques,” in Proceedings of the Australian Software Engineering Conference, ASWEC, pp. 200–209.Google Scholar
  22. Morgan DL (Aug. 1996) Focus groups. Annu Rev Sociol 22(1):129–152CrossRefGoogle Scholar
  23. Morgan D. L. and Krueger, R. A. 1993 “When to use focus groups and why.,” in Successful focus groups: Advancing the state of the art., pp. 3–19.Google Scholar
  24. Porter A, Votta L (1998) Comparing detection methods for software requirements inspections: a replication using professional subjects. Empir Softw Eng 3:355–379CrossRefGoogle Scholar
  25. Porter AA, Votta LG, Basili VR (1995) Comparing detection methods for software requirements inspections: a replicated experiment. IEEE Trans Softw Eng 21:563–575CrossRefGoogle Scholar
  26. Remus, W. 1989 “Using students as subjects in experiments on decision supportsystems,” [1989] Proc. Twenty-Second Annu. Hawaii Int. Conf. Syst. Sci. Vol. III Decis. Support Knowl. Based Syst. Track, vol. 3.Google Scholar
  27. Ricca F, Di Penta M, Torchiano M, Tonella P, Ceccato M (2010) How developers’ experience and ability influence web application comprehension tasks supported by UML stereotypes: a series of four experiments. IEEE Trans Softw Eng 36:96–118CrossRefGoogle Scholar
  28. Runeson, P. 2003 “Using students as experiment subjects–an analysis on graduate and freshmen student data,” Proc. 7th Int. Conf. Empir. Assess. Eval. Softw. Eng., pp. 95–102.Google Scholar
  29. Salman, I. Misirli, A. T. and Juristo, N. 2015 “Are Students Representatives of Professionals in Software Engineering Experiments?,” in International Conference on Software Engineering (ICSE), pp. 666–676.Google Scholar
  30. Salviulo F. and Scanniello, G. 2014 “Dealing with Identifiers and Comments in Source Code Comprehension and Maintenance: Results from an Ethnographically-informed Study with Students and Professionals,” in Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering, p. 48:1--48:10.Google Scholar
  31. Siegmund J, Siegmund N, Apel S (2015) Views on internal and external validity in empirical software engineering. In: International Conference on software engineering (ICSE)Google Scholar
  32. Sjoberg DIK, Anda B, Arisholm E, Dyba T, Jorgensen M, Karahasanovic A, Koren EF, Vokac M (2002) Conducting realistic experiments in software engineering. Proc. Int. Symp. Empir. Softw. EngGoogle Scholar
  33. Svahnberg, M. Aurum, A. and Wohlin, C. 2008 “Using students as subjects - an empirical evaluation,” in Proceedings of the Second ACMIEEE international symposium on Empirical software engineering and measurement, pp. 288–290.Google Scholar
  34. Tremblay, M. Hevner, A. and Berndt, D. 2010 “Focus Groups for Artifact Refinement and Evaluation in Design Research,” Communications of the Association for Information Systems, vol. 26, no. 1.Google Scholar
  35. Turhan B. and Bener, A. 2007 “A template for real world team projects for highly populated software engineering classes,” in Proceedings - International Conference on Software Engineering,pp. 748–751.Google Scholar
  36. Vegas, S. Dieste, O. and Juristo, N. 2015 “Difficulties in Running Experiments in the Software Industry: Experiences from the Trenches,” in Proceedings - 3rd International Workshop on Conducting Empirical Studies in Industry, CESI 2015, pp. 3–9.Google Scholar
  37. Wesslén A (2000) Replicated empirical study of the impact of the methods in the PSP on individual engineers. Empir Softw Eng 5:93–123CrossRefzbMATHGoogle Scholar
  38. Wohlin C, Runeson P, Höst M, Ohlsson MC, Regnell B, Wesslén A (2012) Experimentation in software engineering: an introduction. SpringerGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2017

Authors and Affiliations

  1. 1.California Polytechnic State UniversitySan Luis ObispoUSA
  2. 2.Universidad Politécnica de MadridMadridSpain
  3. 3.Blekinge Institute of TechnologyKarlskronaSweden
  4. 4.Brunel University LondonLondonUK
  5. 5.University of HelsinkiHelsinkiFinland
  6. 6.Reutlingen UniversityReutlingenGermany
  7. 7.Fraunhofer Institute for Experimental Software EngineeringKaiserslauternGermany
  8. 8.University of OuluOuluFinland

Personalised recommendations