Abstract
While being an important and often used research method, survey research has been less often discussed on a methodological level in empirical software engineering than other types of research. This chapter compiles a set of important and challenging issues in survey research based on experiences with several large-scale international surveys. The chapter covers theory building, sampling, invitation and follow-up, statistical as well as qualitative analysis of survey data and the usage of psychometrics in software engineering surveys.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
AERA, APA, NCME (2014) Standards for educational and psychological testing. American Educational Research Association, Washington
Amrhein V, Greenland S, McShane B (2019) Retire statistical significance. Nature 567:305–307
Baltes S, Diehl S (2016) Worse than spam: issues in sampling software developers. In: Proceedings of the 10th ACM/IEEE international symposium on empirical software engineering and measurement, ESEM ’16. ACM, New York, pp 52:1–52:6. http://doi.acm.org/10.1145/2961111.2962628
Binning JF (2016) Construct. https://www.britannica.com/science/construct
Birks M, Mills J (2011) Grounded theory: a practical guide. Sage, Thousand Oaks
Bourque P, Fairley RE et al (2014) Guide to the software engineering body of knowledge (SWEBOK): version 3.0. IEEE Computer Society Press, Washington
Campbell DT, Fiske DW (1959) Convergent and discriminant validation by the multitrait-multimethod matrix. Psychol Bull 56(2):81–105
Charmaz K (2014) Constructing grounded theory. Sage, Thousand Oaks
Ciolkowski M, Laitenberger O, Vegas S, Biffl S (2003) Practical experiences in the design and conduct of surveys in empirical software engineering. In: Conradi R, Wang AI (eds) Empirical methods and studies in software engineering, experiences from ESERNET, vol 2765. Lecture notes in computer science. Springer, Berlin, pp 104–128. https://doi.org/10.1007/978-3-540-45143-3_7
Coaley K (2014) An introduction to psychological assessment and psychometrics. Sage, Thousand Oaks
Cochran WG (1977) Sampling techniques. Wiley, New York
Cohen RJ, Swerdlik ME, Phillips SM (1995) Psychological testing and assessment: an introduction to tests and measurement. Mayfield Publishing, California
Cruz S, da Silva FQ, Capretz LF (2015) Forty years of research on personality in software engineering: a mapping study. Comput Hum Behav 46:94–113
de Mello RM, da Silva PC, Travassos GH (2015) Investigating probabilistic sampling approaches for large-scale surveys in software engineering. J Softw Eng Res Dev 3(1):8. https://doi.org/10.1186/s40411-015-0023-0
DiCiccio TJ, Efron B (1996) Bootstrap confidence intervals. Stat Sci 11(3):189–228
Feldt R, Torkar R, Angelis L, Samuelsson M (2008) Towards individualized software engineering: empirical studies should collect psychometrics. In: Cheng L, Sillito J, Storey MD, Tessem B, Venolia G, de Souza CRB, Dittrich Y, John M, Hazzan O, Maurer F, Sharp H, Singer, J, Sim SE (eds) Proceedings of the 2008 international workshop on cooperative and human aspects of software engineering, CHASE 2008, Leipzig. ACM, New York, pp 49–52. https://doi.org/10.1145/1370114.1370127
Fowler FJ (2013) Survey research methods. Sage, Thousand Oaks
Freedman D, Pisani R, Purves R (2007). Statistics. Norton, New York
Ghazi AN, Petersen K, Reddy SS, Nekkanti H (2019) Survey research in software engineering: problems and mitigation strategies. IEEE Access 7:24703–24718
Glaser BG (1992) Basics of grounded theory analysis: emergence vs. forcing. Sociology Press, Mill Valley
Glaser BG, Strauss AL (1967) Discovery of grounded theory: strategies for qualitative research. Aldine de Gruyter, New York
Graziotin D, Fagerholm F (2019) Happiness and the productivity of software engineers. In: Rethinking productivity in software engineering. Apress, Berkeley, pp 109–124
Graziotin D, Wang X, Abrahamsson P (2015) Understanding the affect of developers: theoretical background and guidelines for psychoempirical software engineering. In: Proceedings of the 7th international workshop on social software engineering, SSE 2015. ACM, New York, pp 25–32. http://doi.acm.org/10.1145/2804381.2804386
Graziotin D, Fagerholm F, Wang X, Abrahamsson P (2017) On the unhappiness of software developers. In: Mendes E, Counsell S, Petersen K (eds) Proceedings of the 21st international conference on evaluation and assessment in software engineering. ACM Press, New York, pp 324–333
Graziotin D, Fagerholm F, Wang, Abrahamsson P (2018) What happens when software developers are (un)happy. J Syst Softw 140:32–47
Gregor S (2006) The nature of theory in information systems. MIS Q 30(3):611–642. http://misq.org/the-nature-of-theory-in-information-systems.html
Gren L (2018) Standards of validity and the validity of standards in behavioral software engineering research. In: Standards of validity and the validity of standards in behavioral software engineering research. ACM Press, New York
Hannay JE, Sjøberg DI, Dybå T (2007) A systematic review of theory use in software engineering experiments. IEEE Trans Softw Eng 33(2):87–107
Hogan R (2017) Personality and the fate of organizations. Erlbaum, Mahwah
Inayat I, Salim SS, Marczak S, Daneva M, Shamshirband S (2015) A systematic literature review on agile requirements engineering practices and challenges. Comput Hum Behav 51:915–929
Kalinowski M, Card DN, Travassos GH (2012) Evidence-based guidelines to defect causal analysis. IEEE Softw 29(4):16–18
Kalinowski M, Curty P, Paes A, Ferreira A, Spínola RO, Fernández DM, Felderer M, Wagner S (2017) Supporting defect causal analysis in practice with cross-company data on causes of requirements engineering problems. In: Proceedings of the 39th IEEE/ACM international conference on software engineering: software engineering in practice track, ICSE-SEIP 2017, Buenos Aires. IEEE Computer Society, Silver Spring, pp 223–232. https://doi.org/10.1109/ICSE-SEIP.2017.14
Kass RE (2011) Statistical inference: the big picture. Stat Sci Rev J Inst Math Stat 26(1):1
Kasunic M (2005) Designing an effective survey. Technical report, Carnegie-Mellon University, Pittsburgh, PA and Software Engineering Institute
Kitchenham BA, Pfleeger SL (2002a) Principles of survey research part 2: designing a survey. ACM SIGSOFT Softw Eng Notes 27(1):18–20. https://doi.org/10.1145/566493.566495
Kitchenham BA, Pfleeger SL (2002b) Principles of survey research: part 3: constructing a survey instrument. ACM SIGSOFT Softw Eng Notes 27(2):20–24. https://doi.org/10.1145/511152.511155
Kitchenham, BA, Pfleeger SL (2002c) Principles of survey research part 4: questionnaire evaluation. ACM SIGSOFT Softw Eng Notes 27(3):20–23. https://doi.org/10.1145/638574.638580
Kitchenham BA, Pfleeger SL (2002d) Principles of survey research: part 5: populations and samples. ACM SIGSOFT Softw Eng Notes 27(5):17–20. https://doi.org/10.1145/571681.571686
Kitchenham BA, Pfleeger SL (2008) Personal opinion surveys. In: Guide to advanced empirical software engineering. Springer, Berlin, pp 63–92
Kline P (2015) A handbook of test construction (psychology revivals): introduction to psychometric design. Routledge, London
Lenberg P, Feldt R, Wallgren LG (2015) Behavioral software engineering: a definition and systematic literature review. J Syst Softw 107:15–37
Levine TR, Weber R, Hullett C, Park HS, Massi Lindsey LL (2008) A critical assessment of null hypothesis significance testing in quantitative communication research. Hum Commun Res 34:171–187
Malhotra MK, Grover V (1998) An assessment of survey research in POM: from constructs to theory. J Oper Manag 16(4):407–425
Mannio M, Nikula U (2001) Requirements elicitation using a combination of prototypes and scenarios. Technical report, Telecom Business Research Center Lappeenranta
Méndez Fernández D, Passoth J-H (2018) Empirical software engineering: from discipline to interdiscipline. J Syst Softw 148:170–179
Méndez Fernández D, Wagner S (2015) Naming the pain in requirements engineering: a design for a global family of surveys and first results from Germany. Inform Softw Tech 57:616–643
Méndez Fernández D, Wagner S, Kalinowski M, Schekelmann A, Tuzcu A, Conte T, Spinola R, Prikladnicki R (2015) Naming the pain in requirements engineering: comparing practices in Brazil and Germany. IEEE Softw 32(5):16–23
Méndez Fernández D, Wagner S, Kalinowski M, Felderer M, Mafra P, Vetrò A, Conte T, Christiansson M-T, Greer D, Lassenius C et al. (2017) Naming the pain in requirements engineering—contemporary problems, causes, and effects in practice. Empir Softw Eng 22(5):2298–2338
Méndez Fernández D, Tießler M, Kalinowski M, Felderer M, Kuhrmann M (2018) On evidence-based risk management in requirements engineering. In: International conference on software quality. Springer, Berlin, pp 39–59
Molléri JS, Petersen K, Mendes E (2019) CERSE-catalog for empirical research in software engineering: a systematic mapping study. Inform Softw Tech 105:117–149
Pfleeger SL, Kitchenham BA (2001) Principles of survey research: part 1: turning lemons into lemonade. ACM SIGSOFT Softw Eng Notes 26(6):16–18. https://doi.org/10.1145/505532.505535
Pinsonneault A, Kraemer K (1993) Survey research methodology in management information systems: an assessment. J Manag Inform Syst 10(2):75–105
Pittenger DJ (1993) Measuring the MBTI…and coming up short. J Career Plan Employ 54(1):48–52
Runeson P, Höst M, Rainer A, Regnell B (2012) Case study research in software engineering. Wiley, London
Rust J (2009) Modern psychometrics: the science of psychological assessment. Routledge, Hove, East Sussex New York
Sjøberg DI, Dybå T, Anda BC, Hannay JE (2008) Building theories in software engineering. In: Guide to advanced empirical software engineering. Springer, Berlin, pp 312–336
Sommerville I, Sawyer P, Viller S (1998) Viewpoints for requirements elicitation: a practical approach. In: Proceedings of the 3rd international conference on requirements engineering (ICRE ’98), Putting requirements engineering to practice, Colorado Springs. IEEE Computer Society, Silver Spring, pp 74–81. https://doi.org/10.1109/ICRE.1998.667811
Stol K-J, Fitzgerald B (2015) Theory-oriented software engineering. Sci Comput Program 101:79–98
Stol K, Ralph P, Fitzgerald B (2016) Grounded theory in software engineering research: a critical review and guidelines. In: Dillon LK, Visser W, Williams L (eds) Proceedings of the 38th international conference on software engineering, ICSE 2016, Austin. ACM, New York, pp 120–131. https://doi.org/10.1145/2884781.2884833
Strauss A, Corbin J (1990) Basics of qualitative research. Sage, Thousand Oaks
Torchiano M, Fernández DM, Travassos GH, de Mello RM (2017) Lessons learnt in conducting survey research. In: Proceedings of the 5th IEEE/ACM international workshop on conducting empirical studies in industry, CESI@ICSE 2017, Buenos Aires. IEEE, Piscataway, pp 33–39. https://doi.org/10.1109/CESI.2017.5
Usman M, Britto R, Börstler J, Mendes E (2017) Taxonomies in software engineering: a systematic mapping study and a revised taxonomy development method. Inform Softw Tech 85:43–59
Wagner S, Méndez Fernández D, Felderer M, Vetrò A, Kalinowski M, Wieringa R, Pfahl D, Conte T, Christiansson M-T, Greer D, Lassenius C, Männistö T, Nayebi M, Oivo M, Penzenstadler B, Prikladnicki R, Ruhe G, Schekelmann A, Sen S, Spínola R, Tuzcu A, De La Vara JL, Winkler D (2019) Status quo in requirements engineering: a theory and a global family of surveys. ACM Trans Softw Eng Methodol. 28(2):9:1–9:48
Wohlin C, Runeson P, Höst M, Ohlsson MC, Regnell B, Wesslén A (2012) Experimentation in software engineering. Springer, Berlin
Yamane T (1973) Statistics: an introductory analysis. Longman, New York
Acknowledgement
We are grateful to all collaborating researchers in the NaPiRE initiative.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Wagner, S., Mendez, D., Felderer, M., Graziotin, D., Kalinowski, M. (2020). Challenges in Survey Research. In: Felderer, M., Travassos, G. (eds) Contemporary Empirical Methods in Software Engineering. Springer, Cham. https://doi.org/10.1007/978-3-030-32489-6_4
Download citation
DOI: https://doi.org/10.1007/978-3-030-32489-6_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-32488-9
Online ISBN: 978-3-030-32489-6
eBook Packages: Computer ScienceComputer Science (R0)