Practical Experiences in the Design and Conduct of Surveys in Empirical Software Engineering

  • Marcus Ciolkowski
  • Oliver Laitenberger
  • Sira Vegas
  • Stefan Biffl
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2765)


A survey is an empirical research strategy for the collection of information from heterogeneous sources. In this way, survey results often exhibit a high degree of external validity. It is complementary to other empirical research strategies such as controlled experiments, which usually have their strengths in the high internal validity of the findings. While there is a growing number of (quasi-)controlled experiments reported in the software engineering literature, few results of large scale surveys have been reported there. Hence, there is still a lack of knowledge on how to use surveys in a systematic manner for software engineering empirical research.

This chapter introduces a process for preparing, conducting, and analyzing a software engineering survey. The focus of the work is on questionnaire-based surveys rather than literature surveys. The survey process is driven by practical experiences from two large-scale efforts in the review and inspection area. There are two main results from this work. First, the process itself allows researchers in empirical software engineering to follow a systematic, disciplined approach. Second, the experiences from applying the process help avoid common pitfalls that endanger both the research process and its results. We report on two (descriptive) surveys on software reviews that applied the survey process, and we present our experiences, as well as models for survey effort and duration factors derived from these experiences.


Software Engineering Software Engineer Software Industry Survey Process Exploratory Survey 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Hays, W.L.: Statistics for the social sciences. Holt, Rinehart and Winston, London (1977)Google Scholar
  2. 2.
    Robson, C.: Real World Research: A Resource for Social Scientists and Practitioners-Researchers. Blackwell, Malden (1993)Google Scholar
  3. 3.
    Freimut, B., Punter, T., Biffl, S., Ciolkowski, M.: State-of-the-Art in Empirical Studies, ViSEK Technical Report 007/E (2002) Google Scholar
  4. 4.
    Ticehurst, G., Veal, A.: Business Research Methods: A Managerial Approach. Addison Wesley Longman, Australia (1999)Google Scholar
  5. 5.
    Dybå, T.: Improvisation in small software organizations. IEEE Software 17(5), 82–87 (2000)CrossRefGoogle Scholar
  6. 6.
    Fagan, M.E.: Design and Code Inspections to Reduce Errors in Program Development. IBM Systems Journal 15(3), 182–211 (1976)CrossRefGoogle Scholar
  7. 7.
    Kim, L.P.W., Sauer, C., Jeffery, R.: A Framework for Software Development Technical Reviews. Software Quality and Productivity: Theory, Practice, Education and Training (1995) Google Scholar
  8. 8.
    Macdonald, F., Miller, J., Brooks, A., Roper, M., Wood, M.: Applying Inspection to Object-Oriented Software. Software Testing 6, 61–82 (1996)Google Scholar
  9. 9.
    Porter, A.A., Votta, L.G., Basili, V.R.: Comparing Detection Methods for Software Requirements Inspections: A Replicated Experiment. IEEE Transactions on Software Engineering 21(6), 563–575 (1995)CrossRefGoogle Scholar
  10. 10.
    Tjahjono, D.: Exploring the effectiveness of formal technical review factor with CSRS, a collaborative software review system, PhD thesis, Department of Information and Computer Science (1996) Google Scholar
  11. 11.
    Wheeler, D.A., Brykczynski, B.J., Meeson, R.N.: Software Peer Reviews. In: Thayer, R.H. (ed.) Software Engineering Project Management. IEEE Computer Society, Los Alamitos (1997)Google Scholar
  12. 12.
    Laitenberger, O., DeBaud, J.: An Encompassing Life-Cycle Centric Survey of Software Inspection. Journal of Systems and Software 50(1) (2000)Google Scholar
  13. 13.
    Freedman, D.P., Weinberg, G.M.: Handbook of Walkthroughs, Inspections, and Technical Reviews. Dorset House Publishing, New York (1990)Google Scholar
  14. 14.
    Weinberg, G.M., Freedman, D.P.: Reviews, Walkthroughs, and Inspections. IEEE Transactions on Software Engineering 12(1), 68–72 (1984)CrossRefGoogle Scholar
  15. 15.
    Yourdon, E.: Structured Walkthroughs, 4th edn. Prentice Hall, N.Y (1989)Google Scholar
  16. 16.
    Macdonald, F., Miller, J.: Modelling Software Inspection Methods for the Application of Tool Support, Technical Report RR–95–196 [EFoCS–16–95], University of Strathclyde, UK (1995) Google Scholar
  17. 17.
    Basili, V.R., Caldiera, G., Rombach, H.D.: Experience Factory. In: Marciniak, J.J. (ed.) Encyclopedia of Software Engineering, pp. 469–476. John Wiley & Sons, Chichester (1994)Google Scholar
  18. 18.
    Ropponen, J., Lyytinen, K.: Components of software development risk: how to address them? A project manager survey. IEEE Transactions on Software Engineering 26(6) (2000)Google Scholar
  19. 19.
    Basili, V.R., Caldiera, G., Rombach, H.D.: Measurement. In: Marciniak, J.J. (ed.) Encyclopedia of Software Engineering. John Wiley & Sons, Chichester (1994)Google Scholar
  20. 20.
    Oppenheim, A.: Questionnaire design, interviewing and attitude measurement. Pinter, London (1992)Google Scholar
  21. 21.
    Salant, P., Dillman, D.A.: How to conduct your own survey? John Wiley and Sons, New York (1994)Google Scholar
  22. 22.
    Paulk, M.C., Goldenson, D., White, D.M.: The 1999 Survey of High Maturity Organizations, Technical Report CMU/SEI–2000–SR–002, SEI (2000) Google Scholar
  23. 23.
    European Software Institute, 1995/1996 Software excellence study. Summary of results (1996) Google Scholar
  24. 24.
    Babbie, E.: Survey Research Methods. Wadsworth, Belmont (1990)Google Scholar
  25. 25.
    Kitchenham, B., Pfleeger, S., Pickard, L., Jones, P., Hoaglin, D., El Emam, K., Rosenberg, J.: Preliminary guidelines for empirical research in software engineering. IEEE Transactions on Software Engineering 28(8), 721–734 (2002)CrossRefGoogle Scholar
  26. 26.
    Ciolkowski, M., Shull, F., Biffl, S.: A Family of Experiments to Investigate the Influence of Context on the Effect of Inspection Techniques. Empirical Assessment of Software Engineering (EASE), Keele, UK (2002) Google Scholar
  27. 27.
    Laitenberger, O., Vegas, S., Ciolkowski, M.: The State of the Practice of Review and Inspection Technologies in Germany, Technical Report ViSEK/010/E, ViSEK (2002) Google Scholar
  28. 28.
    Laitenberger, O., Leszak, M., Stoll, D., Emam, K.E.: Evaluating a Model of Review Success Factors in an Industrial Setting. In: Proceedings of the International Symposium on Software Metrics (1999) Google Scholar
  29. 29.
    Davis, F., Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology. MIS Quarterly, pp. 319–340 (September 1989) Google Scholar
  30. 30.
    Assmann, D., Kempkens, R.: Tools for Measurement Support, Technical Report No. 97.00/E, Fraunhofer Institute for Experimental Software Engineering (December 2000) Google Scholar
  31. 31.
    Basili, V.R., Shull, F., Lanubile, F.: Building Knowledge through Families of Experiments. Transactions on Software Engineering 25(4), 456–473 (1999)CrossRefGoogle Scholar
  32. 32.
    GfK; Fraunhofer IESE; Fraunhofer ISI, Analyse und Evaluation der Softwareentwicklung in Deutschland, Eine Studie für das Bundesministerium für Bildung und Forschung (in German) (2000) Google Scholar
  33. 33.
    Ciolkowski, M., Kalmar, R.: Software-Reviews sind als Instrument zur Qualitätssicherung in der Industrie anerkannt. ViSEK Newsletter (2002)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2003

Authors and Affiliations

  • Marcus Ciolkowski
    • 1
  • Oliver Laitenberger
    • 2
  • Sira Vegas
    • 3
  • Stefan Biffl
    • 4
  1. 1.Dept. of Computer ScienceUniversität KaiserslauternKaiserslauternGermany
  2. 2.Droege & Comp. GmbH, Internationale Unternehmer-BeratungMünchenGermany
  3. 3.Facultad de InformáticaUniversidad Politécnica de MadridBoadilla del Monte, MadridSpain
  4. 4.TU Wien, Inst. f. Softwaretechnik und Interaktive SystemeWienAustria

Personalised recommendations