Skip to main content
SpringerLink
Log in

The Evolution of Empirical Methods in Software Engineering

  • Chapter
  • First Online:
Contemporary Empirical Methods in Software Engineering

Abstract

Empirical methods like experimentation have become a powerful means to drive the field of software engineering by creating scientific evidence on software development, operation, and maintenance, but also by supporting practitioners in their decision-making and learning. Today empirical methods are fully applied in software engineering. However, they have developed in several iterations since the 1960s. In this chapter we tell the history of empirical software engineering and present the evolution of empirical methods in software engineering in five iterations, i.e., (1) mid-1960s to mid-1970s, (2) mid-1970s to mid-1980s, (3) mid-1980s to end of the 1990s, (4) the 2000s, and (5) the 2010s. We present the five iterations of the development of empirical software engineering mainly from a methodological perspective and additionally take key papers, venues, and books, which are covered in chronological order in a separate section on recommended further readings, into account. We complement our presentation of the evolution of empirical software engineering by presenting the current situation and an outlook in Sect. 4 and the available books on empirical software engineering. Furthermore, based on the chapters covered in this book we discuss trends on contemporary empirical methods in software engineering related to the plurality of research methods, human factors, data collection and processing, aggregation and synthesis of evidence, and impact of software engineering research.

Guilherme Horta Travassos is a CNPq Researcher.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.00
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  • Akiyama F (1971) An example of software system debugging. In: IFIP congress (1), vol 71. North-Holland, Amsterdam, pp 353–359

    Google Scholar 

  • Araújo MAP, Monteiro VF, Travassos GH (2012) Towards a model to support studies of software evolution. In: Proceedings of the ACM-IEEE international symposium on empirical software engineering and measurement (ESEM ’12). ACM, New York, pp 281–290

    Google Scholar 

  • Arcuri A, Briand L (2014) A hitchhiker’s guide to statistical tests for assessing randomized algorithms in software engineering. Softw Test Verification Reliab 24(3):219–250

    Google Scholar 

  • Baros MO, Werner CML, Travassos GH (2004) Supporting risks in software project management. J Syst Softw 70(1):21–35

    Google Scholar 

  • Basili VR (1993) The experimental paradigm in software engineering. In: Experimental software engineering issues: critical assessment and future directions. Springer, Berlin, pp 1–12

    Google Scholar 

  • Basili VR, Weiss DM (1984) A methodology for collecting valid software engineering data. IEEE Trans Softw Eng SE-10(6):728–738

    Google Scholar 

  • Basili VR, Zelkowitz MV (2007) Empirical studies to build a science of computer science. Commun Assoc Comput Mach 50(11):33–37

    Google Scholar 

  • Basili VR, Caldiera G, Rombach HD (1994) Experience factory. Encycl Softw Eng 1:469–476

    Google Scholar 

  • Basili VR, Shull F, Lanubile F (1999) Building knowledge through families of experiments. IEEE Trans Softw Eng 25(4):456–473

    Google Scholar 

  • Basili V, Rombach D, Schneider K, Kitchenham B, Pfahl D, Selby R (2007) Empirical software engineering issues. In: Critical assessment and future directions: international workshop, Dagstuhl Castle, June 26–30, 2006, Revised Papers, vol 4336. Springer, Berlin

    Google Scholar 

  • ben Othmane L, Jaatun MG, Weippl E (2017) Empirical research for software security: foundations and experience. CRC Press, Boca Raton

    Google Scholar 

  • Bertin E (1978) Qualitative and semiquantitative analysis. Springer, Berlin, pp 435–457

    Google Scholar 

  • Biolchini MPNATG J (2005) Systematic review in software engineering: relevance and utility. Technical report

    Google Scholar 

  • Bird C, Murphy B, Nagappan N, Zimmermann T (2011) Empirical software engineering at Microsoft research. In: Proceedings of the ACM 2011 conference on computer supported cooperative work. ACM, New York, pp 143–150

    Google Scholar 

  • Bird C, Menzies T, Zimmermann T (2015) The art and science of analyzing software data. Elsevier, Amsterdam

    Google Scholar 

  • Bjarnason E, Smolander K, Engström E, Runeson P (2016) A theory of distances in software engineering. Inf Softw Technol 70:204–219

    Google Scholar 

  • Boehm B, Rombach HD, Zelkowitz MV (2005) Foundations of empirical software engineering: the legacy of Victor R. Basili. Springer, Berlin

    Google Scholar 

  • Briand L, Bianculli D, Nejati S, Pastore F, Sabetzadeh M (2017) The case for context-driven software engineering research: generalizability is overrated. IEEE Softw 34(5):72–75

    Google Scholar 

  • Burkhard DL, Jenster PV (1989) Applications of computer-aided software engineering tools: survey of current and prospective users. ACM SIGMIS Database Database Adv Inf Syst 20(3):28–37

    Google Scholar 

  • Conradi R, Wang AI (2003) Empirical methods and studies in software engineering: experiences from ESERNET, vol 2765. Springer, Berlin

    MATH  Google Scholar 

  • Creswell JW, Creswell JD (2018) Research design: qualitative, quantitative, and mixed methods approaches. SAGE, Los Angeles

    Google Scholar 

  • Curtis B, Krasner H, Iscoe N (1988) A field study of the software design process for large systems. Commun Assoc Comput Mach 31(11):1268–1287

    Google Scholar 

  • de Mello RM, Da Silva PC, Travassos GH (2015) Investigating probabilistic sampling approaches for large-scale surveys in software engineering. J Softw Eng Res Dev 3(1):8

    Google Scholar 

  • Endres A, Rombach HD (2003) A handbook of software and systems engineering: empirical observations, laws, and theories. Pearson Education, Old Tappan

    Google Scholar 

  • Espinosa A, Kraut R, Slaughter S, Lerch J, Herbsleb J, Mockus A (2002) Shared mental models, familiarity, and coordination: a multi-method study of distributed software teams. In: Proceedings of ICIS 2002, p 39

    Google Scholar 

  • Farzat F, Barros MO, Travassos GH (2019) Evolving JavaScript code to reduce load time. IEEE Trans Softw Eng

    Google Scholar 

  • Fink A (2003) The survey handbook. SAGE, Los Angeles

    Google Scholar 

  • Glass RL (1994) The software-research crisis. IEEE Softw 11(6):42–47

    Google Scholar 

  • Grant EE, Sackman H (1967) An exploratory investigation of programmer performance under on-line and off-line conditions. IEEE Trans Hum Factors Electron 1:33–48

    Google Scholar 

  • Guéhéneuc YG, Khomh F (2019) Empirical software engineering. In: Cha S, Taylor RN, Kang KC (eds) Handbook of software engineering. Springer, Berlin, pp 285–320

    Google Scholar 

  • Harman M, McMinn P, De Souza JT, Yoo S (2010) Search based software engineering: techniques, taxonomy, tutorial. In: Empirical software engineering and verification. Springer, Berlin, pp 1–59

    Google Scholar 

  • Harrison W, Basili VR (1996) Editorial. Empir Softw Eng 1:5–10

    Google Scholar 

  • Hey AJ, Hey T, Pápay G (2014) The computing universe: a journey through a revolution. Cambridge University Press, Cambridge

    MATH  Google Scholar 

  • Höst M, Regnell B, Wohlin C (2000) Using students as subjects—a comparative study of students and professionals in lead-time impact assessment. Empir Softw Eng 5(3):201–214

    MATH  Google Scholar 

  • IEEE (1990) 610.12-19919—IEEE standard glossary of software engineering terminology. IEEE, New York

    Google Scholar 

  • IEEE (2010) ISO/IEC/IEEE 24765:2010 systems and software engineering—vocabulary. IEEE, Geneva

    Google Scholar 

  • Ivarsson M, Gorschek T (2011) A method for evaluating rigor and industrial relevance of technology evaluations. Empir Softw Eng 16(3):365–395

    Google Scholar 

  • Johnson P, Ekstedt M (2016) The Tarpit–a general theory of software engineering. Inf Softw Technol 70:181–203

    Google Scholar 

  • Juristo N, Moreno AM (2001) Basics of software engineering experimentation. Springer, Berlin

    MATH  Google Scholar 

  • Juristo N, Moreno AM (2003) Lecture notes on empirical software engineering, vol 12. World Scientific, New Jersey

    MATH  Google Scholar 

  • Kitchenham B (2004) Procedures for performing systematic reviews. Technical report

    Google Scholar 

  • Kitchenham B, Brereton P (2013) A systematic review of systematic review process research in software engineering. Inf Softw Technol 55(12):2049–2075

    Google Scholar 

  • Kitchenham B, Charters S (2007) Guidelines for performing systematic literature reviews in software engineering. Technical report

    Google Scholar 

  • Kitchenham BA, DybÃ¥ T, Jorgensen M (2004) Evidence-based software engineering. In: Proceedings of the 26th international conference on software engineering. IEEE Computer Society, Silver Spring, pp 273–281

    Google Scholar 

  • Kitchenham BA, Budgen D, Brereton P (2015) Evidence-based software engineering and systematic reviews, vol 4. CRC Press, Boca Raton

    Google Scholar 

  • Knuth DE (1971) An empirical study of Fortran programs. Softw Pract Exp 1(2):105–133

    MATH  Google Scholar 

  • Lethbridge TC, Sim SE, Singer J (2005) Studying software engineers: data collection techniques for software field studies. Empir Softw Eng 10(3):311–341

    Google Scholar 

  • Lucas J, Henry C, Kaplan RB (1976) A structured programming experiment. Comput J 19(2):136–138

    Google Scholar 

  • Lyu MR, et al (1996) Handbook of software reliability engineering, vol 222. IEEE Computer Society Press, Los Alamitos

    Google Scholar 

  • Malhotra R (2016) Empirical research in software engineering: concepts, analysis, and applications. Chapman and Hall/CRC, London

    Google Scholar 

  • McGarry F, Pajerski R, Page G, Waligora S, Basili V, Zelkowitz M (1994) Software process improvement in the NASA software engineering laboratory. Technical report, CMU/SEI-94-TR-022. Software Engineering Institute/Carnegie Mellon University, Pittsburgh. http://resources.sei.cmu.edu/library/asset-view.cfm?AssetID=12241

  • Menzies T, Kocaguneli E, Turhan B, Minku L, Peters F (2014) Sharing data and models in software engineering. Morgan Kaufmann, Amsterdam

    Google Scholar 

  • Menzies T, Williams L, Zimmermann T (2016) Perspectives on data science for software engineering. Morgan Kaufmann, Amsterdam

    Google Scholar 

  • Molléri JS, Petersen K, Mendes E (2019) Cerse-catalog for empirical research in software engineering: a systematic mapping study. Inf Softw Technol 105:117–149

    Google Scholar 

  • Münch J, Schmid K (2013) Perspectives on the future of software engineering: essays in honor of Dieter Rombach. Springer, Berlin

    Google Scholar 

  • Myers GJ (1978) A controlled experiment in program testing and code walkthroughs/inspections. Commun Assoc Comput Mach 21(9):760–768

    Google Scholar 

  • Nagappan N, Ball T (2005) Use of relative code churn measures to predict system defect density. In: Proceedings of the 27th international conference on software engineering. ACM, New York, pp 284–292

    Google Scholar 

  • Ogborn J, Miller R (1994) Computational issues in modelling. The Falmer Press, Basingstoke

    Google Scholar 

  • Ostrand TJ, Weyuker EJ, Bell RM (2004) Where the bugs are. In: ACM SIGSOFT software engineering notes, vol 29. ACM, New York, pp 86–96

    Google Scholar 

  • Petersen K, Feldt R, Mujtaba S, Mattsson M (2008) Systematic mapping studies in software engineering. In: Ease, vol 8, pp 68–77

    Google Scholar 

  • Petersen K, Vakkalanka S, Kuzniarz L (2015) Guidelines for conducting systematic mapping studies in software engineering: an update. Inf Softw Technol 64:1–18

    Google Scholar 

  • Pfleeger SL (1995) Experimental design and analysis in software engineering. Ann Softw Eng 1(1):219–253

    Google Scholar 

  • Pfleeger SL, Kitchenham BA (2001) Principles of survey research: part 1: turning lemons into lemonade. ACM SIGSOFT Softw Eng Notes 26(6):16–18

    Google Scholar 

  • Porter AA, Selby RW (1990) Empirically guided software development using metric-based classification trees. IEEE Softw 7(2):46–54

    Google Scholar 

  • Rombach HD, Basili VR, Selby RW (1993) Experimental software engineering issues: critical assessment and future directions. In: Proceedings of international workshop, Dagstuhl Castle, September 14–18, 1992, vol 706. Springer, Berlin

    Google Scholar 

  • Runeson P, Höst M (2009) Guidelines for conducting and reporting case study research in software engineering. Empir Softw Eng 14(2):131

    Google Scholar 

  • Runeson P, Höst M, Rainer A, Regnell B (2012) Case study research in software engineering. In: Guidelines and examples. Wiley, London

    Google Scholar 

  • Seaman CB (1999) Qualitative methods in empirical studies of software engineering. IEEE Trans Softw Eng 25(4):557–572

    Google Scholar 

  • Shadish WR, Cook TD, Campbell DT (2002) Experimental and quasi-experimental designs for generalized causal inference. Mifflin and Company, Boston, MA

    Google Scholar 

  • Sharp H, Dittrich Y, De Souza CR (2016) The role of ethnographic studies in empirical software engineering. IEEE Trans Softw Eng 42(8):786–804

    Google Scholar 

  • Shneiderman B, Mayer R, McKay D, Heller P (1977) Experimental investigations of the utility of detailed flowcharts in programming. Commun Assoc Comput Mach 20(6):373–381

    Google Scholar 

  • Shull F, Carver J, Travassos GH (2001) An empirical methodology for introducing software processes. In: ACM SIGSOFT software engineering notes, vol 26. ACM, New York, pp 288–296

    Google Scholar 

  • Shull F, Mendoncça MG, Basili V, Carver J, Maldonado JC, Fabbri S, Travassos GH, Ferreira MC (2004) Knowledge-sharing issues in experimental software engineering. Empir Softw Eng 9(1–2):111–137

    Google Scholar 

  • Shull F, Singer J, Sjøberg DI (2007) Guide to advanced empirical software engineering. Springer, Berlin

    Google Scholar 

  • Shull FJ, Carver JC, Vegas S, Juristo N (2008) The role of replications in empirical software engineering. Empir Softw Eng 13(2):211–218

    Google Scholar 

  • Sjøberg DI, DybÃ¥ T, Jorgensen M (2007) The future of empirical methods in software engineering research. In: 2007 Future of software engineering. IEEE Computer Society, Silver Spring, pp 358–378

    Google Scholar 

  • Srinivasan K, Fisher D (1995) Machine learning approaches to estimating software development effort. IEEE Trans Softw Eng 21(2):126–137

    Google Scholar 

  • Staron M (2019) Action research in software engineering: theory and applications. Springer, Berlin

    Google Scholar 

  • Stol KJ, Fitzgerald B (2015) Theory-oriented software engineering. Sci Comput Program 101:79–98

    Google Scholar 

  • Stol KJ, Fitzgerald B (2018) The ABC of software engineering research. ACM Trans Softw Eng Methodol 27(3):11

    Google Scholar 

  • Stol KJ, Ralph P, Fitzgerald B (2016) Grounded theory in software engineering research: a critical review and guidelines. In: 2016 IEEE/ACM 38th international conference on software engineering (ICSE). IEEE, Piscataway, pp 120–131

    Google Scholar 

  • Swanson EB, Beath CM (1988) The use of case study data in software management research. J Syst Softw 8(1):63–71

    Google Scholar 

  • Theisen C, Dunaiski M, Williams L, Visser W (2017) Writing good software engineering research papers: revisited. In: Proceedings of the 39th international conference on software engineering companion. IEEE, Piscataway, pp 402–402

    Google Scholar 

  • Thomke SH (2003) Experimentation matters: unlocking the potential of new technologies for innovation. Harvard Business Press, Boston

    Google Scholar 

  • Tian J (1995) Integrating time domain and input domain analyses of software reliability using tree-based models. IEEE Trans Softw Eng 21(12):945–958

    Google Scholar 

  • Travassos GH, Barros MO (2003) Contributions of in virtuo and in silico experiments for the future of empirical studies in software engineering. In: 2nd workshop on empirical software engineering the future of empirical studies in software engineering, pp 117–130

    Google Scholar 

  • Travassos GH, dos Santos PSM, Mian PG, Neto ACD, Biolchini J (2008) An environment to support large scale experimentation in software engineering. In: 13th IEEE international conference on engineering of complex computer systems (ICECCS 2008). IEEE, Piscataway, pp 193–202

    Google Scholar 

  • Wagner S, Fernández DM, Felderer M, Vetrò A, Kalinowski M, Wieringa R, Pfahl D, Conte T, Christiansson MT, Greer D, et al (2019) Status quo in requirements engineering: a theory and a global family of surveys. ACM Trans Softw Eng Methodol 28(2):9

    Google Scholar 

  • Widman L (1989) Expert system reasoning about dynamic systems by semi-quantitative simulation. Comput Methods Prog Biomed Artif Intell Med 6(3):229–247

    Google Scholar 

  • Wieringa R (2014a) Empirical research methods for technology validation: scaling up to practice. J Syst Softw 95:19–31

    Google Scholar 

  • Wieringa RJ (2014b) Design science methodology for information systems and software engineering. Springer, Berlin

    Google Scholar 

  • Wohlin C, Aurum A (2015) Towards a decision-making structure for selecting a research design in empirical software engineering. Empir Softw Eng 20(6):1427–1455

    Google Scholar 

  • Wohlin C, Runeson P, Höst M, Ohlsson M, Regnell B, Wesslén A (2000) Experimentation in software engineering: an introduction. Kluwer Academic Publishers, Norwell, MA

    MATH  Google Scholar 

  • Wohlin C, Runeson P, Höst M, Ohlsson MC, Regnell B, Wesslén A (2012) Experimentation in software engineering. Springer, Berlin

    MATH  Google Scholar 

  • Zelkowitz MV, Wallace DR (1998) Experimental models for validating technology. Computer 31(5):23–31

    Google Scholar 

  • Zendler A (2001) A preliminary software engineering theory as investigated by published experiments. Empir Softw Eng 6(2):161–180

    MATH  Google Scholar 

  • Zimmermann T, Zeller A, Weissgerber P, Diehl S (2005) Mining version histories to guide software changes. IEEE Trans Softw Eng 31(6):429–445

    Google Scholar 

Download references

Acknowledgements

We thank all the authors and reviewers of this book on contemporary empirical methods in software engineering for their valuable contribution.

Author information

Authors and Affiliations

  1. Department of Computer Science, University of Innsbruck, Innsbruck, Austria

    Michael Felderer

  2. Department of Software Engineering, Blekinge Institute of Technology, Karlskrona, Sweden

    Michael Felderer

  3. Department of Systems Engineering and Computer Science, COPPE, Federal University of Rio de Janeiro, Rio de Janeiro, Brazil

    Guilherme Horta Travassos

Authors
  1. Michael Felderer
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Guilherme Horta Travassos
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Michael Felderer .

Editor information

Editors and Affiliations

  1. Institute of Computer Science, University of Innsbruck, Innsbruck, Austria

    Michael Felderer

  2. Systems Engineering and Computer Science, Federal University of Rio de Janeiro, Rio de Janeiro, Rio de Janeiro, Brazil

    Guilherme Horta Travassos

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Felderer, M., Travassos, G.H. (2020). The Evolution of Empirical Methods in Software Engineering. In: Felderer, M., Travassos, G. (eds) Contemporary Empirical Methods in Software Engineering. Springer, Cham. https://doi.org/10.1007/978-3-030-32489-6_1

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-32489-6_1

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-32488-9

  • Online ISBN: 978-3-030-32489-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation