Reporting Empirical Research in Open Source Software: The State of Practice
Background: The number of reported empirical studies of Open Source Software (OSS) has continuously been increasing. However, there has been no effort to systematically review the state of the practice of reporting empirical studies of OSS with respect to the recommended standards of performing and reporting empirical studies in software engineering. It is important to understand, how to report empirical studies of OSS in order to make them useful for practitioners and researchers.
Research aim: The aim of our research is to gain insights in the state of the practice of reporting empirical studies of OSS in order to identify the gaps to be filled for improving the quality of evidence being provided for OSS.
Method: To that end, we decided to systematically review the empirical studies of OSS. A total of 63 papers reporting empirical studies were selected from the four editions of the Proceedings of the International Conference on Open Source Systems. The data were extracted and synthesised from the selected papers for analysis.
Results and conclusions: We have found that the quality of the reported OSS-related empirical studies needs to be significantly improved. Based on the results of our systematic review and general principles of reporting good empirical research, we present a set of guidelines for reporting OSS-related empirical studies. The suggested guidelines are expected to help the research community to improve the quality of reported studies.
KeywordsEmpirical research open source reporting guidelines
- 1.Open Source Initiative. History of the OSI, http://www.opensource.org/history (last accessed on October 16, 2008)
- 3.Šmite, D., Wohlin, C., Feldt, R., Gorschek, T.: Reporting Empirical Research in Global Software Engineering: a Classification Scheme. In: International Conference on Global Software Engineering (2008)Google Scholar
- 5.Jedlitschka, A., Pfahl, D.: Reporting Guidelines for Controlled Experiments in Software Engineering. In: Proceedings of the International Symposium on Empirical Software Engineering (2005)Google Scholar
- 6.Lethbridge, T.C., Sim, S.E., Singer, J.: Studying Software Engineers: Data Collection Techniques for Software Field Studies. Empirical Software Engineering 10, 311–341 (2005)Google Scholar
- 7.Kitchenham, B., Charters, S.: Guidelines for Performing Systematic Literature Reviews in Software Engineering, Tech. Report EBSE-2007-1, Keele University, UK (2007)Google Scholar
- 10.Fink, A.: Conducting Research Literature Reviews: From Internet to Paper. Sage Publication, Inc., Thousand Oaks (2005)Google Scholar
- 12.Kitchenham, B., et al.: Evaluating guidelines for reporting empirical software engineering studies. Empirical Software Engineering 13(1), 219–221 (2008)Google Scholar
- 13.Sjoberg, D., Dyba, T., Jorgensen, M.: The future of empirical methods in software engineering research. In: Proceedings of the International Conference on Software Engineering, Future of Software Engineering Track (2007)Google Scholar
- 14.Host, M., Runeson, P.: Checklists for Software Engineering Case Study Research. In: Proceedings of the First International Symposium on Empirical Software Engineering and Measurement (2007)Google Scholar
- 15.Kitchenham, B., Pfleeger, S.L.: Principles of Survey Research, Parts 1 to 6. Software Engineering Notes (2001–2002)Google Scholar