Reporting Empirical Research in Open Source Software: The State of Practice

  • Klaas-Jan Stol
  • Muhammad Ali Babar
Part of the IFIP Advances in Information and Communication Technology book series (IFIPAICT, volume 299)


Background: The number of reported empirical studies of Open Source Software (OSS) has continuously been increasing. However, there has been no effort to systematically review the state of the practice of reporting empirical studies of OSS with respect to the recommended standards of performing and reporting empirical studies in software engineering. It is important to understand, how to report empirical studies of OSS in order to make them useful for practitioners and researchers.

Research aim: The aim of our research is to gain insights in the state of the practice of reporting empirical studies of OSS in order to identify the gaps to be filled for improving the quality of evidence being provided for OSS.

Method: To that end, we decided to systematically review the empirical studies of OSS. A total of 63 papers reporting empirical studies were selected from the four editions of the Proceedings of the International Conference on Open Source Systems. The data were extracted and synthesised from the selected papers for analysis.

Results and conclusions: We have found that the quality of the reported OSS-related empirical studies needs to be significantly improved. Based on the results of our systematic review and general principles of reporting good empirical research, we present a set of guidelines for reporting OSS-related empirical studies. The suggested guidelines are expected to help the research community to improve the quality of reported studies.


Empirical research open source reporting guidelines 


  1. 1.
    Open Source Initiative. History of the OSI, (last accessed on October 16, 2008)
  2. 2.
    Dyba, T., Dingsoyr, T.: Empirical Studies of Agile Software Development: A Systematic Review. Information and Software Technology 50(9-10), 833–859 (2008)CrossRefGoogle Scholar
  3. 3.
    Šmite, D., Wohlin, C., Feldt, R., Gorschek, T.: Reporting Empirical Research in Global Software Engineering: a Classification Scheme. In: International Conference on Global Software Engineering (2008)Google Scholar
  4. 4.
    Kitchenham, B.A., et al.: Preliminary guidelines for empirical research in software engineering. IEEE Transactions on Software Engineering 28(8), 721–734 (2002)CrossRefGoogle Scholar
  5. 5.
    Jedlitschka, A., Pfahl, D.: Reporting Guidelines for Controlled Experiments in Software Engineering. In: Proceedings of the International Symposium on Empirical Software Engineering (2005)Google Scholar
  6. 6.
    Lethbridge, T.C., Sim, S.E., Singer, J.: Studying Software Engineers: Data Collection Techniques for Software Field Studies. Empirical Software Engineering 10, 311–341 (2005)Google Scholar
  7. 7.
    Kitchenham, B., Charters, S.: Guidelines for Performing Systematic Literature Reviews in Software Engineering, Tech. Report EBSE-2007-1, Keele University, UK (2007)Google Scholar
  8. 8.
    Landis, J.R., Koch, G.G.: The measurement of observer agreement for categorical data. Biometrics 33(1), 159–174 (1977)MathSciNetCrossRefzbMATHGoogle Scholar
  9. 9.
    Staples, M., Niazi, M.: Systematic review of organizational motivation for adopting CMM-based SPI. Information and Software Technology 50(7-8), 605–620 (2008)CrossRefGoogle Scholar
  10. 10.
    Fink, A.: Conducting Research Literature Reviews: From Internet to Paper. Sage Publication, Inc., Thousand Oaks (2005)Google Scholar
  11. 11.
    Wohlin, C., et al.: Experimentation in Software Engineering: An Introduction. Kluwer Academic Publications, Dordrecht (2000)CrossRefzbMATHGoogle Scholar
  12. 12.
    Kitchenham, B., et al.: Evaluating guidelines for reporting empirical software engineering studies. Empirical Software Engineering 13(1), 219–221 (2008)Google Scholar
  13. 13.
    Sjoberg, D., Dyba, T., Jorgensen, M.: The future of empirical methods in software engineering research. In: Proceedings of the International Conference on Software Engineering, Future of Software Engineering Track (2007)Google Scholar
  14. 14.
    Host, M., Runeson, P.: Checklists for Software Engineering Case Study Research. In: Proceedings of the First International Symposium on Empirical Software Engineering and Measurement (2007)Google Scholar
  15. 15.
    Kitchenham, B., Pfleeger, S.L.: Principles of Survey Research, Parts 1 to 6. Software Engineering Notes (2001–2002)Google Scholar
  16. 16.
    Miller, J., et al.: Statistical power and its subcomponents - missing and misunderstood concepts in empirical software engineering research. Information and Software Technology 39, 285–295 (1997)CrossRefGoogle Scholar

Copyright information

© IFIP International Federation for Information Processing 2009

Authors and Affiliations

  • Klaas-Jan Stol
    • 1
  • Muhammad Ali Babar
    • 1
  1. 1.Lero—The Irish Software Engineering Research CentreUniversity of LimerickLimerickIreland

Personalised recommendations