Background: One major problem for integrating study results into a common body of knowledge is the heterogeneity of reporting styles: (1) It is difficult to locate relevant information and (2) important information is often missing.
Objective: A guideline for reporting results from controlled experiments is expected to support a systematic, standardized presentation of empirical research, thus improving reporting in order to support readers in (1) finding the information they are looking for, (2) understanding how an experiment is conducted, and (3) assessing the validity of its results.
Method: The guideline for reporting is based on (1) a survey of the most prominent published proposals for reporting guidelines in software engineering and (2) an iterative development incorporating feedback from members of the research community.
Result: This chapter presents the unification of a set of guidelines for reporting experiments in software engineering.
Limitation: The guideline has not been evaluated broadly yet.
Conclusion: The resulting guideline provides detailed guidance on the expected content of the sections and subsections for reporting a specific type of empirical study, i.e., experiments (controlled experiments and quasi-experiments).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Altman, D.G., Schulz, K.F., Moher, D., Egger, M., Davidoff, F., Elbourne, D., Gøtzsche, P.C., Lang, T. for the CONSORT Group (2001). The Revised CONSORT Statement for Reporting Randomized Trials, Explanation and Elaboration. Annals of Internal Medicine, Vol. 134, No. 8, pp. 663–694.
American Psychological Association (2001). Publication Manual of the American Psychological Association, 5th edn, American Psychological Association, Washington, DC.
Basili, V.R., Caldiera, G., Rombach, H.D. (2001). Goal Question Metric Paradigm, in Marciniak, J.J. (Ed.), Encyclopedia of Software Engineering, Vol. 1, Wiley, New York, pp. 528–532.
Basili, V.R., Zelkowitz, M., Sjøberg, D.I.K., Johnson, P., Cowling, T. (2007). Protocols in the use of Empirical Software Engineering Artifacts. Journal of Empirical Software Engineering, 12(1), pp. 107–119.
Bayley, L., Eldredge, J. (2003). The Structured Abstract, An Essential Tool for Researchers, In Hypothesis. The Journal of the Research Section of the Medical Library Association, Vol. 17, No. 1, 4 pp.
Ciolkowski, M., Differding, C., Laitenberger, O., Münch, J. (1997). Empirical Investigation of Perspective-based Reading, A Replicated Experiment, Fraunhofer Institute for Experimental Software Engineering, Germany, ISERN-97–13.
Dybå, T., Kampenes, B.V., Sjøberg, D.I.K. (2006). A Systematic Review of Statistical Power in Software Engineering Experiments, A Survey of Controlled Experiments in Software Engineering. Information and Software Technology, Vol. 48, pp. 745–755.
Glass, R.L. (2004). Matching Methodology to Problem Domain. Communications of the ACM, Vol. 47, No. 5, pp. 19–21.
Harris, P. (2002). Designing and Reporting Experiments in Psychology, 2nd edn, Open University Press, Buckingham.
Hartley, J. (2003). Improving the Clarity of Journal Abstracts in Psychology, The Case for Structure. Science Communication, Vol. 24, No. 3, pp. 366–379.
Hartley, J. (2004). Current Findings from Research on Structured Abstracts. Journal of the Medical Library Association, Vol. 92, No. 3, pp. 368–371.
Hayward, R.S.A., Wilson, M.C., Tunis, S.R., Bass, E.B., Rubin, H.R., Haynes, R.B. (1993). More Informative Abstracts of Articles Describing Clinical Practice Guidelines. Annals of Internal Medicine Vol. 118, No. 9, pp. 731–737.
Jedlitschka, A. (2005). Minutes from Third International Workshop on Empirical Software Engineering “Guidelines for Empirical Work in Software Engineering”. IESE-Report 052.05/E, Oulu.
Jedlitschka, A., Ciolkowski, M. (2004). Towards Evidence in Software Engineering, In Proceedings of ACM/IEEE International Symposium on Software Engineering 2004 (ISESE2004). Redondo Beach, California, pp. 261–270.
Jedlitschka, A., Pfahl, D. (2005a). Reporting Guidelines for Controlled Experiments in Software Engineering. IESE-Report IESE-035.5/E.
Jedlitschka, A., Pfahl, D. (2005b). Reporting Guidelines for Controlled Experiments in Software Engineering, In Proceedings of ACM/IEEE International Symposium on Software Engineering 2005 (ISESE2005). Noosa Heads, Australia, pp. 95–104.
Jedlitschka, A., Ciolkowski, M. (2006). Reporting Guidelines for Controlled Experiments in Software Engineering, Fraunhofer Institute for Experimental Software Engineering, Germany, ISERN-06–1.
Jedlitschka, A., Ciolkowski, M. Pfahl, D. (2007). Reporting Guidelines for Controlled Experiments in Software Engineering, Fraunhofer Institute for Experimental Software Engineering, Germany, ISERN-07–1.
Juristo, N., Moreno, A. (2001). Basics of Software Engineering Experimentation, Kluwer Academic Publishers, Boston, MA.
Kampenes, B.V., Dybå, T., Hannay, J., Sjøberg, D.I.K. (2007). A Systematic Review of Effect Size in Software Engineering Experiments. Information and Software Technology, Vol. 49, No. 11–12, pp. 1073–1086.
Kitchenham, B. (2004). Procedures for Performing Systematic Reviews, Keele University Joint Technical Report TR/SE-0401, ISSN, 1353–7776 and National ICT Australia Ltd. NICTA Technical Report 0400011T.1.
Kitchenham, B., Al-Khilidar, H., Ali Babar, M., Berry, M., Cox, C., Keung, J., Kurniawati, F., Staples, M., Zhang, H., Zhu, L. (2006). Evaluating Guidelines for Empirical Software Engineering Studies, In Proceedings of ACM/IEEE International Symposium on Software Engineering 2006 (ISESE2006).
Kitchenham, B., Dybå, T., Jørgensen, M. (2004). Evidence-Based Software Engineering, In Proceedings of 26th International Conference on Software Engineering (ICSE’04), pp. 273–281.
Kitchenham, B.A., Hughes, R.T., Linkman, S.G. (2001). Modeling Software Measurement, IEEE Transactions on Software Engineering, Vol. 27, No. 9, pp. 788–804.
Kitchenham, B.A., Pfleeger, S.L., Pickard, L.M., Jones, P.W., Hoaglin, D.C., El Emam, K., Rosenberg, J. (2002). Preliminary Guidelines for Empirical Research in Software Engineering, IEEE Transactions on Software Engineering, Vol. 28, No. 8, pp. 721–734.
Lott, C.M., Rombach, H.D. (1996). Repeatable Software Engineering Experiments for Comparing Defect–Detection Techniques, Empirical Software Engineering Journal, Vol. 3.1, pp. 241–277.
Moher, D., Schulz, K.F., Altman, D. for the CONSORT Group (2001). The CONSORT Statement, Revised Recommendations for Improving the Quality of Reports of Parallel-Group Randomized Trials, Journal of the American Medical Association (JAMA) Vol. 285, No. 15, pp. 1987–1991.
Pickard, L.M., Kitchenham, B.A., Jones, P.W. (1998). Combining Empirical Results in Software Engineering, Information and Software Technology, Vol. 40, No. 14, pp. 811–821.
Shaw, M. (2003). Writing Good Software Engineering Research Papers–Minitutorial, In Proceedings of the 25th International Conference on Software Engineering (ICSE’03). IEEE Computer Society, Portland, Oregon, pp. 726–736.
Shiffman, R.N., Shekelle, P., Overhage, J.M., Slutsky, J., Grimshaw, J., Deshpande, A.M. (2003). Standardized Reporting of Clinical Practice Guidelines, A Proposal from the Conference on Guideline Standardization, Annals of Internal Medicine, Vol. 139, No. 6, pp. 493–498.
Shull, F., Carver, J., Travassos, G.H., Maldonado, J.C., Conradi, R., Basili, V.R. (2003). Replicated Studies, Building a Body of Knowledge about Software Reading Techniques, In Juristo, N., Moreno, A. (Eds.), Lecture Notes on Empirical Software Engineering, World Scientific Publishing, River Edge, NJ, USA, pp. 39–84.
Singer, J. (1999). Using the APA Style Guidelines to Report Experimental Results, In Proceedings of Workshop on Empirical Studies in Software Maintenance, pp. 71–75. (dec.bmth.ac.uk/ESERG/WESS99/singer.ps).
Sjøberg, D.I.K., Hannay, J., Hansen, O., Kampenes, B.V., Karahasanovic, A., Liborg, N.-K., Rekdal, A. (2005). A Survey of Controlled Experiments in Software Engineering. Transactions on Software Engineering, Vol. 31, No. 9, pp. 733–753.
The Editors of Annals of Internal Medicine (2004). Addressing the Limitations of Structured Abstracts (Editorial). Annals of Internal Medicine, Vol. 140, No. 6, pp. 480–481.
Vegas, S., Juristo, N., Basili, V. (2003). A Process for Identifying Relevant Information for a Repository, A Case Study for Testing Techniques. In Aurum, A., Jeffery, R., Wohlin, C., Handzic, M. (Eds.). Managing Software Engineering Knowledge, Springer-Verlag, Berlin, pp. 199–230.
Wohlin, C., Petersson, H., Aurum, A. (2003). Combining Data from Reading Experiments in Software Inspections, In Juristo, N., Moreno, A. (Eds.), Lecture Notes on Empirical Software Engineering, World Scientific Publishing, River Edge, NJ, USA, pp. 85–132.
Wohlin, C., Runeson, P., Höst, M., Ohlsson, M.C., Regnell, B., Wesslén, A. (2000). Experimentation in Software Engineering–An Introduction, Kluwer Academic Publishers, Boston, MA.
Zelkowitz, M.V., Wallace, D.R., Binkley, D.W. (2003). Experimental Validation of New Software Technology. In Juristo, N., Moreno, A. (Eds.), Lecture Notes on Empirical Software Engineering, World Scientific Publishing, River Edge, NJ, USA, pp. 229–263.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2008 Springer-Verlag London Limited
About this chapter
Cite this chapter
Jedlitschka, A., Ciolkowski, M., Pfahl, D. (2008). Reporting Experiments in Software Engineering. In: Shull, F., Singer, J., Sjøberg, D.I.K. (eds) Guide to Advanced Empirical Software Engineering. Springer, London. https://doi.org/10.1007/978-1-84800-044-5_8
Download citation
DOI: https://doi.org/10.1007/978-1-84800-044-5_8
Publisher Name: Springer, London
Print ISBN: 978-1-84800-043-8
Online ISBN: 978-1-84800-044-5
eBook Packages: Computer ScienceComputer Science (R0)