Empirical Software Engineering

, Volume 13, Issue 4, pp 435–468 | Cite as

Presenting software engineering results using structured abstracts: a randomised experiment

  • David BudgenEmail author
  • Barbara A. Kitchenham
  • Stuart M. Charters
  • Mark Turner
  • Pearl Brereton
  • Stephen G. Linkman


When conducting a systematic literature review, researchers usually determine the relevance of primary studies on the basis of the title and abstract. However, experience indicates that the abstracts for many software engineering papers are of too poor a quality to be used for this purpose. A solution adopted in other domains is to employ structured abstracts to improve the quality of information provided. This study consists of a formal experiment to investigate whether structured abstracts are more complete and easier to understand than non-structured abstracts for papers that describe software engineering experiments. We constructed structured versions of the abstracts for a random selection of 25 papers describing software engineering experiments. The 64 participants were each presented with one abstract in its original unstructured form and one in a structured form, and for each one were asked to assess its clarity (measured on a scale of 1 to 10) and completeness (measured with a questionnaire that used 18 items). Based on a regression analysis that adjusted for participant, abstract, type of abstract seen first, knowledge of structured abstracts, software engineering role, and preference for conventional or structured abstracts, the use of structured abstracts increased the completeness score by 6.65 (SE 0.37, p < 0.001) and the clarity score by 2.98 (SE 0.23, p < 0.001). 57 participants reported their preferences regarding structured abstracts: 13 (23%) had no preference; 40 (70%) preferred structured abstracts; four preferred conventional abstracts. Many conventional software engineering abstracts omit important information. Our study is consistent with studies from other disciplines and confirms that structured abstracts can improve both information content and readability. Although care must be taken to develop appropriate structures for different types of article, we recommend that Software Engineering journals and conferences adopt structured abstracts.


Structured abstract Randomised controlled laboratory experiment 



This work was supported by an award from the U.K.’s Engineering and Physical Sciences Research Council (EPSRC). The authors would also like to thank Dag Sjøberg for providing a random selection of papers to use in this study; all those who helped by participating in the study as ‘judges’; John Bailey who organised the data collection; and Professor Jim Hartley of Keele University for his advice and guidance.


  1. Automated Readability Index (2006) In: Wikipedia, the free encyclopedia. Retrieved 29 September 2006 URL:
  2. Bayley L, Eldredge J (2003) The structured abstract: an essential tool for researchers. Hypothesis 17(1):11–13Google Scholar
  3. Booth A (2003) Bridging the research–practice gap? The role of evidence based librarianship. New Review of Information and Library Research, pp 3–23Google Scholar
  4. Brereton O, Kitchenham B, Budgen D, Turner M, Khalil M (2007) Lessons from applying the systematic literature review process within the software engineering domain. J Syst Softw 80(4):571–583CrossRefGoogle Scholar
  5. Budgen D, Kitchenham B, Charters S, Turner M, Brereton P, Linkman S (2007a) Preliminary results of a study of the completeness and clarity of structured abstracts. In: EASE 2007: evaluation and assessment in software engineering. BCS-eWiC, pp 64–72Google Scholar
  6. Budgen D, Kitchenham B, Charters S, Turner M, Brereton P, Linkman S (2007b) Protocol for an experimental study of the use of structured abstracts. Technical report, EBSE Project. Version 1.3.2.Google Scholar
  7. Editorial (2004) Addressing the limitations of structured abstracts. Ann Intern Med 140:480–481Google Scholar
  8. Flesch R (1948) A new readability yardstick. J Appl Psychol 32:221–233CrossRefGoogle Scholar
  9. Flesch-Kincaid Readability Test (2006) In: Wikipedia, the free encyclopedia. URL: Retrieved 29 September 2006
  10. Glasser B, Strauss A (1967) The discovery of grounded theory. Aldine, ChicagoGoogle Scholar
  11. Hartley J (2000) Typographic settings for structured abstracts. J Tech Writ 30(4):355–365CrossRefGoogle Scholar
  12. Hartley J (2003) Improving the clarity of journal abstracts in psychology: the case for structure. Sci Commun 24:366–379CrossRefGoogle Scholar
  13. Hartley J (2004) Current findings from research on structured abstracts. J Med Libr Assoc 92:368–371Google Scholar
  14. Hartley J, Benjamin M (1998) An evaluation of structured abstracts in journals published by the British Psychological Society. Br J Educ Psychol 68:443–456Google Scholar
  15. Hartley J, Sydes M (1996) Which layout do you prefer? An analysis of readers’ preferences for different typographical layouts of structured abstracts. J Inf Sci 22(1):27–37CrossRefGoogle Scholar
  16. Hartley J, Sydes M (1997) Are structured abstracts easier to read than traditional ones? J Res Read 20:122–136CrossRefGoogle Scholar
  17. Jedlitschka A, Pfahl D (2005) Reporting guidelines for controlled experiments in software engineering. In: Proc. ACM/IEEE international symposium on empirical software engineering (ISESE) 2005. IEEE Computer Society Press, pp 95–195Google Scholar
  18. Jedlitschka A, Ciolkowski M, Pfahl D (2008) Reporting experiments in software engineering. In: Shull F, Singer J, Sjøberg D (eds) Guide to advanced empirical software engineering, Chapter 8. Springer, LondonGoogle Scholar
  19. Kitchenham B (2004) Procedures for undertaking systematic reviews. Technical Report TR/SE-0401. Department of Computer Science, Keele University and National ICT, Australia Ltd. Joint Technical ReportGoogle Scholar
  20. Kitchenham B, Al-Khilidar H, Babar MA, Berry M, Cox K, Keung J, Kurniawati F, Staples M, Zhang H, Zhu L (2006) Evaluating guidelines for empirical software engineering studies. In: Proceedings ACM/IEEE international symposium on empirical software engineering (ISESE 2006). IEEE Computer Society PressGoogle Scholar
  21. Kitchenham B, Brereton P, Owen S, Butcher J, Jefferies C (2008) Length and readability of structured software engineering abstracts. IET Softw 2:37–45CrossRefGoogle Scholar
  22. Kitchenham B, Budgen D, Brereton P, Turner M (2007) 2nd international workshop on realising evidence-based software engineering (REBSE-2): overview and introduction. In: Proceedings of REBSE-2 workshop, ICSE 2007. IEEE Computer Society Press, pp 1–5Google Scholar
  23. Kitchenham B, Dybå T, Jørgensen M (2004) Evidence-based software engineering. In: Proceedings of ICSE 2004. IEEE Computer Society Press, pp 273–281Google Scholar
  24. Kitchenham B, Pfleeger SL, Pickard L, Jones P, Hoaglin D, Emam KE, Rosenberg J (2002) Preliminary guidelines for empirical research in software engineering. IEEE Trans Softw Eng 28:721–734CrossRefGoogle Scholar
  25. Milliken G, Johnson D (1992) Analysis of messy data—volume 1: designed experiments. Chapman and HallGoogle Scholar
  26. Petticrew M, Roberts H (2006) Systematic reviews in the social sciences: a practical guide. BlackwellGoogle Scholar
  27. Senn S (2002) Cross-over trials in clinical research, 2nd edn. WileyGoogle Scholar
  28. Sharma S, Harrison JE (2006) Structured abstracts: do they improve the quality of information in abstracts? Am J Orthod Dentofac Orthop 130(4):523–530CrossRefGoogle Scholar
  29. Sjøberg D, Hannay J, Hansen O, Kampenes V, Karahasanovic A, Liborg N, Rekdal A (2005) A survey of controlled experiments in software engineering. IEEE Trans Softw Eng 31(9):733–753CrossRefGoogle Scholar
  30. Webster J, Watson R (2002) Analysing the past to prepare for the future: writing a literature review. MIS Quarterly 26, xiii–xxiiiGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2008

Authors and Affiliations

  • David Budgen
    • 1
    Email author
  • Barbara A. Kitchenham
    • 2
  • Stuart M. Charters
    • 3
  • Mark Turner
    • 2
  • Pearl Brereton
    • 2
  • Stephen G. Linkman
    • 2
  1. 1.Department of Computer ScienceDurham UniversityDurhamUK
  2. 2.School of Computing and MathematicsKeele UniversityStaffordshireUK
  3. 3.School of Applied ComputingLincoln UniversityLincolnNew Zealand

Personalised recommendations