Skip to main content
Log in

Do Systematic Literature Reviews Outperform Informal Literature Reviews in the Software Engineering Domain? An Initial Case Study

  • Research Article - Computer Engineering and Computer Science
  • Published:
Arabian Journal for Science and Engineering Aims and scope Submit manuscript

Abstract

The systematic literature reviews (SLRs) are a formally planned approach in finding, evaluating and summarising all available evidence on a specific research question. The objective of this paper is to compare formal SLRs and informal literature reviews in software engineering. For this purpose, a SLR has been conducted to compare the results with a previously conducted informal literature review of software process improvement success factors. Previous research using an informal literature review approach found 47 articles via the snow balling technique. For formal literature reviews, the SLR was conducted by applying customised search strings covering the time to 30 June 2004 (this is the deadline of the informal literature review). In total, 38 relevant articles were identified via the formal literature review. The results show that the data extraction process in the formal literature review enabled more success factors to be extracted (i.e. 34) than the informal literature review (i.e. 18). In the formal literature review, the publication inclusion and exclusion criteria and selecting primary studies helped in identifying the right list of publications. A real challenge in the formal literature review was to define a search string. In general it was observed that the SLR methodology is better than the informal literature review with respect to the planning for literature review, the design of search string, sources to be searched, publication inclusion and exclusion criteria, publication quality assessment and the data extraction process. However, the informal literature review has identified more articles than the formal literature review. Snow balling technique should be used with the formal literature review in order to identify all the relevant articles.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Kitchenham, B.; Charters, C.: Guidelines for performing systematic literature reviews in software engineering. Keele University and Durham University Joint Report—EBSE 2007-001 (2007)

  2. Niazi M., Wilson D., Zowghi D.: Critical success factors for software process improvement: an empirical study. Softw. Process Improv. Pract. J. 11(2), 193–211 (2006)

    Article  Google Scholar 

  3. Dyba T., Kitchenham B., Jorgensen M.: Evidence based software engineering for practitioners. IEEE Softw. 22(1), 158–165 (2005)

    Article  Google Scholar 

  4. Sackett D.L., Straus S.E., Richardson W.S., Rosenberg W., Haynes R.B.: Evidence-Based Medicine: How to Practice and Teach EBM, 2nd edn. Churchill Livingstone, Edinburgh (2000)

    Google Scholar 

  5. Khan S.U., Niazi M., Ahmad R.: Factors influencing clients in the selection of offshore software outsourcing vendors: an exploratory study using a systematic literature review. J. Syst. Softw. 84(4), 686–699 (2011)

    Article  Google Scholar 

  6. Smite D., Wohlin C., Gorscheck T., Feldt R.: Empirical evidence in global software engineering: a systematic review. Empir. Softw. Eng. 15(1), 91–118 (2010)

    Article  Google Scholar 

  7. Niazi M., Ikram N., Bano M., Imtiaz S., Khan S.U.: Establishing trust in offshore software outsourcing relationships: an exploratory study using a systematic literature review. IET Softw. 7(5), 283–293 (2013)

    Article  Google Scholar 

  8. Dyba T., Dingsoyr T.: Empirical studies of agile software development: a systematic review. Inf. Softw. Technol. 50(9), 833–859 (2008)

    Article  Google Scholar 

  9. Staples M., Niazi M.: Systematic review of organizational motivations for adopting CMM-based SPI. Inf. Softw. Technol. J. 50(7–8), 605–620 (2008)

    Article  Google Scholar 

  10. Verner J., Brereton O.P., Kitchenham B., Turner M., Niazi M.: Risks and risk mitigation in global software development: a tertiary study. Inf. Softw. Technol. J. 56(1), 54–78 (2014)

    Article  Google Scholar 

  11. Kitchenham B., Mendes E., Travassos G.H.: Cross vs within-company cost estimation studies: a systematic review. IEEE Trans. Softw. Eng. 33(5), 316–329 (2007)

    Article  Google Scholar 

  12. Staples M., Niazi M.: Experiences using systematic review guidelines. J. Syst. Softw. 80(9), 1425–1437 (2007)

    Article  Google Scholar 

  13. Imtiaz, S.; Bano, M.; Ikram, N.; Niazi, M.: Experiences of conducting systematic literature reviews in software engineering: a tertiary study. In: 17th International Conference on Evaluation and Assessment in Software Engineering, EASE 2013 (2013)

  14. Fabio, D.S.; André, S.; Sérgio, S.; César, F.; Cleviton, M.: A critical appraisal of systematic reviews in software engineering from the perspective of the research questions asked in the reviews. In: Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement (2010)

  15. Ali Barbar, M.; Zhang, H.: Systematic literature reviews in software engineering: preliminary results from interviews with researchers. In: Third International Symposium on Empirical Software Engineering and Measurement (2009)

  16. Kitchenham B., Brereton P., Budgen D., Turner M., Bailey J., Linkman S.: Systematic literature reviews in software engineering: a systematic literature review. Inf. Softw. Technol. 51(1), 7–15 (2009)

    Article  Google Scholar 

  17. Kitchenham B., Pretorius R., Brereton P., Budgen D., Turner M., Niazi M., Linkman S.: Systematic literature reviews in software engineering—a tertiary study. Inf. Softw. Technol. J. 52(8), 792–805 (2010)

    Article  Google Scholar 

  18. Verner, J.; Brereton, O.P.; Kitchenham, B.; Turner, M.; Niazi, M.: Systematic literature reviews in global software development: a tertiary study. In: International Conference on Evaluation & Assessment in Software Engineering (EASE 2012) (2012)

  19. Kitchenham B., Pretorius R., Brereton P., Budgen D., Turner M., Niazi M., Linkman S.: Refining the systematic literature review process—two participant-observer case studies. Empir. Softw. Eng. J. 15(6), 618–653 (2010)

    Article  Google Scholar 

  20. Niazi M.: An exploratory study of software process improvement implementation risks. J. Softw. Evol. Process 24(8), 877–894 (2012)

    Article  Google Scholar 

  21. Khan S.U., Niazi M., Rashid A.: Barriers in the selection of offshore software development outsourcing vendors: an exploratory study using a systematic literature review. J. Inf. Softw. Technol. 53(7), 693–706 (2011)

    Article  Google Scholar 

  22. Niazi M., Wilson D., Zowghi D.: A maturity model for the implementation of software process improvement: an empirical study. J. Syst. Softw. 74(2), 155–172 (2005)

    Article  Google Scholar 

  23. Niazi, M.: Software process improvement: a road to success. In: The 7th International Conference on Product Focused Software Process Improvement, LNCS (2006)

  24. Cox K., Niazi M., Verner J.: An empirical study of Sommerville’s requirements engineering practices. IET Softw. J. 3(5), 339–355 (2009)

    Article  Google Scholar 

  25. Niazi, M.; Cox, K.; Verner, J.: An empirical study identifying high perceived value requirements engineering practices. In: Advances in Information Systems Development, Fourteenth International Conference on Information Systems Development (ISD’2005) Karlstad University, Sweden 15–17 Aug. Springer (2006)

  26. McCabe G.P., Moore D.S.: Introduction to the Practice of Statistics, 4th edn. W.H.Freeman and Co, New York (2002)

    Google Scholar 

  27. Chen, L.: Systematic literature review: experiences and observations from the trenches. Keele University UK Seminar series (2009)

  28. Hossain, E.; Ali-Babar, M.; Paik, H.-Y.: Using scrum in global software development: a systematic literature review. In: IEEE International Conference on Global Software Engineering, ICGSE09. Lero, Limerick, Ireland (2009)

  29. Alshayeb M.: The impact of refactoring to patterns on software quality attributes. Arab. J. Sci. Eng. 36(7), 1241–1251 (2011)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Mahmood Niazi.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Niazi, M. Do Systematic Literature Reviews Outperform Informal Literature Reviews in the Software Engineering Domain? An Initial Case Study. Arab J Sci Eng 40, 845–855 (2015). https://doi.org/10.1007/s13369-015-1586-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13369-015-1586-0

Keywords

Navigation