Journal of Information Technology

, Volume 30, Issue 2, pp 161–173 | Cite as

On being ‘systematic’ in literature reviews in IS

  • Sebastian K Boell
  • Dubravka Cecez-Kecmanovic
Debates and Perspectives

Abstract

General guidelines for conducting literature reviews often do not address the question of literature searches and dealing with a potentially large number of identified sources. These issues are specifically addressed by so-called systematic literature reviews (SLRs) that propose a strict protocol for the search and appraisal of literature. Moreover, SLRs are claimed to be a ‘standardized method’ for literature reviews that is replicable, transparent, objective, unbiased and rigorous, and thus superior to other approaches for conducting literature reviews. These are significant and consequential claims that – despite increasing adoption of SLRs – remained largely unnoticed in the information systems (IS) literature. The objective of this debate is to draw attention of the IS community to SLR’s claims, to question their justification and reveal potential risks of their adoption. This is achieved by first examining the origins of SLR and the prescribed SLR process and then by critically assessing their claims and implications. In this debate, we show that SLRs are applicable and useful for a very specific kind of literature review, a meta study that identifies and summarizes evidence from earlier research. We also demonstrate that the claims that SLRs provide superior quality are not justified. More importantly, we argue that SLR as a general approach to conducting literature reviews is highly questionable, concealing significant perils. The paper cautions that SLR could undermine critical engagement with literature and what it means to be scholarly in academic work.

Keywords

systematic literature review (SLR) systematic review conducting literature reviews literature review narrative literature review database searches literature search 

References

  1. Agrawal, A., Boese, M. and Sarker, S. (2010). A Review of the HCI Literature in IS: The Missing Links of Computer-Mediated Communication, Culture, and Interaction, in AMCIS 2010 Proceedings, Lima, Peru, Paper 523.Google Scholar
  2. Alvesson, M. and Sandberg, J. (2011). Generating Research Questions through Problematization, Academy of Management Review 36 (2): 247–271.Google Scholar
  3. Amrollahi, A., Ghapanchi, A.H. and Talaei-Khoei, A. (2013). A Systematic Literature Review on Strategic Information Systems Planning: Insights from the past decade, Pacific Asia Journal of the Association for Information Systems 5 (2): 39–66.Google Scholar
  4. Atkins, C. and Louw, G. (2000). Reclaiming Knowledge: A Case for Evidence Based Information Systems Reclaiming Knowledge: A Case for Evidence-Based Information Systems, in ECIS 2000, Vienna, Austria, Paper 28.Google Scholar
  5. Bandara, W., Miskon, S. and Fielt, E. (2011). A Systematic, Tool-Supported Method for Conducting Literature Reviews in Information Systems, in ECIS 2011 Proceedings, Helsinki, Finnland, Paper 221.Google Scholar
  6. Basten, D. and Sunyaev, A. (2014). A Systematic Mapping of Factors Affecting Accuracy of Software Development Effort Estimation, Communications of the Association for Information Systems 34: 51–86.Google Scholar
  7. Baumeister, R.E. and Leary, M.R. (1997). Writing Narrative Literature Reviews, Review of General Psychology 1 (3): 311–320.CrossRefGoogle Scholar
  8. Blair, D. (2006). Wittgenstein, Language and Information. Back to the Rough Ground! Dordrecht: Springer.Google Scholar
  9. Bodoff, D. (2009). Emergence of Terminological Conventions as a Searcher-Indexer Coordination Game, Journal of the American Society for Information Science and Technology 60 (12): 2509–2529.CrossRefGoogle Scholar
  10. Boell, S.K. and Cecez-Kecmanovic, D. (2014). A Hermeneutic Approach for Conducting Literature Reviews and Literature Searches, Communications of the Association for Information Systems 34: 257–286.Google Scholar
  11. Boote, D.N. and Beile, P. (2005). Scholars Before Researchers: On the centrality of the dissertation literature review in research preparation, Educational Researcher 34 (6): 3–15.CrossRefGoogle Scholar
  12. vom Brocke, J., Simons, A., Niehaves, B., Reimer, K., Plattfaut, R. and Cleven, A. (2009). Reconstructing the Giant: On the Importance of Rigour in Documenting the Literature Search Process, in ECIS 2009 Proceedings, Verona, Italy, Paper 161.Google Scholar
  13. Buckland, M. and Gey, F. (1994). The Relationship between Recall and Precision, Journal of the American Society for Information Science 45 (1): 12–19.CrossRefGoogle Scholar
  14. Campbell Collaboration (2007). The Campbell Collaboration. What Helps? What Harms? Based on what Evidence? [www document] http://www.campbellcollaboration.org (accessed 10 May 2014).
  15. Chalmers, I. and Altman, D.G. (1995). Systematic Reviews, London: BMJ.Google Scholar
  16. Checkland, P. and Holwell, S. (1998). Information, Systems and Information Systems: Making sense of the field, Chichester: Wiley.Google Scholar
  17. Cheung, C.M.K. and Thadani, D.R. (2010). The Effectiveness of Electronic Word-of-Mouth Communication: A Literature Analysis, in BLED 2010 Proceedings, Bled, Slovenia, Paper 18.Google Scholar
  18. Clarke, M.J. and Stewart, L.A. (1995). Obtaining Data from Randomized Controlled Trials: How much do we need for reliable and informative meta-analyses? in I. Chalmers and D.G. Altman (eds.) Systematic Reviews, London: BMJ, pp. 37–47.Google Scholar
  19. Combs, J.P., Bustamante, R.M. and Onwuegbuzie, A.J. (2010). An Interactive Model for Facilitating Development of Literature Reviews, International Journal of Multiple Research Approaches 4 (2): 159–182.CrossRefGoogle Scholar
  20. Constantinides, P., Chiasson, M.W. and Introna, L.D. (2012). The Ends of Information Systems Research: A pragmatic framework, MIS Quarterly 36 (1): 1–19.Google Scholar
  21. Cruzes, D.S. and Dybå, T. (2011). Research Synthesis in Software Engineering: A tertiary study, Information and Software Technology 53 (5): 440–455.CrossRefGoogle Scholar
  22. Davies, W.M. and Beaumont, T.J. (2007). Literature Reviews, Melbourne: Teaching and Learning Unit.Google Scholar
  23. Davis, F.D. (1989). Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology, MIS Quarterly 13 (3): 319–339.CrossRefGoogle Scholar
  24. Dellinger, A.B. (2005). Validity and the Review of the Literature, Research in the Schools 12 (2): 41–54.Google Scholar
  25. Denyer, D. and Tranfield, D. (2009). Producing a Systematic Review, in D.A. Buchanan and A. Bryman (eds.) The Sage Handbook of Organizational Research Methods, Thousand Oaks, CA: Sage, pp. 671–689.Google Scholar
  26. DiMaggio, P.J. (1995). Comments on ‘What Theory is Not’, Administrative Science Quarterly 40 (3): 391–397.CrossRefGoogle Scholar
  27. Dong, Y.R. (1996). Learning How to Use Citations for Knowledge Transformation: Non-native doctoral students’ dissertation writing in science, Research in Teaching of English 30 (4): 428–457.Google Scholar
  28. Dwivedi, Y., Williams, M.D., Lal, B. and Schwarz, A. (2008). Profiling Adoption, Acceptance and Diffusion Research in the Information Systems Discipline, in ECIS 2008 Proceedings, Galway, Ireland, Paper 112.Google Scholar
  29. Evidence-based Medicine Working Group (1992). Evidence-based Medicine: A new approach to teaching the practice of medicine, Journal of the American Medical Association 268 (17): 2420–2425.Google Scholar
  30. Eysenck, H.J. (1995). Problems with Meta-Analysis, in I. Chalmers and D.G. Altman (eds.) Systematic Reviews, London: BMJ, pp. 64–74.Google Scholar
  31. Feak, C.B. and Swales, J.M. (2009). Telling a Research Story: Writing a literature review, Ann Arbor, MI: University of Michigan Press.CrossRefGoogle Scholar
  32. Finfgeld-Connett, D. and Johnson, E.D. (2013). Literature Search Strategies for Conducting Knowledge-Building and Theory-Generating Qualitative Systematic Reviews, Journal of Advanced Nursing 69 (1): 194–204.CrossRefGoogle Scholar
  33. Finn, J.A. (2005). Getting a PhD: An action plan to help manage your research, your supervisor and your project, London: Routledge.Google Scholar
  34. Fugmann, R. (2007). Informationstheorie: Der Jahrhundertbluff. EineZeitkritische Betrachtung (Teil 1), Information Wissenschaft und Praxis 58 (8): 449–458.Google Scholar
  35. Goeken, M. and Patas, J. (2010). Evidence-Based Structuring and Evaluation of Empirical Research in Requirements Engineering: Fundamentals, framework, research map, Business and Information Systems Engineering 2 (3): 175–185.CrossRefGoogle Scholar
  36. Goodfellow, J. (1998). Constructing a Narrative, in J. Higgs (ed.) Writing Qualitative Research, Sydney: Hampden Press, pp. 175–187.Google Scholar
  37. Grahlmann, K.R., Helms, R.W., Hilhorst, C., Brinkkemper, S. and Van Amerongen, S. (2012). Reviewing Enterprise Content Management: A functional framework, European Journal of Information Systems 21 (3): 268–286.CrossRefGoogle Scholar
  38. Gräning, A., Felden, C. and Piechocki, M. (2011). Status Quo and Potential of XBRL for Business and Information Systems Engineering, Business and Information Systems Engineering 3 (4): 231–239.CrossRefGoogle Scholar
  39. Green, B.N., Johnson, C.D. and Adams, A. (2006). Writing Narrative Literature Reviews for Peer-Reviewed Journals: Secrets of the trade, Journal of Chiropractic Medicine 5 (3): 101–117.CrossRefGoogle Scholar
  40. Greenhalgh, T. and Peacock, R. (2005). Effectiveness and Efficiency of Search Methods in Systematic Reviews of Complex Evidence: Audit of primary sources, British Medical Journal 331 (7524): 1064–1065.CrossRefGoogle Scholar
  41. Guillemette, M.G. and Paré, G. (2012). Toward a New Theory of the Contribution of the IT Function in Organizations, MIS Quarterly 36 (2): 529–551.Google Scholar
  42. Hammersley, M. (2001). On ‘Systematic’ Reviews of Research Literatures: A ‘narrative’ response to Evans & Benefield, British Educational Research Journal 27 (5): 543–554.CrossRefGoogle Scholar
  43. Hart, C. (1998). Doing a Literature Review: Releasing the social science research imagination, Thousand Oaks: Sage Publications.Google Scholar
  44. Hartley, J. and Betts, L. (2009). Common Weaknesses in Traditional Abstracts in the Social Sciences, Journal of the American Society for Information Science and Technology 60 (10): 2010–2018.CrossRefGoogle Scholar
  45. Hjorland, B. (2011). Evidence-Based Practice: An analysis based on the philosophy of science, Journal of the American Society for Information Science 62 (7): 1301–1310.CrossRefGoogle Scholar
  46. Holmes, D., Murray, S.J., Perron, A. and McCabe, J. (2008). Nursing Best Practice Guidelines: Reflecting on the obscene rise of the void, Journal of Nursing Management 16 (4): 394–403.CrossRefGoogle Scholar
  47. Hood, W.W. and Wilson, C.S. (2001). The Scatter of Documents Over Databases in Different Subject Domains: How many databases are needed? Journal of the American Society for Information Science and Technology 52 (14): 1242–1254.CrossRefGoogle Scholar
  48. Hummel, M., Rosenkranz, C. and Holten, R. (2003). The Role of Communication in Agile Systems Development: An analysis of the state of the art, Business and Information Systems Engineering 5 (5): 343–355.CrossRefGoogle Scholar
  49. Jalali, S. and Wohlin, C. (2012). Systematic Literature Studies: Database Searches vs. Backward Snowballing, in International Conference on Empirical Software Engineering and Measurement, Lund, Sweden: ACM, 29–38.Google Scholar
  50. Khoo, C.S.G., Na, J.-C. and Jaidka, K. (2011). Analysis of the Macro-level Discourse Structure of Literature Reviews, Online Information Review 35 (2): 255–271.CrossRefGoogle Scholar
  51. Kitchenham, B. (2004). Procedures for Performing Systematic Reviews, Keele, Eversleigh: Keele University and NICTA, Technical Report.Google Scholar
  52. Kitchenham, B. and Charters, S. (2007). Guidelines for performing systematic literature reviews in software engineering [www document] http://www.dur.ac.uk/ebse/resources/guidelines/Systematic-reviews-5-8.pdf (accessed 10 May 2014).
  53. Knipschild, P. (1995). Some Examples of Systematic Reviews, in I. Chalmers and D.G. Altman (eds.) Systematic Reviews, London: BMJ, pp. 9–16.Google Scholar
  54. Kuhn, T.S. (1962). The Structure of Scientific Revolutions, Chicago: University of Chicago Press.Google Scholar
  55. Kwan, B.S.C. (2008). The Nexus of Reading, Writing and Researching in the Doctoral Undertaking of Humanities and Social Sciences: Implications for literature reviewing, English for Specific Purposes 27 (1): 42–56.CrossRefGoogle Scholar
  56. Kwan, B.S.C., Chan, H. and Lam, C. (2012). Evaluating Prior Scholarship in Literature Reviews of Research Articles: A comparative study of practices in two research paradigms, English for Specific Purposes 31 (3): 188–201.CrossRefGoogle Scholar
  57. Lacity, M.C., Solomon, S., Yan, A. and Willcocks, L.P. (2011). Business Process Outsourcing Studies: A critical review and research directions, Journal of Information Technology 26 (4): 221–258.CrossRefGoogle Scholar
  58. Leite, J. C. S. do P. and Cappelli, C. (2010). Software Transparency, Business and Information Systems Engineering 2 (3): 127–139.CrossRefGoogle Scholar
  59. Leonardi, P.M. and Barley, S.R. (2010). What’s Under Construction Here? Social Action, Materiality, and Power in Constructivist Studies of Technology and Organizing, The Academy of Management Annals 4 (1): 1–51.CrossRefGoogle Scholar
  60. LePine, J.A. and Wilcox-King, A. (2010). Editors’ Comments: Developing novel theoretical insight from reviews of existing theory and research, Academy of Management Review 35 (4): 506–509.CrossRefGoogle Scholar
  61. Levy, Y. and Ellis, T.J. (2006). A Systems Approach to Conduct an Effective Literature Review in Support of Information Systems Research, Informing Science Journal 9: 181–212.Google Scholar
  62. Machi, L.A. and McEvoy, B.T. (2012). The Literature Review: Six steps to success. 2nd ed. Thousand Oaks: Corwin Press.Google Scholar
  63. MacLure, M. (2005). ‘Clarity Bordering on Stupidity’: Where’s the quality in systematic review? Journal of Education Policy 20 (4): 393–416.CrossRefGoogle Scholar
  64. Merschbrock, C. and Munkvold, B.E. (2012). A Research Review on Building Information Modeling in Construction — An Area Ripe for IS Research, Communications of the Association for Information Systems 31: 207–228.Google Scholar
  65. Mettler, T., Eurich, M. and Winter, R. (2014). On the Use of Experiments in Design Science Research: A Proposition of an evaluation framework, Communications of the Association for Information Systems 34 (1): 223–240.Google Scholar
  66. MISQ (2006). Objectives of the MISQ theory and review [www document] http://www.misq.org/skin/frontend/default/misq/pdf/TheoryReview/TRObjectives.pdf (accessed 10 May 2014).
  67. Mohan, K. and Ahlemann, F. (2011). Understanding Acceptance of Information System Development and Management Methodologies by actual Users: A Review and Assessment of Existing Literature, in Wirtschaftsinformatik Proceedings 2011, Zürich, Switzerland, Paper 41.Google Scholar
  68. Morrell, K. (2008). The Narrative of ‘Evidence Based’ Management: A polemic, Journal of Management Studies 45 (3): 613–635.CrossRefGoogle Scholar
  69. Mulrow, C.D. (1995). Rationale for Systematic Reviews, in I. Chalmers and D.G. Altman (eds.) Systematic Reviews, London: BMJ, pp. 1–8.Google Scholar
  70. Murray, S.J., Holmes, D., Perron, A. and Rail, G. (2007). No Exit? Intellectual Integrity Under the Regime of ‘Evidence’ and ‘Best-practices’, Journal of Clinical Practice 13 (4): 512–516.Google Scholar
  71. Neuman, W.L. (2011). Social Research Methods: Quantitative and qualitative approaches, 7th ed. Boston, MA: Pearson.Google Scholar
  72. Oancea, A. and Pring, R. (2008). The Importance of Being Thorough: On systematic accumulations of ‘what works’ in education research, Journal of Philosophy of Education 42 (4): 15–39.CrossRefGoogle Scholar
  73. Oates, B. (2011). Evidence Based Information Systems: A Decade Later, in ECIS 2011 Proceedings, Helsinki, Finnland, Paper 222.Google Scholar
  74. Oates, B.J., Edwards, H.M. and Wainwright, D.W. (2012). A Model-Driven Method for the Systematic Literature Review of Qualitative Empirical Research, in ICIS 2012 Proceedings, Shanghai, China, 1–18.Google Scholar
  75. Okoli, C. and Schabram, K. (2009). Protocol for a Systematic Literature Review of Research on the Wikipedia, Sprouts: Working Papers on Information Systems 9(65).Google Scholar
  76. Okoli, C. and Schabram, K. (2010). A Guide to Conducting a Systematic Literature Review of Information Systems Research, Sprouts: Working Papers on Information Systems 10(26).Google Scholar
  77. Onwuegbuzie, Anthony J., Collins, K.M.T., Leech, N.L., Dellinger, A.B. and Jiao, Q.G. (2007). Mixed Methods+Literature Reviews=Mixed Research Syntheses: A Framework for Conducting and Writing Rigorous, Comprehensive, and Insightful Literature Reviews, in The World of Educational Quality. Annual Meeting of the American Educational Research Association, Chicago.Google Scholar
  78. Oxman, A.D. (1995). Checklists for Review Articles, in I. Chalmers and D.G. Altman (eds.) Systematic Reviews, London: BMJ, pp. 75–85.Google Scholar
  79. Pawson, R. (2006). Digging for Nuggets: How ‘bad’ research can yield ‘good’ evidence, in I. Chalmers and D.G. Altman (eds.) International Journal of Social Research Methodology 9 (2): 127–142.CrossRefGoogle Scholar
  80. Perry, C. (1998). A Structured Approach for Presenting Theses, Australasian Marketing Journal 6 (1): 63–85.CrossRefGoogle Scholar
  81. Petticrew, M. and Roberts, H. (2006). Systematic Reviews in the Social Sciences: A practical guide, Malden: Blackwell.CrossRefGoogle Scholar
  82. Ramsay, C.R., Grant, A.M., Wallace, S.A., Garthwaite, P.H., Monk, A.F. and Russell, I.T. (2000). Assessment of the Learning Curve in Health Technologies – A Systematic Review, International Journal of Technology Assessment in Health Care 16 (4): 1095–1108.CrossRefGoogle Scholar
  83. Ridley, D. (2008). The Literature Review: A step-by-step guide for students, Thousand Oaks: Sage.Google Scholar
  84. Rose, S., Bisson, J. and Wessely, S. (2003). A Systematic Review of Single-Session Psychological Interventions (‘Debriefing’) Following Trauma, Psychotherapy and Psychosomatics 72 (4): 176–184.CrossRefGoogle Scholar
  85. Roztocki, N. and Weistroffer, H.R. (2008). Event Studies in Information Systems Research: A Review, in AMCIS 2008 Proceedings, Toronto, Canada, Paper 248.Google Scholar
  86. Salton, G. and McGill, M.J. (1983). Introduction to Modern Information Retrieval, New York: McGraw-Hill.Google Scholar
  87. Sandelowski, M. (2008). Reading, Writing and Systematic Review, Journal of Advanced Nursing 64 (1): 104–10.CrossRefGoogle Scholar
  88. Sandelowski, M., Voils, C.I. and Barroso, J. (2007). Comparability Work and the Management of Difference in Research Synthesis Studies, Social Science & Medicine (1982) 64 (1): 236–47.CrossRefGoogle Scholar
  89. Schultze, U. and Leidner, D.E. (2002). Studying Knowledge Management in Information Systems Research: Discourses and theoretical assumptions, MIS Quarterly 26 (3): 213–242.CrossRefGoogle Scholar
  90. Schwarz, A., Mehta, M., Johnson, N. and Chin, W.W. (2007). Understanding Frameworks and Reviews: A commentary to assist us in moving our field forward by analyzing our past, DATA BASE for Advances in Information Systems 38 (3): 29–50.CrossRefGoogle Scholar
  91. Shiffman, R.N., Liaw, Y., Brandt, C.A. and Corb, G.J. (1999). Computer-based Guideline Implementation Systems: A systematic review of functionality and effectiveness, Journal of the American Medical Informatics Association 6 (2): 104–114.CrossRefGoogle Scholar
  92. Staples, M. and Niazi, M. (2007). Experiences Using Systematic Review Guidelines, Journal of Systems and Software 80 (9): 1425–1437.CrossRefGoogle Scholar
  93. Stol, K.-J., Babar, M.A., Russo, B. and Fitzgerald, B. (2009). The Use of Empirical Methods in Open Source Software Research: Facts, Trends and Future Directions, in FLOSS’09, May 18, 2009, Vancouver, Canada, 19–24.Google Scholar
  94. Sutton, R. and Staw, B.M. (1995). Forum What Theory is Not, Administrative Science Quarterly 40 (3): 371–384.CrossRefGoogle Scholar
  95. Tamm, T., Seddon, P.B., Shanks, G. and Reynolds, P. (2011). How Does Enterprise Architecture Add Value to Organisations?, Communications of the Association for Information Systems 28: 141–168.Google Scholar
  96. Thompson, S.G. (1995). Why Sources of Heterogenity in Meta-Analysis Should be Investigated, in I. Chalmers and D.G. Altman (eds.) Systematic Reviews, London: BMJ, pp. 48–63.Google Scholar
  97. Tranfield, D., Denyer, D. and Smart, P. (2003). Towards a Methodology for Developing Evidence-Informed Management Knowledge by Means of Systematic Review, British Journal of Management 14 (3): 207–222.CrossRefGoogle Scholar
  98. Wang, Q.E., Myers, M.D. and Sundaram, D. (2013). Digital Natives and Digital Immigrants: Towards a model of digital fluency, Business and Information Systems Engineering 5 (6): 409–419.CrossRefGoogle Scholar
  99. Webster, J. and Watson, R.T. (2002). Analyzing the Past to Prepare for the Future: Writing a literature review, MIS Quarterly 26 (2): xiii–xxiii.Google Scholar
  100. Weerakkody, V., Dwivedi, Y.K. and Irani, Z. (2009). The Diffusion and Use of Institutional Theory: A cross-disciplinary longitudinal literature survey, Journal of Information Technology 24 (4): 354–368.CrossRefGoogle Scholar
  101. Whetten, D.A. (1989). What Constitutes a Theoretical Contribution?, Academy of Management Review 14 (4): 490–495.CrossRefGoogle Scholar
  102. Williams, M.D., Dwivedi, Y.K., Lal, B. and Schwarz, A. (2009). Contemporary Trends and Issues in IT Adoption and Diffusion Research, Journal of Information Technology 24 (1): 1–10.CrossRefGoogle Scholar
  103. Wolfswinkel, J.F., Furtmueller, E. and Wilderom, C.P.M. (2013). Using Grounded Theory as a Method for Rigorously Reviewing Literature, European Journal of Information Systems 22 (1): 45–55.CrossRefGoogle Scholar
  104. Wolfswinkel, J., Furtmueller, E. and Wilderom, C. (2010). Reflecting on E-Recruiting Research Using Grounded Theory, in ECIS 2010 Proceedings, Pretoria, South Africa, Paper 52.Google Scholar
  105. Wright Mills, C. (1978 [1959]). The Sociological Imagination, Oxford: Oxford University Press.Google Scholar
  106. Zhang, P., Li, N., Scialdone, M.J. and Carey, J. (2008). The Intellectual Advancement of Human-Computer Interaction Research: A critical assessment of the MIS literature (1990–2008), AIS Transactions on Human-Computer Interaction 1 (3): 55–107.Google Scholar

Copyright information

© Association for Information Technology Trust 2014

Authors and Affiliations

  • Sebastian K Boell
    • 1
  • Dubravka Cecez-Kecmanovic
    • 2
  1. 1.Discipline of Business Information Systems, The University of Sydney Business SchoolSydneyAustralia
  2. 2.UNSW Business School, University of New South WalesSydneyAustralia

Personalised recommendations