General guidelines for conducting literature reviews often do not address the question of literature searches and dealing with a potentially large number of identified sources. These issues are specifically addressed by so-called systematic literature reviews (SLRs) that propose a strict protocol for the search and appraisal of literature. Moreover, SLRs are claimed to be a ‘standardized method’ for literature reviews that is replicable, transparent, objective, unbiased and rigorous, and thus superior to other approaches for conducting literature reviews. These are significant and consequential claims that – despite increasing adoption of SLRs – remained largely unnoticed in the information systems (IS) literature. The objective of this debate is to draw attention of the IS community to SLR’s claims, to question their justification and reveal potential risks of their adoption. This is achieved by first examining the origins of SLR and the prescribed SLR process and then by critically assessing their claims and implications. In this debate, we show that SLRs are applicable and useful for a very specific kind of literature review, a meta study that identifies and summarizes evidence from earlier research. We also demonstrate that the claims that SLRs provide superior quality are not justified. More importantly, we argue that SLR as a general approach to conducting literature reviews is highly questionable, concealing significant perils. The paper cautions that SLR could undermine critical engagement with literature and what it means to be scholarly in academic work.
This is a preview of subscription content, access via your institution.
On 22 February 2014 we searched for the phrases ‘SLR’ and ‘SR’ in the AIS eLibrary and in Scopus. Our search specifically included all search fields and no limitation regarding publication date. However, in Scopus we limited our search to articles published in the AIS basket of eight journals: European Journal of Information Systems, Information Systems Journal, Information Systems Research, Journal of Information Technology, Journal of Management Systems, Journal of Strategic Management Systems, Journal of the AIS, and MIS Quarterly. Our search returned 498 results from the AIS eLibrary and 42 from Scopus, with only 3 publications overlapping.
Agrawal, A., Boese, M. and Sarker, S. (2010). A Review of the HCI Literature in IS: The Missing Links of Computer-Mediated Communication, Culture, and Interaction, in AMCIS 2010 Proceedings, Lima, Peru, Paper 523.
Alvesson, M. and Sandberg, J. (2011). Generating Research Questions through Problematization, Academy of Management Review 36 (2): 247–271.
Amrollahi, A., Ghapanchi, A.H. and Talaei-Khoei, A. (2013). A Systematic Literature Review on Strategic Information Systems Planning: Insights from the past decade, Pacific Asia Journal of the Association for Information Systems 5 (2): 39–66.
Atkins, C. and Louw, G. (2000). Reclaiming Knowledge: A Case for Evidence Based Information Systems Reclaiming Knowledge: A Case for Evidence-Based Information Systems, in ECIS 2000, Vienna, Austria, Paper 28.
Bandara, W., Miskon, S. and Fielt, E. (2011). A Systematic, Tool-Supported Method for Conducting Literature Reviews in Information Systems, in ECIS 2011 Proceedings, Helsinki, Finnland, Paper 221.
Basten, D. and Sunyaev, A. (2014). A Systematic Mapping of Factors Affecting Accuracy of Software Development Effort Estimation, Communications of the Association for Information Systems 34: 51–86.
Baumeister, R.E. and Leary, M.R. (1997). Writing Narrative Literature Reviews, Review of General Psychology 1 (3): 311–320.
Blair, D. (2006). Wittgenstein, Language and Information. Back to the Rough Ground! Dordrecht: Springer.
Bodoff, D. (2009). Emergence of Terminological Conventions as a Searcher-Indexer Coordination Game, Journal of the American Society for Information Science and Technology 60 (12): 2509–2529.
Boell, S.K. and Cecez-Kecmanovic, D. (2014). A Hermeneutic Approach for Conducting Literature Reviews and Literature Searches, Communications of the Association for Information Systems 34: 257–286.
Boote, D.N. and Beile, P. (2005). Scholars Before Researchers: On the centrality of the dissertation literature review in research preparation, Educational Researcher 34 (6): 3–15.
vom Brocke, J., Simons, A., Niehaves, B., Reimer, K., Plattfaut, R. and Cleven, A. (2009). Reconstructing the Giant: On the Importance of Rigour in Documenting the Literature Search Process, in ECIS 2009 Proceedings, Verona, Italy, Paper 161.
Buckland, M. and Gey, F. (1994). The Relationship between Recall and Precision, Journal of the American Society for Information Science 45 (1): 12–19.
Campbell Collaboration (2007). The Campbell Collaboration. What Helps? What Harms? Based on what Evidence? [www document] http://www.campbellcollaboration.org (accessed 10 May 2014).
Chalmers, I. and Altman, D.G. (1995). Systematic Reviews, London: BMJ.
Checkland, P. and Holwell, S. (1998). Information, Systems and Information Systems: Making sense of the field, Chichester: Wiley.
Cheung, C.M.K. and Thadani, D.R. (2010). The Effectiveness of Electronic Word-of-Mouth Communication: A Literature Analysis, in BLED 2010 Proceedings, Bled, Slovenia, Paper 18.
Clarke, M.J. and Stewart, L.A. (1995). Obtaining Data from Randomized Controlled Trials: How much do we need for reliable and informative meta-analyses? in I. Chalmers and D.G. Altman (eds.) Systematic Reviews, London: BMJ, pp. 37–47.
Combs, J.P., Bustamante, R.M. and Onwuegbuzie, A.J. (2010). An Interactive Model for Facilitating Development of Literature Reviews, International Journal of Multiple Research Approaches 4 (2): 159–182.
Constantinides, P., Chiasson, M.W. and Introna, L.D. (2012). The Ends of Information Systems Research: A pragmatic framework, MIS Quarterly 36 (1): 1–19.
Cruzes, D.S. and Dybå, T. (2011). Research Synthesis in Software Engineering: A tertiary study, Information and Software Technology 53 (5): 440–455.
Davies, W.M. and Beaumont, T.J. (2007). Literature Reviews, Melbourne: Teaching and Learning Unit.
Davis, F.D. (1989). Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology, MIS Quarterly 13 (3): 319–339.
Dellinger, A.B. (2005). Validity and the Review of the Literature, Research in the Schools 12 (2): 41–54.
Denyer, D. and Tranfield, D. (2009). Producing a Systematic Review, in D.A. Buchanan and A. Bryman (eds.) The Sage Handbook of Organizational Research Methods, Thousand Oaks, CA: Sage, pp. 671–689.
DiMaggio, P.J. (1995). Comments on ‘What Theory is Not’, Administrative Science Quarterly 40 (3): 391–397.
Dong, Y.R. (1996). Learning How to Use Citations for Knowledge Transformation: Non-native doctoral students’ dissertation writing in science, Research in Teaching of English 30 (4): 428–457.
Dwivedi, Y., Williams, M.D., Lal, B. and Schwarz, A. (2008). Profiling Adoption, Acceptance and Diffusion Research in the Information Systems Discipline, in ECIS 2008 Proceedings, Galway, Ireland, Paper 112.
Evidence-based Medicine Working Group (1992). Evidence-based Medicine: A new approach to teaching the practice of medicine, Journal of the American Medical Association 268 (17): 2420–2425.
Eysenck, H.J. (1995). Problems with Meta-Analysis, in I. Chalmers and D.G. Altman (eds.) Systematic Reviews, London: BMJ, pp. 64–74.
Feak, C.B. and Swales, J.M. (2009). Telling a Research Story: Writing a literature review, Ann Arbor, MI: University of Michigan Press.
Finfgeld-Connett, D. and Johnson, E.D. (2013). Literature Search Strategies for Conducting Knowledge-Building and Theory-Generating Qualitative Systematic Reviews, Journal of Advanced Nursing 69 (1): 194–204.
Finn, J.A. (2005). Getting a PhD: An action plan to help manage your research, your supervisor and your project, London: Routledge.
Fugmann, R. (2007). Informationstheorie: Der Jahrhundertbluff. EineZeitkritische Betrachtung (Teil 1), Information Wissenschaft und Praxis 58 (8): 449–458.
Goeken, M. and Patas, J. (2010). Evidence-Based Structuring and Evaluation of Empirical Research in Requirements Engineering: Fundamentals, framework, research map, Business and Information Systems Engineering 2 (3): 175–185.
Goodfellow, J. (1998). Constructing a Narrative, in J. Higgs (ed.) Writing Qualitative Research, Sydney: Hampden Press, pp. 175–187.
Grahlmann, K.R., Helms, R.W., Hilhorst, C., Brinkkemper, S. and Van Amerongen, S. (2012). Reviewing Enterprise Content Management: A functional framework, European Journal of Information Systems 21 (3): 268–286.
Gräning, A., Felden, C. and Piechocki, M. (2011). Status Quo and Potential of XBRL for Business and Information Systems Engineering, Business and Information Systems Engineering 3 (4): 231–239.
Green, B.N., Johnson, C.D. and Adams, A. (2006). Writing Narrative Literature Reviews for Peer-Reviewed Journals: Secrets of the trade, Journal of Chiropractic Medicine 5 (3): 101–117.
Greenhalgh, T. and Peacock, R. (2005). Effectiveness and Efficiency of Search Methods in Systematic Reviews of Complex Evidence: Audit of primary sources, British Medical Journal 331 (7524): 1064–1065.
Guillemette, M.G. and Paré, G. (2012). Toward a New Theory of the Contribution of the IT Function in Organizations, MIS Quarterly 36 (2): 529–551.
Hammersley, M. (2001). On ‘Systematic’ Reviews of Research Literatures: A ‘narrative’ response to Evans & Benefield, British Educational Research Journal 27 (5): 543–554.
Hart, C. (1998). Doing a Literature Review: Releasing the social science research imagination, Thousand Oaks: Sage Publications.
Hartley, J. and Betts, L. (2009). Common Weaknesses in Traditional Abstracts in the Social Sciences, Journal of the American Society for Information Science and Technology 60 (10): 2010–2018.
Hjorland, B. (2011). Evidence-Based Practice: An analysis based on the philosophy of science, Journal of the American Society for Information Science 62 (7): 1301–1310.
Holmes, D., Murray, S.J., Perron, A. and McCabe, J. (2008). Nursing Best Practice Guidelines: Reflecting on the obscene rise of the void, Journal of Nursing Management 16 (4): 394–403.
Hood, W.W. and Wilson, C.S. (2001). The Scatter of Documents Over Databases in Different Subject Domains: How many databases are needed? Journal of the American Society for Information Science and Technology 52 (14): 1242–1254.
Hummel, M., Rosenkranz, C. and Holten, R. (2003). The Role of Communication in Agile Systems Development: An analysis of the state of the art, Business and Information Systems Engineering 5 (5): 343–355.
Jalali, S. and Wohlin, C. (2012). Systematic Literature Studies: Database Searches vs. Backward Snowballing, in International Conference on Empirical Software Engineering and Measurement, Lund, Sweden: ACM, 29–38.
Khoo, C.S.G., Na, J.-C. and Jaidka, K. (2011). Analysis of the Macro-level Discourse Structure of Literature Reviews, Online Information Review 35 (2): 255–271.
Kitchenham, B. (2004). Procedures for Performing Systematic Reviews, Keele, Eversleigh: Keele University and NICTA, Technical Report.
Kitchenham, B. and Charters, S. (2007). Guidelines for performing systematic literature reviews in software engineering [www document] http://www.dur.ac.uk/ebse/resources/guidelines/Systematic-reviews-5-8.pdf (accessed 10 May 2014).
Knipschild, P. (1995). Some Examples of Systematic Reviews, in I. Chalmers and D.G. Altman (eds.) Systematic Reviews, London: BMJ, pp. 9–16.
Kuhn, T.S. (1962). The Structure of Scientific Revolutions, Chicago: University of Chicago Press.
Kwan, B.S.C. (2008). The Nexus of Reading, Writing and Researching in the Doctoral Undertaking of Humanities and Social Sciences: Implications for literature reviewing, English for Specific Purposes 27 (1): 42–56.
Kwan, B.S.C., Chan, H. and Lam, C. (2012). Evaluating Prior Scholarship in Literature Reviews of Research Articles: A comparative study of practices in two research paradigms, English for Specific Purposes 31 (3): 188–201.
Lacity, M.C., Solomon, S., Yan, A. and Willcocks, L.P. (2011). Business Process Outsourcing Studies: A critical review and research directions, Journal of Information Technology 26 (4): 221–258.
Leite, J. C. S. do P. and Cappelli, C. (2010). Software Transparency, Business and Information Systems Engineering 2 (3): 127–139.
Leonardi, P.M. and Barley, S.R. (2010). What’s Under Construction Here? Social Action, Materiality, and Power in Constructivist Studies of Technology and Organizing, The Academy of Management Annals 4 (1): 1–51.
LePine, J.A. and Wilcox-King, A. (2010). Editors’ Comments: Developing novel theoretical insight from reviews of existing theory and research, Academy of Management Review 35 (4): 506–509.
Levy, Y. and Ellis, T.J. (2006). A Systems Approach to Conduct an Effective Literature Review in Support of Information Systems Research, Informing Science Journal 9: 181–212.
Machi, L.A. and McEvoy, B.T. (2012). The Literature Review: Six steps to success. 2nd ed. Thousand Oaks: Corwin Press.
MacLure, M. (2005). ‘Clarity Bordering on Stupidity’: Where’s the quality in systematic review? Journal of Education Policy 20 (4): 393–416.
Merschbrock, C. and Munkvold, B.E. (2012). A Research Review on Building Information Modeling in Construction — An Area Ripe for IS Research, Communications of the Association for Information Systems 31: 207–228.
Mettler, T., Eurich, M. and Winter, R. (2014). On the Use of Experiments in Design Science Research: A Proposition of an evaluation framework, Communications of the Association for Information Systems 34 (1): 223–240.
MISQ (2006). Objectives of the MISQ theory and review [www document] http://www.misq.org/skin/frontend/default/misq/pdf/TheoryReview/TRObjectives.pdf (accessed 10 May 2014).
Mohan, K. and Ahlemann, F. (2011). Understanding Acceptance of Information System Development and Management Methodologies by actual Users: A Review and Assessment of Existing Literature, in Wirtschaftsinformatik Proceedings 2011, Zürich, Switzerland, Paper 41.
Morrell, K. (2008). The Narrative of ‘Evidence Based’ Management: A polemic, Journal of Management Studies 45 (3): 613–635.
Mulrow, C.D. (1995). Rationale for Systematic Reviews, in I. Chalmers and D.G. Altman (eds.) Systematic Reviews, London: BMJ, pp. 1–8.
Murray, S.J., Holmes, D., Perron, A. and Rail, G. (2007). No Exit? Intellectual Integrity Under the Regime of ‘Evidence’ and ‘Best-practices’, Journal of Clinical Practice 13 (4): 512–516.
Neuman, W.L. (2011). Social Research Methods: Quantitative and qualitative approaches, 7th ed. Boston, MA: Pearson.
Oancea, A. and Pring, R. (2008). The Importance of Being Thorough: On systematic accumulations of ‘what works’ in education research, Journal of Philosophy of Education 42 (4): 15–39.
Oates, B. (2011). Evidence Based Information Systems: A Decade Later, in ECIS 2011 Proceedings, Helsinki, Finnland, Paper 222.
Oates, B.J., Edwards, H.M. and Wainwright, D.W. (2012). A Model-Driven Method for the Systematic Literature Review of Qualitative Empirical Research, in ICIS 2012 Proceedings, Shanghai, China, 1–18.
Okoli, C. and Schabram, K. (2009). Protocol for a Systematic Literature Review of Research on the Wikipedia, Sprouts: Working Papers on Information Systems 9(65).
Okoli, C. and Schabram, K. (2010). A Guide to Conducting a Systematic Literature Review of Information Systems Research, Sprouts: Working Papers on Information Systems 10(26).
Onwuegbuzie, Anthony J., Collins, K.M.T., Leech, N.L., Dellinger, A.B. and Jiao, Q.G. (2007). Mixed Methods+Literature Reviews=Mixed Research Syntheses: A Framework for Conducting and Writing Rigorous, Comprehensive, and Insightful Literature Reviews, in The World of Educational Quality. Annual Meeting of the American Educational Research Association, Chicago.
Oxman, A.D. (1995). Checklists for Review Articles, in I. Chalmers and D.G. Altman (eds.) Systematic Reviews, London: BMJ, pp. 75–85.
Pawson, R. (2006). Digging for Nuggets: How ‘bad’ research can yield ‘good’ evidence, in I. Chalmers and D.G. Altman (eds.) International Journal of Social Research Methodology 9 (2): 127–142.
Perry, C. (1998). A Structured Approach for Presenting Theses, Australasian Marketing Journal 6 (1): 63–85.
Petticrew, M. and Roberts, H. (2006). Systematic Reviews in the Social Sciences: A practical guide, Malden: Blackwell.
Ramsay, C.R., Grant, A.M., Wallace, S.A., Garthwaite, P.H., Monk, A.F. and Russell, I.T. (2000). Assessment of the Learning Curve in Health Technologies – A Systematic Review, International Journal of Technology Assessment in Health Care 16 (4): 1095–1108.
Ridley, D. (2008). The Literature Review: A step-by-step guide for students, Thousand Oaks: Sage.
Rose, S., Bisson, J. and Wessely, S. (2003). A Systematic Review of Single-Session Psychological Interventions (‘Debriefing’) Following Trauma, Psychotherapy and Psychosomatics 72 (4): 176–184.
Roztocki, N. and Weistroffer, H.R. (2008). Event Studies in Information Systems Research: A Review, in AMCIS 2008 Proceedings, Toronto, Canada, Paper 248.
Salton, G. and McGill, M.J. (1983). Introduction to Modern Information Retrieval, New York: McGraw-Hill.
Sandelowski, M. (2008). Reading, Writing and Systematic Review, Journal of Advanced Nursing 64 (1): 104–10.
Sandelowski, M., Voils, C.I. and Barroso, J. (2007). Comparability Work and the Management of Difference in Research Synthesis Studies, Social Science & Medicine (1982) 64 (1): 236–47.
Schultze, U. and Leidner, D.E. (2002). Studying Knowledge Management in Information Systems Research: Discourses and theoretical assumptions, MIS Quarterly 26 (3): 213–242.
Schwarz, A., Mehta, M., Johnson, N. and Chin, W.W. (2007). Understanding Frameworks and Reviews: A commentary to assist us in moving our field forward by analyzing our past, DATA BASE for Advances in Information Systems 38 (3): 29–50.
Shiffman, R.N., Liaw, Y., Brandt, C.A. and Corb, G.J. (1999). Computer-based Guideline Implementation Systems: A systematic review of functionality and effectiveness, Journal of the American Medical Informatics Association 6 (2): 104–114.
Staples, M. and Niazi, M. (2007). Experiences Using Systematic Review Guidelines, Journal of Systems and Software 80 (9): 1425–1437.
Stol, K.-J., Babar, M.A., Russo, B. and Fitzgerald, B. (2009). The Use of Empirical Methods in Open Source Software Research: Facts, Trends and Future Directions, in FLOSS’09, May 18, 2009, Vancouver, Canada, 19–24.
Sutton, R. and Staw, B.M. (1995). Forum What Theory is Not, Administrative Science Quarterly 40 (3): 371–384.
Tamm, T., Seddon, P.B., Shanks, G. and Reynolds, P. (2011). How Does Enterprise Architecture Add Value to Organisations?, Communications of the Association for Information Systems 28: 141–168.
Thompson, S.G. (1995). Why Sources of Heterogenity in Meta-Analysis Should be Investigated, in I. Chalmers and D.G. Altman (eds.) Systematic Reviews, London: BMJ, pp. 48–63.
Tranfield, D., Denyer, D. and Smart, P. (2003). Towards a Methodology for Developing Evidence-Informed Management Knowledge by Means of Systematic Review, British Journal of Management 14 (3): 207–222.
Wang, Q.E., Myers, M.D. and Sundaram, D. (2013). Digital Natives and Digital Immigrants: Towards a model of digital fluency, Business and Information Systems Engineering 5 (6): 409–419.
Webster, J. and Watson, R.T. (2002). Analyzing the Past to Prepare for the Future: Writing a literature review, MIS Quarterly 26 (2): xiii–xxiii.
Weerakkody, V., Dwivedi, Y.K. and Irani, Z. (2009). The Diffusion and Use of Institutional Theory: A cross-disciplinary longitudinal literature survey, Journal of Information Technology 24 (4): 354–368.
Whetten, D.A. (1989). What Constitutes a Theoretical Contribution?, Academy of Management Review 14 (4): 490–495.
Williams, M.D., Dwivedi, Y.K., Lal, B. and Schwarz, A. (2009). Contemporary Trends and Issues in IT Adoption and Diffusion Research, Journal of Information Technology 24 (1): 1–10.
Wolfswinkel, J.F., Furtmueller, E. and Wilderom, C.P.M. (2013). Using Grounded Theory as a Method for Rigorously Reviewing Literature, European Journal of Information Systems 22 (1): 45–55.
Wolfswinkel, J., Furtmueller, E. and Wilderom, C. (2010). Reflecting on E-Recruiting Research Using Grounded Theory, in ECIS 2010 Proceedings, Pretoria, South Africa, Paper 52.
Wright Mills, C. (1978 ). The Sociological Imagination, Oxford: Oxford University Press.
Zhang, P., Li, N., Scialdone, M.J. and Carey, J. (2008). The Intellectual Advancement of Human-Computer Interaction Research: A critical assessment of the MIS literature (1990–2008), AIS Transactions on Human-Computer Interaction 1 (3): 55–107.
About this article
Cite this article
Boell, S., Cecez-Kecmanovic, D. On being ‘systematic’ in literature reviews in IS. J Inf Technol 30, 161–173 (2015). https://doi.org/10.1057/jit.2014.26
- systematic literature review (SLR)
- systematic review
- conducting literature reviews
- literature review
- narrative literature review
- database searches
- literature search