Abstract
The range of legitimate methods in IS research has expanded considerably over the past 20 years, a process to which IFIP Working Group 8.2 is seen to have made an important contribution. This has probably made it even harder, however, for IS researchers to know what constitutes good methodological practice. This paper addresses this issue from two angles: first through a critical analysis of claims made in the IS literature regarding the characteristics of good research; and second through an examination of the use of methodology, as reported in a number of IS research papers. The characteristics of good research considered are that it should follow the scientific method; that it should fulfil certain criteria; that it should be relevant; and that it should employ multiple methods Each of these is shown to have limitations. With respect to methodology in practice, the analysis indicates a remarkable lack of consistency in the reporting of IS research. The implications of these findings are discussed.
Chapter PDF
Similar content being viewed by others
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
Agar, M. H. The Professional Stranger: An Informal Introduction to Ethnography, New York: Academic Press, 1980.
Allen, D., and Ellis, D. “Beyond Paradigm Closure in Information Systems Research: Theoretical Possibilities for Pluralism,” in R. Galliers, C. Murphy, H. Hansen, R. O’Callaghan, S. Carlsson, and C. Loebbecke (Eds.), Proceedings of the 5 th European Conference on Information Systems, Cork, Ireland: Cork Publishing Company, 1997, pp. 760–776.
Aanestad, M., and Hanseth, O. “Implementing Open Network Technologies in Complex Work Practices: A Case from Telemedicine,” in R. Baskerville, J. Stage, and J. I. DeGross (Eds.), Organizational and Social Perspectives on Information Technology, Boston: Kluwer Academic Publishers, 2000.
Applegate, L. M., and King, J. L. “Rigor vs. Relevance: Careers on the Line,” MIS Quarterly (23:1), 1999, pp. 17–18
Benbasat, I.; Goldstein, D. K.; and Mead, M. “The Case Study Research Strategy in Studies of Information Systems,” MIS Quarterly (11:4), 1987, pp. 369–386.
Benbasat, I., and Zmud, R. W. “Empirical Research in Information Systems: The Practice of Relevance,” MIS Quarterly (23:1), 1999, pp. 3–16.
Burrell, G., and Morgan, G. Sociological Paradigms and Organizational Analysis, Portsmouth, NJ: Heinemann, 1979.
Broadbent, M.; Weill, P.; O’Brien, T.; and Neo, B. S. “Firm Context and Patterns of IT Infrastructure Capability,” in J. I. DeGross, S. Jarvenpaa, and A. Srinivasan (Eds.), Proceedings of the 17th International Conference on Information Systems, Cleveland, OH, 1996, pp. 174–194.
Calloway, L. J., and Ariav, G. “Developing and Using a Qualitative Methodology to Study Relationships Among Designers and Tools,” in H-E. Nissen, H. K. Klein, and R. Hirschheim (Eds.), Information Systems Research: Contemporary Approaches and Emergent Traditions, Amsterdam: Elsevier, 1990.
Ciborra, C.; Patriotta, G.; and Erlicher, L. “Disassembling Frames on the Assembly Line: The Theory and Practice of the New Division of Learning in Advanced Manufacturing,” in W. J. Orlikowski, G. Walsham, M. R. Jones, and J. I. DeGross (Eds.), Information Technology and Changes in Organizational Work, London: Chapman & Hall, 1996.
Collins, H. Changing Order: Replication and Induction in Scientific Practice, London: Sage Publications, 1985.
Cooper, D. R., and Schindler, P. S. Business Research Methods, London: McGraw-Hill, 1998.
Davenport, T. H., and Markus, M. L. “Rigor vs. Relevance Revisited: Response to Benbasat and Zmud,” MS Quarterly (23:1), 1999, pp. 19–23
Davies, L. J. “Researching the Organizational Culture Contexts of Information Systems Strategy: A Case Study of the British Army,” in H-E. Nissen, H. K. Klein, and R. Hirschheim (Eds.), Information Systems Research: Contemporary Approaches and Emergent Traditions, Amsterdam: Elsevier, 1990.
DeSanctis, G. “Theory and Research: Goals, Priorities and Approaches,” MIS Quarterly (17:1), 1993, pp. vi–viii.
Elam, J.; Walz; D. R.; Curtis, B.; and Krasner, H. “Measuring Group Processes in Software Design Teams,” in H-E. Nissen, H. K. Klein, and R. Hirschheim (Eds.), Information Systems Research: Contemporary Approaches and Emergent Traditions, Amsterdam: Elsevier, 1990.
Emery, J. C. “Editor’s Comments,” MIS Quarterly (13:3), 1989, pp. xi–xii
Feyerabend, P. Against Method, London: Verso, 1975.
Fitzgerald, G.; Hirschheim, R.; Mumford, E.; and Wood-Harper, A. T. “Information Systems Research Methodology: An Introduction to the Debate,” in E. Mumford, R. Hirschheim, G. Fitzgerald, and A. T. Wood-Harper (Eds.), Research Methods in Information Systems, Amsterdam: North-Holland, 1985, pp. 3–9.
Gable, G. G. “Integrating Case Study and Survey Research Methods: An Example in Information Systems,” European Journal of Information Systems (3:2), 1994, pp. 112–126.
Gallivan, M. J. “Contradictions Among Stakeholder Assessments of a Radical Change Initiative: A Cognitive Frames Analysis,” in W. J. Orlikowski, G. Walsham, M. R. Jones, and J. I. DeGross (Eds.), Information Technology and Changes in Organizational Work, London: Chapman & Hall, 1996.
Gasson, S., and Holland, N. “The Nature and Processes of IT-Related Change,” in W. J. Orlikowski, G. Walsham, M. R. Jones, and J. I. DeGross (Eds.), Information Technology and Changes in Organizational Work, London: Chapman & Hall, 1996.
Geertz, C. Works and Lives: The Anthropologist as Author, Cambridge: Polity Press, 1988.
Giddens, A. New Rules of Sociological Method, London: Hutchinson, 1976.
Giddens, A. New Rules of Sociological Method (2nd ed.), Cambridge: Polity Press, 1993.
Glaser, B., and Strauss, A. The Discovery of Grounded Theory, London: Weidenfeld and Nicholson, 1968.
Golden-Biddle, K., and Locke, K. D. Composing Qualitative Research, London: Sage Publications, 1997.
Grunden, K. “MOA-S: A Scenario Model for Integrating Work Organization Aspects into the Design Process of CSCW Systems,” in R. Baskerville, J. Stage, and J. I. DeGross (Eds.), Organizational and Social Perspectives on Information Technology, Boston: Kluwer Academic Publishers, 2000.
Guimaraes, T. “Assessing Employee Turnover Intentions Before/After BPR,” in W. J. Orlikowski, G. Walsham, M. R. Jones, and J. I. DeGross (Eds.), Information Technology and Changes in Organizational Work, London: Chapman & Hall, 1996.
Hamilton, D., and Atchison, M. “The COMIS Plan: IT-Mediated Business Reengineering in Telecom Australia During the 1960s,” in W. J. Orlikowski, G. Walsham, M. R. Jones, and J. I. DeGross (Eds.), Information Technology and Changes in Organizational Work, London: Chapman & Hall, 1996.
Harari, O. “Ten Reasons Why TQM Doesn’t Work,” Management Review (86:2), 1997, pp. 38–44.
Hess, C. M., and Kemerer, C. F. “Computerized Loan Origination Systems: An Industry Case Study of the Electronic Markets Hypothesis,” MIS Quarterly (18:3), September 1994, pp. 251–275.
Hitt, L. M., and Brynjolfsson, E. “Productivity, Business Profitability, and Consumer Surplus: Three Different Measures of Information Technology Value,” MIS Quarterly (20:2), 1996, pp. 121–142.
Hitt, L., and Brynjolfsson, E. “The Three Faces of IT Value: Theory and Evidence,” in J. I. DeGross, S. L. Huff, and M. C. Munro (Eds.), Proceedings of the 15th International Conference on Information Systems, Vancouver, British Columbia, Canada, 1994, pp. 263–277.
Jasperson, J.; Carte, T. A.; Saunders, C. S.; Butler, B. S.; Croes, H. J. P.; and Zheng, W. “Review: Power and Information Technology Research: A Metatriangulation Review,” MIS Quarterly (26:4), 2002, pp. 397–459.
Jones, M. R. “Mission Impossible? Pluralism and Multiparadigm IS Research,” Information Systems Review (1:1), 2000a, pp. 217–232.
Jones, M. R. “The Moving Finger: The Use of Social Theory in WG8.2 Conference Papers, 1975–1999,” in R. Baskerville, J. Stage, and J. I. DeGross (Eds), Organizational and Social Perspectives on Information Technology, Boston: Kluwer Academic Publishers, 2000b, pp. 15–31.
Kaplan, B., and Duchon, D. “Combining Qualitative and Quantitative Methods in Information Systems Research: A Case Study,” MIS Quarterly (12:4), 1988, pp. 571–586.
Kauffman, R. J., and Wood, C. A. “Follow the Leader? Strategic Pricing of E-Commerce,” in W. J. Orlikowski, S. Ang, P. Weill, H. C. Krcmar, and J. I. DeGross (Eds.), Proceedings of the 21th International Conference on Information Systems, Brisbane, 2000, pp. 145–151.
Keen, P. G. W. “Relevance and Rigor in Information Systems Research: Improving Quality, Confidence, Citation and Impact,” in H-E. Nissen, H. K. Klein, and R. Hirschheim (Eds.), Information Systems Research: Contemporary Approaches and Emergent Traditions, Amsterdam: Elsevier, 1990, pp. 27–49.
King, G.; Keohane, R. O.; and Verba, S. Designing Social Inquiry, Princeton: Princeton University Press, 1994.
Klein, H. K., and Myers, M. D. “A Set of Principles for Conducting and Evaluating Interpretive Field Studies in Information Systems,” MIS Quarterly (23:1), 1999, pp. 67–94.
Kumar, K.; van Dissel, H. G.; and Bielli, P. “The Merchant of Prato-Revisited: Toward a Third Rationality of Information Systems,” MIS Quarterly (22:2), 1998, pp. 199–226.
Landry, M., and Banville, C. “A Disciplined Methodological Pluralism for MIS Research,” Accounting, Management and Information Technology (2:2), 1992, pp. 77–97.
Latour, B. Science in Action, Milton Keynes, UK: Open University Press, 1987.
Latour, B., and Woolgar, S. Laboratory Life: The Social Construction of Scientific Facts, London: Sage Publications, 1979.
Lee, A. S. “Electronic Mail as a Medium for Rich Communication: An Empirical Investigation Using Hermeneutic Interpretation,” in J. I. DeGross, R. P. Bostrom, and D. Robey (Eds.), Proceedings of the 14th International Conference on Information Systems, Tampa, FL, 1993, pp. 13–21.
Lee, A. S. “Integrating Positivist and Interpretive Approaches to Organizational Research,” Organization Science (2:4), 1991, pp. 342–365.
Lee, A. S. “Rigor and Relevance in IS Research: Beyond the Approach of Positivism Alone,” MIS Quarterly (23:1), 1999, pp. 29–34.
Lee, A. S. “A Scientific Methodology for MIS Case Studies,” MIS Quarterly (13:1), 1989, pp. 32–51.
Lee, A. S.; Baskerville, R. L.; Liebenau, J; and Myers, M. D. “Judging Qualitative Research in Information Systems: Criteria for Accepting and Rejecting Manuscripts,” in J. I. DeGross, G. Ariav, C. Beath, R. Hoyer, and C. Kemerer, Proceedings of the 16th International Conference on Information Systems, Amsterdam, 1995, p. 367.
Lee, S.; Goldstein, D. K.; and Guinan, P. J. “Informant Bias in Information Systems Design Team Research,” in H-E. Nissen, H. K. Klein, and R. Hirschheim (Eds.), Information Systems Research: Contemporary Approaches and Emergent Traditions, Amsterdam: Elsevier, 1990.
Leidner, D. E., and Jarvenpaa, S. L. “The Use of Information Technology to Enhance Management School Education: A Theoretical View,” MIS Quarterly (19:3), 1995, pp. 265–291.
Majchrzak, A.; Rice, R. E.; Malhotra, A.; King, N.; and Ba, S. “Technology Adaptation: The Case of a Computer-Supported Inter-Organizational Virtual Team,” MIS Quarterly (24:4), 2000, pp. 569–600.
Mark, G. “Some Challenges Facing Virtually Collocated Teams,” in R. Baskerville, J. Stage, and J. I. DeGross (Eds.), Organizational and Social Perspectives on Information Technology, Boston: Kluwer Academic Publishers, 2000.
Markus, M. L. “Electronic Mail as the Medium of Managerial Choice,” Organization Science (5:4), 1994, pp. 502–527.
Mingers, J. “Combining IS Research Methods: Towards a Pluralist Methodology,” Information Systems Research (12:3), 2001, pp. 240–259.
Mingers, J. “The Paucity of Multimethod Research: A Review of the Information Systems Literature,” Information Systems Journal (13), 2003, pp. 233–249.
Mukhopadhyay, T.; Kekre, S.; and Kalathur, S. “Business Value of Information Technology: A Study of Electronic Data Interchange,” MIS Quarterly (19:2), 1995, pp. 137–156.
Nandhakumar, J., and Jones, M. R. “Designing in the Dark: The Changing User-Developer Relationship in Information Systems Development,” in K. Kumar and J. I. DeGross (Eds.), Proceedings of the 18th International Conference on Information Systems, Atlanta, GA, 1997, pp. 75–87.
Nandhakumar, J., and Jones, M. R. “Too Close for Comfort? Distance and Engagement in Interpretive Information Systems Research,” Information Systems Journal (7), 1997, pp. 109–131.
Nelson, Kay M. and Cooprider, Jay G. “The Relationship of Software System Flexibility to Software System and Team Performance,” in J. I. DeGross, R. P. Bostrom, and D. Robey (Eds.), Proceedings of the 14th International Conference on Information Systems, Tampa, FL, 1993, pp. 23–32.
Ngwenyama, O. K., and Lee, A. S. “Communication Richness in Electronic Mail: Critical Social Theory and the Contextuality of Meaning,” MIS Quarterly (21:2), 1997, pp. 145–167.
Orlikowski, W. “CASE Tools as Organizational Change: Investigating Incremental and Radical Changes in Systems Development,” MIS Quarterly (17:3), 1993, pp. 309–340.
Orlikowski, W. J., and Baroudi, J. J. “Studying Information Technology in Organizations: Research Approaches and Assumptions,” Information Systems Research (2:1), 1991, pp. 1–28.
Parker, M., and McHugh, G. “Five Texts in Search of an Author: A Response to John Hassard’s ‘Multiple Paradigms and Organizational Analysis’,” Organization Studies (12:3), 1991, pp. (3), 451–456.
Pickering, A. Science as Practice and Culture, London: University of Chicago Press, 1992.
Remenyi, D., and Williams, B. “Some Aspects of Methodology for Research in Information Systems,” Journal of Information Technology (10), 1995, pp. 191–201.
Remenyi, D.; Williams, B.; Money, A.; and Swartz, E. Doing Research in Business and Management, London: Sage Publications, 1998.
Robey, D. “Theories that Explain Contradiction: Accounting for Contradictory Organizational Consequences of Information Technology,” in J. I. DeGross, G. Ariav, C. Beath, R. Hoyer, and C. Kemerer, Proceedings of the 16th International Conference on Information Systems, Amsterdam, 1995, pp. 55–63.
Sawyer, S. “Analysis by Long Walk: Some Approaches to the Synthesis of Multiple Sources of Evidence,” in E. Trauth (Ed.), Qualitative Research in IS: Issues and Trends, Hershey, PA: Idea Group Publishing, 2001.
Sawyer, S. “Conceptualizing Information Technology in the Study of Information Systems: Trends and Issues,” in E. Wynn, E. A. Whitley, M. Myers, and J. I. DeGross (Eds.), Global and Organizational Discourse About Information Technology, Boston: Kluwer Academic Publishers, 2002, pp. 109–132.
Sawyer, S. “Studying Organizational Computing Infrastructures: Multi-Method Approaches,” in R. Baskerville, J. Stage, and J. I. DeGross (Eds.), Organizational and Social Perspectives on Information Technology, Boston: Kluwer Academic Publishers, 2000, pp. 213–222.
Schultze, U. “A Confessional Account of an Ethnography About Knowledge Work,” MIS Quarterly (24:1), 2000, pp. 3–41.
Seale, C. The Quality of Qualitative Research, London: Sage Publications, 1999.
Sekaran, U. Research Methods for Business, New York: Wiley, 1992.
Smithson, S. “Combining Different Approaches,” in H-E. Nissen, H. K. Klein, and R. Hirschheim (Eds.), Information Systems Research: Contemporary Approaches and Emergent Traditions, Amsterdam: Elsevier, 1990, pp. 365–369.
Strauss, A., and Corbin, J. Basics of Qualitative Research, London: Sage Publications, 1990.
Subramani, M., and Walden, E. “The Dot Com Effect: The Impact of E-Commerce Announcements on the Market Value of Firms,” in P. De and J. I. DeGross (Eds.), Proceedings of the 20th International Conference on Information Systems, Charlotte, NC, 1999, pp. 193–207.
Tashakkori, A., and Teddlie, C. Mixed Methodology: Combining Qualitative and Quantitative Approaches, London: Sage Publications, 1998.
Taylor, S., and Bogdan, R. Introduction to Qualitative Research Methods, Chichester, UK: Wiley, 1998.
Te’eni, D. “Review: A Cognitive-Affective Model of Organizational Communication for Designing IT,” MIS Quarterly (25:2), 2001, pp. 251–212.
Toraskar, K. “How Managerial Users Evaluate Their Decision Support: A Grounded Theory Approach,” in H-E. Nissen, H. K. Klein, and R. Hirschheim (Eds.), Information Systems Research: Contemporary Approaches and Emergent Traditions, Amsterdam: Elsevier, 1990.
Trauth, E. M., and Jessup, L. M. “Understanding Computer-Mediated Discussions: Positivist and Interpretive Analysis of Group Support System Use,” MIS Quarterly (24:1), 2000, pp. 43–79.
Turner, J. A.; Bikson, T. K.; Lyytinen, K.; Mathiassen, L.; and Orlikowski, W. “Relevance Versus Rigor in Information Systems Research: An Issue of Quality,” in H-E. Nissen, H. K. Klein, and R. Hirschheim (Eds.), Information Systems Research: Contemporary Approaches and Emergent Traditions, Amsterdam: Elsevier, 1990, pp. 715–745.
Van Maanen, J. “Some Notes on the Importance of Writing in Organization Studies,” in J. I. Cash and P. R. Lawrence (Eds.), The Information Systems Research Challenge: Volume 1, Boston: Harvard Business School Press, 1989, pp. 27–33.
Vessey, I.; Ramesh, V.; and Glass, R. L. “Research in Information Systems: An Empirical Study of Diversiry in the Discipline and its Journals,” Journal of Management Information Systems (19:2), 2002, pp. 129–173.
Vidgen, R., and McMaster, T. “Black Boxes, Non-Human Stakeholders and the Translation of IT through Mediation,” in W. J. Orlikowski, G. Walsham, M. R. Jones, and J. I. DeGross (Eds.), Information Technology and Changes in Organizational Work, London: Chapman & Hall, 1996.
Watson, W. “Types of Pluralism,” The Monist (73:3), 1990, pp. 350–365.
Wilson, M., and Howcroft, D. “The Role of Gender in User Resistance and Information Systems Failure,” in R. Baskerville, J. Stage, and J. I. DeGross (Eds.), Organizational and Social Perspectives on Information Technology, Boston: Kluwer Academic Publishers, 2000.
Yap, A. Y., and Bjørn-Andersen, N. “Energizing the Nexus of Corporate Knowledge: A Portal Toward the Virtual Organization,” in R. Hirschheim, M. Newman, and J. I. DeGross (Eds.), Proceedings of the 19th International Conference on Information Systems, Helsinki, Finland, 1998, pp. 273–286.
Yin, R. K. Applications of Case Study Research, London: Sage Publications, 1993.
Zmud, R. W. “Editor’s Comments,” MIS Quarterly (23:3), 1996, pp. xxxvii–xxxix.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2004 Springer Science + Business Media, Inc.
About this chapter
Cite this chapter
Jones, M.R. (2004). Debatable Advice and Inconsistent Evidence: Methodology in Information Systems Research. In: Kaplan, B., Truex, D.P., Wastell, D., Wood-Harper, A.T., DeGross, J.I. (eds) Information Systems Research. IFIP International Federation for Information Processing, vol 143. Springer, Boston, MA. https://doi.org/10.1007/1-4020-8095-6_8
Download citation
DOI: https://doi.org/10.1007/1-4020-8095-6_8
Publisher Name: Springer, Boston, MA
Print ISBN: 978-1-4020-8094-4
Online ISBN: 978-1-4020-8095-1
eBook Packages: Springer Book Archive