European Journal of Information Systems

, Volume 14, Issue 3, pp 229–243 | Cite as

A critical approach to evaluation



Within information systems, the question of evaluation remains as a major contentious issue, and perhaps even more so in the field of health informatics where the traditions of medicine meet and mingle with the information systems field. A review of the literature in these areas indicates that there is little agreement on the essential role of evaluation, a ‘best way’ to evaluate, on what and how to evaluate, whom to involve and within what paradigm to proceed. Reflecting on discussion within both traditions, this paper develops an approach to evaluation broadly based on critical theory, and argues that such an approach, while not offering a solution to all the problems of evaluation, does bring into focus fundamental questions relating to evaluation process and content. To illustrate this, the paper reflects on the strengths and weaknesses of an evaluation of a primary care intranet, and suggests how a critical approach might lead to a more meaningful evaluation and provide more significant and useful findings.


Evaluation health information systems critical theory 


  1. Abdulrasul S (2001) An evaluation of the use of the NHSnet in GP surgeries in terms of user satisfaction. Department of Information Systems and Computing. Brunel University, London.Google Scholar
  2. Alvesson M and Willmott H (Eds) (1992) Critical Management Studies. Sage Publications, London.Google Scholar
  3. Ammenwerth E, Graber S, Herrmann G, Burkle T and Konig J (2003) Evaluation of health information systems – problems and challenges. International Journal of Medical Informatics 71, 125–135.CrossRefGoogle Scholar
  4. Anderson JD and Aydin CE (1994) Overview: Theoretical perspectives and methodologies for the evaluation of health care information systems. In Evaluaing Health Care Information Systems (ANDERSON JD, AYDIN CE and JAY JS, Eds), pp 5–29, Sage Publications, Thousands Oaks, CA.Google Scholar
  5. Avgerou C (1995) Evaluating information systems by consultation and negotiation. International Journal of Information Management 15(6), 427–436.CrossRefGoogle Scholar
  6. Bannister F (1999) What did we pay for it? The awkward problem of IT costs. In Sixth European Conference on Information Technology Evaluation (Brown A and Remenyi D, Eds), pp 55–64, Uxbridge, London.Google Scholar
  7. Banta D (2001) The management of health technology from a national perspective. International Journal of Healthcare Technology and Management 3(2/3/4), 177–190.CrossRefGoogle Scholar
  8. Berg M (1999) Patient care information systems and health care work: a sociotechnical approach. International Journal of Medical Informatics 55, 87–101.CrossRefGoogle Scholar
  9. Bijker WE (1997) Of Bicycles, Bakelites, and Bulbs: Toward a Theory of Sociotechnical Change. MIT Press, Cambridge, MA.Google Scholar
  10. Bloomfield BP and Best A (1992) Management consultants: systems development, power and the translation of problems. Sociological Review 40, 533–560.CrossRefGoogle Scholar
  11. Bowling A (1997) Research Methods in Health: Investigating Health and Health Services. Open University Press, Buckingham.Google Scholar
  12. Bryans P, Williamson B, Stalker B and Gormley N (1999) Evaluation as learning and dialogue: towards an open model of evaluation. In Critical Management Studies Conference. Manchester.Google Scholar
  13. Burrell G and Morgan G (1985) Sociological Paradigms and Organizational Analysis: Elements of the Sociology of Corporate Life. Gower, Aldershot.Google Scholar
  14. Cornford T, Doukidis GI and Forster D (1994) Experience with structure, process and outcome framework for evaluating an information system. Omega International Journal of Management Science 22(5), 491–504.CrossRefGoogle Scholar
  15. Cornford T and Klecun-Dabrowska E (2001) Ethical perspectives in evaluation of telehealth. Cambridge Quarterly of Healthcare Ethics (Special Issue on New Information Technologies in Healthcare) 10(2), 161–169.CrossRefGoogle Scholar
  16. Cribb A and Barber N (1997) Prescribers, patients and policy: towards the limits of technique. Health Care Analysis 5 (4), 292–298.CrossRefGoogle Scholar
  17. Dabrowska E (1999) The evaluation of the deployment of Intranet facilities within primary care. In Second Oxford Clinical Intranet and Internet Conference. Oxford.Google Scholar
  18. Davis GB, Lee AS, Nickles KR, Chaterjee S, Hartung R and Wu Y (1992) Diagnosis of an information system failure: a framework and interpretive process. Information Management 23(5), 293–318.CrossRefGoogle Scholar
  19. DeLone WH and McLean ER (1992) Information systems success: the quest for the dependent variable. Information Systems Research 3(1), 60–95.CrossRefGoogle Scholar
  20. Denzin NK and Lincoln YS (Eds) (1994) The Handbook of Qualitative Research. Sage Publications, London.Google Scholar
  21. DOH (1997) The New NHS: Modern.Dependable. HMSO, London.Google Scholar
  22. DOH (2002) Delivering the 21st Century IT Support for the NHS: National Strategic Programme. HMSO, London.Google Scholar
  23. Donabedian A (1966) Evaluating the quality of medical care. Milbank Medical Fund Quarterly 44, 166.CrossRefGoogle Scholar
  24. Donabedian A (1978) The quality of medical care – methods for assessing and monitoring the quality of care for research and for quality assurance programs. Science 200, 856–864.CrossRefGoogle Scholar
  25. Doolin B (1998) Information technology as disciplinary technology: being critical in interpretive research on information systems. Journal of Information Technology 13, 301–311.CrossRefGoogle Scholar
  26. Farbey B, Land F and Targett D (1999) The moving staircase: problems of appraisal and evaluation in a turbulent environment. Information Technology & People 12(3), 238–252.CrossRefGoogle Scholar
  27. Farbey B, Land F and Targett D (1995) A taxonomy of information systems applications: the benefits' evaluation ladder. European Journal of Information Systems 4, 41–50.CrossRefGoogle Scholar
  28. Farbey B, Targett D and Land F (1993) How to Evaluate IT Investment: A Study of Methods and Practices. Butterworth Heinemann, Oxford.Google Scholar
  29. Feenberg A (1991) Critical Theory of Technology. Oxford University Press, New York.Google Scholar
  30. Field MJ (Ed) (1996) Telemedicine: A Guide to Assessing Telecommunications in Health Care Institute of Medicine National Academy Press, Washington, D.C.Google Scholar
  31. Friedman CP and Wyatt JC (1996) Evaluation Methods in Medical Informatics. Springer-Verlag, New York.Google Scholar
  32. Gadamer H-G (1976) Philosophical Hermeneutics. University of California Press, Berkeley.Google Scholar
  33. Ginzberg MJ (1978) Finding an adequate measure of OR/MS effectiveness. Interfaces 8(4), 59–62.CrossRefGoogle Scholar
  34. Habermas J (1979) Communication and the Evolution of Society. Beacon Press, Boston.Google Scholar
  35. Habermas J (1987) The Theory of Communicative Action – The Critique of Functionalist Reason. Beacon Press, Boston, MA.Google Scholar
  36. Heathfield H, Hudson P, Kay S, Mackay L, Marley T, Nicholson L, Peel V, Roberts R and Williams J (1999) Issues in the multi-disciplinary assessment of healthcare information systems. Information Technology & People 12(3), 253–275.CrossRefGoogle Scholar
  37. Heathfield H, Pitty D and Hanka R (1998) Evaluating information technology in health care: barriers and challenges. British Medical Journal 316, 1959–1961.CrossRefGoogle Scholar
  38. Hillier S (1987) Rationalism, bureaucracy, and the organization of the health services: Max Weber's contribution to understanding modern health care systems. In Sociological Theory and Medical Sociology (Scambler G, Ed), pp 194–220, Tavistock, London.Google Scholar
  39. Horkheimer M (1972) Traditional and Critical Theory. In Critical Theory: Selected Essays (Matthew J. O'Connell et al. Transls), pp 188–243, Herder and Herder, New York.Google Scholar
  40. Horkheimer M and Adorno TW (1972 [1944]) Dialectic of Enlightenment. Herder, New York.Google Scholar
  41. Howcroft D and Mitev N (2000) An empirical study of internet usage and difficulties amongst medical practice management in the UK. Internet Research 10(2), 170–181.CrossRefGoogle Scholar
  42. IMG (1996) Project review: objective evaluation – guidance for NHS managers on evaluating information systems projects. Information Management Group, NHS Executive Volume 2000, host accessed 22/05/2000. host updated 13/07/1998.
  43. Introna LD (1997) Management, Information and Power: A Narrative of the Involved Manager. Macmillan, Basingstoke.CrossRefGoogle Scholar
  44. Introna LD and Whittaker L (2002) The phenomenology of information systems evaluation: Overcoming the subject/object dualism. In Global and Organizational Discourse about Information Technology (WYNN EH, WHITLEY EA, MYERS MD and DEGROSS JI, Eds), pp 155–175, Kluwer Academic Publishers, Boston.Google Scholar
  45. Ives B, Olson MH and Baroudi JJ (1983) The measurement of user information satisfaction. Communication of the ACM 26(10), 785–793.CrossRefGoogle Scholar
  46. Jones S and Hughes J (2001) Understanding IS evaluation as a complex social process: a case study of a UK local authority. European Journal of Information Systems 10(4), 189–203.CrossRefGoogle Scholar
  47. Kaplan B (1997) Addressing the organisational issues into the evaluation of medical systems. Journal of the American Medical Informatics Association 4(2), 94–101.CrossRefGoogle Scholar
  48. Kaplan B (1998) Social interactionist framework for information systems studies: the 4Cs. In IFIP WG8.2 & WG8.6 Joint Working Conference on Information Systems: Current Issues and Future Changes, pp 327–340, Helsinki.Google Scholar
  49. Kaplan B (2001a) Evaluating informatics applications – clinical decision support systems literature review. International Journal of Medical Informatics 64, 15–37.CrossRefGoogle Scholar
  50. Kaplan B (2001b) Evaluating informatics applications – some alternative approaches: theory, social interactionism, and call for methodological pluralism. International Journal of Medical Informatics 64, 39–56.CrossRefGoogle Scholar
  51. Kaplan B and Shaw NT (2004) Future directions in evaluation research: people, organizational, and social issues. Methods of Information in Medicine 43(3), 215–231.Google Scholar
  52. Kaplan R and Norton D (1992) The balanced score card: measures that drive performance. Harvard Business Review 70(1), 71–79.Google Scholar
  53. Klecun E and Cornford T (2003) An interpretative evaluation of a health care intranet. International Journal of Healthcare Technology and Management 5(6), 407–421.CrossRefGoogle Scholar
  54. Klecun-Dabrowska E and Cornford T (2000) Telehealth acquires meanings: information and communication technologies within health policy. Information Systems Journal 10(1), 41–63.CrossRefGoogle Scholar
  55. Kling R (1996) Beyond outlaws, hackers, and pirates: ethical issues in the work of information and computer science professionals. In Computerization and Controversy: Value Conflicts and Social Choices (Kling R, Ed), Academic Press, San Diego.Google Scholar
  56. Land F (2000) Evaluation in a socio-technical context. In Organizational and Social Perspectives on Information Technology (BAKERVILLE R, STAGE J and DEGROSS JI, Eds), pp 115–126, Kluwer Academic Publishers, Boston.CrossRefGoogle Scholar
  57. Lau F, Doze S, Vincent D, Wilson D, Noseworthy T, Hayward R and Penn A (1999) Patterns of improvisation for evidence-based practices in clinical settings. Information, Technology & People 12(3), 287–303.CrossRefGoogle Scholar
  58. Littlejohns P, Wyatt J and Garvican L (2003) Evaluating computerised health information systems: hard lessons still to be learned. British Medical Journal 326, 860–863.CrossRefGoogle Scholar
  59. Lobley D (1997) The economics of telemedicine. Journal of Telemedicine and Telecare 3, 117–125.CrossRefGoogle Scholar
  60. Mackenzie D and Wajcman J (Eds) (1985) The Social Shaping of Technology: How the Refrigerator Got Its Hum. Open University Press, Milton Keynes.Google Scholar
  61. Marcuse H (1970) One-Dimensional Man. Sphere Books Ltd, London.Google Scholar
  62. May C and Ellis NT (2001) When protocols fail: technical evaluation, biomedical knowledge, and a social production of ‘facts’ about a telemedicine clinic. Social Science & Medicine 53, 989–1002.CrossRefGoogle Scholar
  63. May C, Mort M, Williams T, Mair F and Gask L (2003) Health technology assessment in its local contexts: studies of telehealthcare. Social Science & Medicine 57, 697–710.CrossRefGoogle Scholar
  64. Mcaulay L, Doherty N and Keval N (2002) The stakeholder dimension in information systems evaluation. Journal of Information Technology 17(4), 241–255.CrossRefGoogle Scholar
  65. Miller J (1993) Measuring and aligning information systems with the organization. Information and Management 25(4), 217–228.CrossRefGoogle Scholar
  66. Mingers J (2000) The contribution of critical realism as an underpinning philosophy for OR/MS and systems. Journal of Operational Research 51(11), 1256–1270.CrossRefGoogle Scholar
  67. Morrow RD and Brown DD (1994) Critical Theory and Methodology. Sage Publications, London.Google Scholar
  68. Mumford E (1983) Designing Human Systems for New Technology: The ETHICS Method. Manchester Business School, Manchester.Google Scholar
  69. O'Donnell D and Henriksen LB (2002) Philosophical foundations for a critical evaluation of the social impact of ICT. Journal of Information Technology 17, 89–99.CrossRefGoogle Scholar
  70. O'Keefe RM (1989) The evaluation of decision-aiding systems: guidelines and methods. Information and Management 17(4), 217–226.CrossRefGoogle Scholar
  71. Orlikowski WJ (2000) Using technology and constituting structures: a practice lens for studying technology in organizations. Organization Science 11(4), 404–428.CrossRefGoogle Scholar
  72. Orlikowski WJ and Hofman JD (1997) An improvisational model for change management: the case of groupware technologies. Sloan Management Review 38(2), 11–21.Google Scholar
  73. Powell P (1992) Information technology evaluation: is it different? Journal of Operational Research Society 43(1), 29–42.CrossRefGoogle Scholar
  74. Remenyi DSJ and Money A (1991) A user-satisfaction approach to IS effectiveness measurement. Journal of Information Technology 6, 162–175.CrossRefGoogle Scholar
  75. Smithson S and Hirschheim RA (1998) Analysing information systems evaluation: another look at an old problem. European Journal of Information Systems 7(3), 155–174.CrossRefGoogle Scholar
  76. Southon G (1999) IT, change and evaluation: an overview of the role of evaluation in health services. International Journal of Medical Informatics 56, 125–133.CrossRefGoogle Scholar
  77. Srinivasan A (1985) Alternative measures of system effectiveness: associations and implications. MIS Quarterly September, pp 243–253.Google Scholar
  78. Stoop AP and Berg M (2003) Integrating quantitative and qualitative methods in patient care information system evaluation. Methods of Information in Medicine 42(4), 458–462.Google Scholar
  79. Symons VJ (1991) A review of information systems evaluation: content, context and process. European Journal of Information Systems 1(3), 205–212.CrossRefGoogle Scholar
  80. Talmon J, Enning J, Castaneda G, Eurlings F, Hoyer D, Nykanen P, Sanz F, Thayer C and Vissers M (1999) The VATAM guidelines. International Journal of Medical Informatics 56, 107–155.CrossRefGoogle Scholar
  81. Taylor P (1999) Evaluating telemedicine systems and services. In Introduction to Telemedicine (CRAIG J and WOOTTON R, Eds), pp 105–120, Royal Society of Medicine Press, London.Google Scholar
  82. Van Gennip EMSJ and Lorenzi NM (1999) Results of discussions at the IMIA WG 13 and 15 working conference. International Journal of Medical Informatics 56, 177–180.CrossRefGoogle Scholar
  83. Wallace S (1998) Telemedicine in the NHS for the millennium and beyond. In Rethinking IT and Health (LENAGHAN J, Ed), pp 55–99, Institute for Public Policy Research, London.Google Scholar
  84. Walsham G (1993) Interpreting Information Systems in Organizations. Wiley, Chichester, England.Google Scholar
  85. Willcocks L and Lester S (1999) Information technology: transformer or sink hole. In Beyond the IT Productivity Paradox (WILLCOCKS L and LESTER S, Eds), Chichester, Wiley.Google Scholar
  86. Williams R and Edge D (1996) The social shaping of technology. In Information and Communication Technologies: Visions and Realities (DUTTON WH and PELTU M, Eds), Oxford University Press, Oxford (England) New York.Google Scholar
  87. Wilson M and Howcroft D (2000) Power, politics and persuasion: a social shaping perspective on IS evaluation. In 23rd IRIS Conference (SVENSSON L, SNIS U, SORENSEN C, FAGERLIND H, LINDROTH T, MAGNUSSON M and OSTLUND C, Eds), pp 725–739.Google Scholar
  88. Winner L (1977) Autonomous Technology: Technics-out-of-Control as a Theme in Political Thought. MIT Press, Cambridge, MA; London.Google Scholar
  89. Woolgar S (1996) Technologies as cultural artefacts. In Information and Communication Technologies: Visions and Realities (DUTTON WH and PELTU M, Eds), Oxford University Press, Oxford.Google Scholar
  90. Wyatt JC and Wyatt SM (2003) When and how to evaluate health information systems? International Journal of Medical Informatics 69, 251–259.CrossRefGoogle Scholar

Copyright information

© Operational Research Society 2005

Authors and Affiliations

  1. 1.Department of Information SystemsLondon School of Economics and Political ScienceLondonU.K.

Personalised recommendations