Journal of Medical Systems

, Volume 31, Issue 5, pp 397–432 | Cite as

Methods to Evaluate Health information Systems in Healthcare Settings: A Literature Review



Although information technology (IT)-based applications in healthcare have existed for more than three decades, methods to evaluate outputs and outcomes of the use of IT-based systems in medical informatics is still a challenge for decision makers, as well as to those who want to measure the effects of ICT in healthcare settings. The aim of this paper is to review published articles in the area evaluations of IT-based systems in order to gain knowledge about methodologies used and findings obtained from the evaluation of IT-based systems applied in healthcare settings. The literature review includes studies of IT-based systems between 2003 and 2005. The findings show that economic and organizational aspects dominate evaluation studies in this area. However, the results focus mostly on positive outputs such as user satisfaction, financial benefits and improved organizational work. This review shows that there is no standard framework for evaluation effects and outputs of implementation and use of IT in the healthcare setting and that until today no studies explore the impact of IT on the healthcare system’ productivity and effectiveness.


Evaluation studies Medical informatics Literature review 


  1. 1.
    Kaplan, B., Evaluating informatics applications—some alternative approaches: theory, social interactionism, and call for methodological pluralism. Int. J. Med. Inform. 64:39–56, 2001.CrossRefGoogle Scholar
  2. 2.
    Ammenwerth, E., Brender, J., Nykanen, P., Prokosch, H. U., Rigby, M., and Talmon, J., Visions and strategies to improve evaluation of health information systems. Reflections and lessons based on the HIS-EVAL workshop in Innsbruck. Int. J. Med. Inform. 73:479–491, 2004.CrossRefGoogle Scholar
  3. 3.
    Ammenwerth, E., Iller, C., and Mansmann, U., Can evaluation studies benefit from triangulation? A case study. Int. J. Med. Inform. 70:237–248, 2003.CrossRefGoogle Scholar
  4. 4.
    Andersen, J. G., Aydin, C. E., and Jay, S. J., Evaluating Healthcare Information Systems: Methods and Applications. Thousand Oaks, CA: Sage publication, 1994.Google Scholar
  5. 5.
    Friedman, C. P., and Wyatt, J. C., Evaluation Methods in Medical Informatics. New York: Springer, 1996.Google Scholar
  6. 6.
    Kaplan, B., and Shaw, N. T., Future directions in evaluation research: People, organizational, and social issues. Methods Inf. Med. 43:215–231, 2004.Google Scholar
  7. 7.
    Phil, W., An assessment of strategies for information system evaluation: Lessons for education. Int. J. Educ. Res. 25:361–371, 1996.CrossRefGoogle Scholar
  8. 8.
    Delpierre, C., Cuzin, L., Fillaux, J., Alvarez, M., Massip, P., and Lang, T., A systematic review of computer-based patient record systems and quality of care: More randomized clinical trials or a broader approach. Int. J. Qual. Health Care. 16:407–416, 2004.CrossRefGoogle Scholar
  9. 9.
    Wang, S. J., Middleton, B., Bardon, C. G., Spurr, C. D., Carchidi, P. J., Kittler, A. f., Goldszar, R. C., Fairchild, D. G., Sussman, A. J., Kuperman, G. J., and Bates, D. W., A cost-benefit analysis of electronic medical records in primary care. Am. J. Med. 114:397–403, 2003.CrossRefGoogle Scholar
  10. 10.
    Laerum, H., Karlsen, T. H., and Faxvaag, A., Effects of scanning and eliminating paper-based medical records on hospital physicians’ clinical work practice. J. Am. Med. Inform. Assoc. 10:160–588, 2003.CrossRefGoogle Scholar
  11. 11.
    Ruland, C. M., and Ravn, I. H., Usefulness and effects on costs and staff management of a nursing resource management information system. J. Nurs. Manag. 11:208–215, 2003.Google Scholar
  12. 12.
    Houston, T. K., Ray, M. N., Crawford, M. A., Giddens, T., and Berner, E. S., Patient perceptions of physician use of handheld computers. AMIA Annu. Symp. Proc. 299–303, 2003.Google Scholar
  13. 13.
    Ammenwerth, E., Mansmann, U., Iller, C., and Eichstädter, R., Factors affecting and affected by user acceptance of computer-based nursing documentation: Results of a two-year study. J. Am. Med. Inform. Assoc. 10:69–84, 2003.CrossRefGoogle Scholar
  14. 14.
    Zhang, W. P., Yamauchi, K., Mizuno, S., Zhang, R., and Huang, D. M., Analysis of cost and assessment of computerized patient record systems in Japan based on questionnaire survey. Med. Inform. Internet Med. 29:229–238, 2004.CrossRefGoogle Scholar
  15. 15.
    Laerum, H., Karlsen, T. H., and Faxvaag, A., Use of and attitudes to a hospital information system by medical secretaries, nurses and physicians deprived of the paper-based medical record: A case report BMC Med. Inform. Decis. Mak. 4:18, 2004.CrossRefGoogle Scholar
  16. 16.
    Hier, D. B., Rothschild, A., LeMaistre, A., and Keeler, J., Differing faculty and housestaff acceptance of an electronic health record one year after implementation. Medinfo. 11:1300–1303, 2004.Google Scholar
  17. 17.
    Rotich, J. K., Hannan, T. J., Smith, F. E., Bii, J., Odero, W. W., Vu, N., Mamlin, B. W., Mamlin, J. J., Einterz, R. M., and Tierney, W. M., Installing and implementing a computer-based patient record system in sub-Saharan Africa: The Mosoriot Medical Record System. Med. Inform. Internet Med. 10:295–303, 2003.Google Scholar
  18. 18.
    Stengel, D., Bauwens, K., Walter, M., Kopfer, T., and Ekkernkamp, A., Comparison of handheld computer-assisted and conventional paper chart documentation of medical records. A randomized, controlled trial. J. Bone Joint Surg. Am. 86:553–560, 2004.Google Scholar
  19. 19.
    Likourezos, A., Chalfin, D. B., Murphy, D. G., Sommer, B., Darcy, K., and Davidson, S. J., Physician and nurse satisfaction with an Electronic Medical Record System. J. Emerg. Med. 27:419–424, 2004.CrossRefGoogle Scholar
  20. 20.
    Pagliari, C., Gilmour, M., and Sullivan, F., Evaluating national electronic clinical communications implementation in Scotland. Medinfo. 11:1794, 2004.Google Scholar
  21. 21.
    Baum, A., Figar, S., Serverino, J., Assale, D., Schachner, B., Otero, P., Luna, D., and de Quiros, F. G., Assessing the impact of change in the organization of a technical support system for a health information systems (HIS). Medinfo. 11:1367–1370, 2004.Google Scholar
  22. 22.
    Hobson, J. C., Khemani, S., and Singh, A., Prospective audit of the quality of ENT emergency clinic notes before and after introduction of a computerized template. J. Laryngol. Otol. 119:264–266, 2005.CrossRefGoogle Scholar
  23. 23.
    Kaushal, R., Blumenthal, D., Poon, E. G., Jha, A. K., Franz, C., Middleton, B., Glaser, J., Kuperman, G., Christino, M., Fernandopulle, R., Newhouse, J. P., and Bates, D. W., The costs of a national health information network. Ann. Intern. Med. 143:165–173, 2005.Google Scholar
  24. 24.
    Raja, E. E., Mahal, R., and Masih, V. B., An exploratory study to assess the computer knowledge, attitude and skill among nurses in healthcare setting of a selected hospital in Ludhiana, Punjab, India. Medinfo. 11:1304–1307, 2004.Google Scholar
  25. 25.
    Clayton, P. D., Naus, S. P., Bowes, W. A., Madsen, T. S., Wilcox, A. B., Orsmond, G., Rocha, B., Thornton, S. N., Jones, S., Jacobsen, C. A., Udall, M. R., Rhodes, M. L., Wallace, B. E., Cannon, W., Gardner, J., Huff, S. M., and Leckman, L., Physician use of electronic medical records: issues and successes with direct data entry and physician productivity. AMIA Annu. Symp. Proc. 141–145, 2005.Google Scholar
  26. 26.
    White, M., Computerised versus handwritten records. Paediatr. Nurs. 17:15–18, 2005.Google Scholar
  27. 27.
    Wilcox, A. B., Jones, S. S., Dorr, D. A., Cannon, W., Burns, L., Radican, K., Christensen, K., Brunker, C., Larsen, A., Narus, S. P., Thornton, S. N., and Clayton, P. D., Use and impact of a computer-generated patient summary worksheet for primary care. AMIA Annu. Symp. Proc. 824–828, 2005.Google Scholar
  28. 28.
    Makela, K., Virjo, I., Aho, J., Kalliola, P., Koivukoski, A. M., Kurunmaki, H., Kahara, M., Uusitalo, L., Valli, M., Vuotari, V., and Ylinen, S., Electronic patient record systems and the general practitioner: an evaluation study. J. Telemed. Telecare. 11:S2:66–68, 2005.CrossRefGoogle Scholar
  29. 29.
    Zurita, L., and Nohr, C., Patient opinion–EHR assessment from the users perspective. Medinfo. 11:1333–1336, 2004.Google Scholar
  30. 30.
    Breslow, M. J., Rosenfeld, B. A., Doerfler, M., Burke, G., Yates, G., Stone, D. J., Tomaszewicz, P., Hochman, R., and Plocher, D. W., Effect of a multiple-site intensive care unit telemedicine program on clinical and economic outcomes: an alternative paradigm for intensivist staffing. Crit. Care Med. 32:31–38, 2004.CrossRefGoogle Scholar
  31. 31.
    Scalvini, S., Capomolla, S., Zanelli, E., Benigno, M., Domenighini, D., Paletta, L., Glisenti, F., and Giordano, A., Effect of home-based telecardiology on chronic heart failure: Costs and outcomes. J. Telemed. Telecare. 11:S1:16–18, 2005.CrossRefGoogle Scholar
  32. 32.
    Noble, S. M., Coast, J., and Benger, J. R., A cost-consequences analysis of minor injuries telemedicine. J. Telemed. Telecare. 11:15–19, 2005.CrossRefGoogle Scholar
  33. 33.
    Persaud, D. D., Jreige, S., Skedgel, C., Finley, J., Sargeant, J., and Hanlon, N., An incremental cost analysis of telehealth in Nova Scotia from a societal perspective. J. Telemed. Telecare. 11:77–84, 2005.CrossRefGoogle Scholar
  34. 34.
    Dillon, E., Loermans, J., Davis, D., and Xu, C., Evaluation of the westerrn Australian Department of Health telehealth project. J. Telemed. Telecare. 11:S2:19–S2:121, 2005.CrossRefGoogle Scholar
  35. 35.
    Larcher, B., Arisi, E., Berloffa, F., Demichelis, F., Eccher, C., Galligioni, E., Galvagni, M., Martini, G., Sboner, A., Tomio, L., Zumiani, G., Graiff, A., and Forti, S., Analysis of user-satisfaction with the use of a teleconsultation system in oncology. Med. Inform. Internet. Med. 28:73–84, 2003.CrossRefGoogle Scholar
  36. 36.
    Bratan, T., Clarke, M., Jones, R., Larkworthy, A., and Paul, R., Evaluation of the practical feasibility and acceptability of home monitoring in residential homes. J. Telemed. Telecare. 11(Suppl 1):29–31, 2005.CrossRefGoogle Scholar
  37. 37.
    Mair, F. S., Goldstein, P., May, C., Angus, R., Shiels, C., Hibbert, D., O’Connor, J., Boland, A., Roberts, C., Haycox, A., and Capewell, S., Patient and provider perspectives on home telecare: preliminary results from a randomized controlled trial. J. Telemed. Telecare. 11(Suppl 1):95–97, 2005.CrossRefGoogle Scholar
  38. 38.
    Linassi, A. G., and Li Pi Shan, R., User satisfaction with a telemedicine amputee clinic in Saskatchewan. J. Telemed. Telecare. 11:414–418, 2005.CrossRefGoogle Scholar
  39. 39.
    Young, T. L., and Ireson, C., Effectiveness of school-based telehealth care in urban and rural elementary schools. Pediatrics. 112:1088–1094, 2003.CrossRefGoogle Scholar
  40. 40.
    Baldwin, L., Clarke, M., Hands, L., Knott, M., and Jones, R., The effect of telemedicine on consultation time. J. Telemed. Telecare. 9:S1:71–73, 2003.CrossRefGoogle Scholar
  41. 41.
    Finkelstein, J., Khare, R., and Ansell, J., Feasibility and patients’ acceptance of home automated telemanagement of oral anticoagulation therapy. Proc. AMIA Symp. 230–234, 2003.Google Scholar
  42. 42.
    Vontetsianos, T., Giovas, P., Katsaras, T., Rigopoulou, A., Mpirmpa, G., Giaboudakis, P., Koyrelea, S., Kontopyrgias, G., and Tsoulkas, B., Telemedicine-assisted home support for patients with advanced chronic obstructive pulmonary disease: preliminary results after nine-month follow-up. J. Telemed. Telecare. 11:S1:86–S1:88, 2005.CrossRefGoogle Scholar
  43. 43.
    Grady, B. J., and Melcer, T., A retrospective evaluation of TeleMental Healthcare services for remote military populations. Telemed J E Health. 11:551–558, 2005.CrossRefGoogle Scholar
  44. 44.
    Wong, Y. K., Hui, E., and Woo, J., A community-based exercise programme for older persons with knee pain using telemedicine. J. Telemed. Telecare. 11:310–315, 2005.CrossRefGoogle Scholar
  45. 45.
    Beuscart-Zephir, M. C., Pelayo, S., Degoulet, P., Anceaux, F., Guerlinger, S., and Meaux, J. J., A usability study of CPOE’s medication administration functions: impact on physician-nurse cooperation. Medinfo. 11:1018–1022, 2004.Google Scholar
  46. 46.
    Galanter, W. L., Polikaitis, A., and DiDomenico, R. J., A trial of automated safety alerts for inpatient digoxin use with computerized physician order entry. J. Am. Med. Inform. Assoc. 11:270–277, 2004.CrossRefGoogle Scholar
  47. 47.
    Shulman, R., Singer, M., Goldstone, J., and Bellingan, G., Medication errors: a prospective cohort study of hand-written and computerised physician order entry in the intensive care unit. Crit. Care. 9:R516–R521, 2005.CrossRefGoogle Scholar
  48. 48.
    Galanter, W. L., Didomenico, R. J., and Polikaitis, A., A trial of automated decision support alerts for contraindicated medications using computerized physician order entry. J. Am. Med. Inform. Assoc. 12:269–274, 2005.CrossRefGoogle Scholar
  49. 49.
    Sintchenko, V., Iredell, J. R., Gilbert, G. L., and Coiera, E., Handheld computer-based decision support reduces patient length of stay and antibiotic prescribing in critical care. J. Am. Med. Inform. Assoc. 12:398–402, 2005.CrossRefGoogle Scholar
  50. 50.
    Short, D., Frischer, M., and Bashford, J., The development and evaluation of a computerised decision support system for primary care based upon ‘patient profile decision analysis’. Inform. Prim. Care. 11:195–202, 2003.Google Scholar
  51. 51.
    Ruland, C. M., White, T., Stevens, M., Fanciullo, G., and Khilani, S. M., Effects of a computerized system to support shared decision making in symptom management of cancer patients: Preliminary results. J. Am. Med. Inform. Assoc. 10:573–579, 2003.CrossRefGoogle Scholar
  52. 52.
    Park, R. W., Shin, S. S., Choi, Y. I., Ahn, J. O., and Hwang, S. C., Computerized physician order entry and electronic medical record systems in Korean teaching and general hospitals: Results of a 2004 survey. J. Am. Med. Inform. Assoc. 12:642–647, 2005.CrossRefGoogle Scholar
  53. 53.
    Gandhi, T. K., Weingart, S. N., Seger, A. C., Borus, J., Burdick, E., Poon, E. G., Leape, L. L., and Bates, D. W., Outpatient prescribing errors and the impact of computerized prescribing. J. Gen. Intern. Med. 20:837–841, 2005.CrossRefGoogle Scholar
  54. 54.
    Ohsfeldt, R. L., Ward, M. M., Schneider, J. E., Jaana, M., Miller, T. R., Lei, Y., and Wakefield, S., Implementation of hospital computerized physician order entry systems in a rural state: Feasibility and financial impact. J. Am. Med. Inform. Assoc. 12:20–27, 2005, Jan.CrossRefGoogle Scholar
  55. 55.
    Bogucki, B., Jacobs, B. R., and Hingle, J., Computerized reminders reduce the use of medications during shortages. J. Am. Med. Inform. Assoc. 11:278–280, 2004.CrossRefGoogle Scholar
  56. 56.
    Van den Brink, J. L., Moorman, P. W., de Boer, M. F., Pruyn, J. F., Verwoerd, C. D., and van emmel, J. H., Involving the patient: a prospective study on use, appreciation and effectiveness of an information system in head and neck cancer care. Int. J. Med. Inform. 74:839–849, 2005.CrossRefGoogle Scholar
  57. 57.
    Choi, S. S., Jazayeri, D. G., Mitnick, C. D., Chalco, K., Bayona, J., and Fraser, H. S., Implementation and initial evaluation of a Web-based nurse order entry system for multidrug-resistant tuberculosis patients in Peru. Medinfo. 11:202–206, 2004.Google Scholar
  58. 58.
    Odhiambo-Otieno, G. W., Evaluation of existing district health management information systems a case study of the district health systems in Kenya. Int. J. Med Inform. 74:733–744, 2005.CrossRefGoogle Scholar
  59. 59.
    Scott, J. T., Rundall, T. G., Vogt, T. M., and Hsu, J., Kaiser Permanente’s experience of implementing an electronic medical record: A qualitative study. BMJ. 331:1313–1316, 2005.CrossRefGoogle Scholar
  60. 60.
    Aarts, J., and Berg, M., A tale of two hospitals: a sociotechnical appraisal of the introduction of computerized physician order entry in two Dutch hospitals. Medinfo. 11:999–1002, 2004.Google Scholar
  61. 61.
    Koppel, R., Metlay, J. P., Cohen, A., Abaluck, B., Localio, A. R., Kimmel, S. E., and Strom, B. L., Role of computerized physician order entry systems in facilitating medication errors. JAMA. 293:1197–1203, 2005.CrossRefGoogle Scholar
  62. 62.
    Vimarlund, V., and Olve, N. G., Economic analyses for ICT in elderly healthcare: Questions and challenges. Health Informatics J. 11:309–321, 2005.CrossRefGoogle Scholar
  63. 63.
    Olve, N. G., Vimarlund, V., and Agerbo, M. Evaluation as multi-actor trade-off—A challenge in introducing ICT innovations in the health sector. Invited paper, 4th WSEAS International Conference on E-Activities. Miami, Florida, USA, November 17–19, 2005.Google Scholar
  64. 64.
    Adams, W. G., Mann, A. M., and Bauchner, H., Use of an electronic medical record improves the quality of urban pediatric primary care. Pediatrics. 111:626–632, 2003.CrossRefGoogle Scholar
  65. 65.
    Pizziferri, L., Kittler, A. F., Volk, L. A., Honour, M. M., Gupta, S., Wang, S., Wang, T., Lippincott, M., Li, Q., and Bates, D. W., Primary care physician time utilization before and after implementation of an electronic health record: A time-motion study. J. Biomed. Inform. 38:176–188, 2005.CrossRefGoogle Scholar
  66. 66.
    Bryson, M., Tidy, N., Smith, M., and Levy, S., An online survey of nurses’ perceptions, knowledge and expectations of the National Health Service modernization programme. J. Telemed. Telecare. 11:S1:64–66, 2005.CrossRefGoogle Scholar
  67. 67.
    Hebert, M. A., and Korabek, B., Stakeholder readiness for telehomecare: implications for implementation. Telemed J E Health. 10:85–92, 2004.CrossRefGoogle Scholar
  68. 68.
    Kaufman, D. R., Starren, J., Patel, V. L., Morin, P. C., Hilliman, C., Pevzner, J., Weinstock, R. S., Goland, R., and Shea, S. A cognitive framework for understanding barriers to the productive use of a diabetes home telemedicine system. AMIA Annu. Symp. Proc. 356–360, 2003.Google Scholar
  69. 69.
    Ash, J. S., Sittig, D. F., Seshadri, V., Dykstra, R. H., Carpenter, J. D., and Stavri, P. Z., Adding insight: a qualitative cross-site study of physician order entry. Medinfo. 11:1013–1017, 2004.Google Scholar
  70. 70.
    Ash, J. S., Gorman, P. N., Seshadri, V., and Hersh, W. R., Computerized physician order entry in U.S. hospitals: results of a 2002 survey. J. Am. Med. Inform. Assoc. 11:95–99, 2004.CrossRefGoogle Scholar
  71. 71.
    Ammenwerth, E., and de Keizer, N., An inventory of evaluation studies of information technology in healthcare trends in evaluation research 1982–2002. Methods Inf. Med. 44:44–56, 2005.Google Scholar
  72. 72.
    Kaplan, B., Evaluating informatics applications—Clinical decision support systems literature review. Int. J. Med. Inform. 64:39–56, 2001.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2007

Authors and Affiliations

  1. 1.Department of Computer and Information ScienceLinköping UniversityLinköpingSweden
  2. 2.Department of Social MedicineUrmia University of Medical SciencesUrmiaIran

Personalised recommendations