Skip to main content

Evaluation for Public Health Informatics

Part of the Health Informatics book series (HI)

Abstract

Evaluation is the application of specific criteria to determine the value or merit of the object of the study. Ensuring that public health information systems (ISs) and programs are managed wisely is essential. Evaluation answers the question of “why” a system is necessary, by collecting the data and performing the analysis needed to make determinations of efficiency and effectiveness and is a critical component to any public health informatics (PHI) project. Evaluation should occur at all stages of a PHI project. By using a combination of formative and summative evaluation, a well-designed plan provides key data to stakeholders that allow for informed decision-making about continuing, replacing, enhancing or retiring a public health IS. The design of the evaluation plan begins with identifying a mental model (e.g., information value cycle or data-information system-context-rings) from which to view the project and the evaluation objectives and determine what to evaluate. Conceptual frameworks, evaluation strategies, and methodology toolkits help define how the evaluation plan is developed and executed. A comprehensive program (e.g., the Centers for Disease Control and Prevention’s six-step evaluation framework) provides an example of an evaluation template.

Keywords

  • Evaluation
  • Formative evaluation
  • Summative evaluation
  • Mental models
  • Evaluation frameworks
  • Evaluation strategies
  • Information value cycle
  • Data-information-system-context rings
  • Mental models
  • Information Value Cycle
  • Data-Information System-Context Rings
  • CDC Six-Step Evaluation Framework

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-1-4471-4237-9_13
  • Chapter length: 22 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   44.99
Price excludes VAT (USA)
  • ISBN: 978-1-4471-4237-9
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Fig. 13.1
Fig. 13.2
Fig. 13.3

References

  1. Worthen BR, Sanders JR, Fitzpatrick JL. Program evaluation. 2nd ed. New York: Addison Wesley Longman; 1987.

    Google Scholar 

  2. Wagner JA. Encyclopedia of the Hundred Years War. Westport: Greenwood Publishing Group; 2006. p. 16–8.

    Google Scholar 

  3. Centers for Disease Control and Prevention. Framework for program evaluation in public health. MMWR 1999;48(No. RR-11). Available at: http://www.cdc.gov/eval/framework/index.htm. Accessed June 19, 2013.

  4. Friedman CP. A “fundamental theorem” of biomedical informatics. J Am Med Inform Assoc. 2009;16(2):169–70. URL: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2649317/?tool=pubmed. Last accessed 12 Mar 2013.

  5. Scriven M. The methodology of evaluation. Perspectives of curriculum evaluation. In: AERA monograph series on curriculum evaluation, No. 1. Chicago: Rand McNally; 1967. p. 39–83.

    Google Scholar 

  6. Stetler CB, Legro MW, Wallace CM, Bowman C, Guihan M, Hagedorn H, Kimmel B, Sharp ND, Smith JL. The role of formative evaluation in implementation research and the QUERI experience. J Gen Intern Med. 2006;21 Suppl 2:S1–8.

    PubMed  CrossRef  Google Scholar 

  7. Benedict LG. Traditional research versus evaluation. U.S. Department of Education. 1975. ERIC. Web. ED100959. Available from: http://files.eric.ed.gov/fulltext/ED100959.pdf. Cited 8 Apr 2013.

  8. Unite for Sight. Module 10: Distinguishing evaluation from research. Available at: http://www.uniteforsight.org/evaluation-course/module10. Cited 1 Aug 2013.

  9. Guba E. Significant differences. Edu Res. 1969;20:4–5.

    Google Scholar 

  10. Bainbridge L. Mental models in cognitive skill: the example of industrial process operation. In: Rogers Y, Rutherford A, Bibby PA, editors. Models in the mind. San Diego: Academic; 1992. p. 119–43.

    Google Scholar 

  11. Ammenwerth E, et al. Evaluation of health information systems—problems and challenges. Int J Med Inform. 2003;71(2):125–35.

    PubMed  CrossRef  Google Scholar 

  12. Taylor RS. Value-added processes in the information life cycle. J Am Soc Inf Sci. 1982;33:341–6.

    CrossRef  Google Scholar 

  13. Thacker SB, Qualters JR, Lee LM. Public health surveillance in the United States: evolution and challenges. MMWR Surveill Summ. 2012;61(Suppl):3–9.

    PubMed  Google Scholar 

  14. Laventure M. State and local perspective on Public Health Data Standards: opportunities for strategic action. Presented at: Annual meeting on Public Health Data Standards Consortium, Bethesda, 16–18 March 2004. Powerpoint slides.

    Google Scholar 

  15. Avgerou C. The significance of context in information systems and organizational change. Inf Syst J. 2001;11(1):43–63.

    CrossRef  Google Scholar 

  16. Lorenzi NM, et al. Antecedents of the people and organizational aspects of medical informatics review of the literature. J Am Med Inform Assoc. 1997;4(2):79–93.

    PubMed  CrossRef  CAS  Google Scholar 

  17. Ramaprasad A, Rai A. Envisioning management of information. Omega. 1996;24(2):179–93.

    CrossRef  Google Scholar 

  18. Harrison MI, Koppel R, Bar-Lev S. Unintended consequences of information technologies in health care—an interactive sociotechnical analysis. J Am Med Inform Assoc. 2007;14(5):542–9.

    PubMed  CrossRef  Google Scholar 

  19. Heeks R, Bathnagar S. Understanding success and failure in information age reform. In: Heeks R, editor. Reinventing government in the information age. London: Routledge; 1999. p. 49–74.

    CrossRef  Google Scholar 

  20. One Health Initiative Web site. URL: http://www.onehealthinitiative.com/. Last accessed 1 Apr 2013.

  21. Kaufman D, et al. Applying an evaluation framework for health information system design, development, and implementation. Nurs Res. 2006;55(2):S37–42.

    PubMed  CrossRef  Google Scholar 

  22. Stead WW, et al. Designing medical informatics research and library – resource projects to increase what is learned. J Am Med Inform Assoc. 1994;1(1):28–33.

    PubMed  CrossRef  CAS  Google Scholar 

  23. Currie LM. Evaluation frameworks for nursing informatics. Int J Med Inform 2005;74(11–12):908–16.

    Google Scholar 

  24. Yusof MM, Parazafeiropoulou A, Paul RJ, Stergioulas LK. Investigating evaluation frameworks for health information systems. Int J Med Inform 2008;77(6):377–85.

    Google Scholar 

  25. Cronholm S, Goldkuhl G. Strategies for information systems evaluation-six generic types. Electronic Journal of Information Systems Evaluation 2003;6(2):65–74.

    Google Scholar 

  26. Friedman CP, Wyatt J. Evaluation methods in biomedical informatics. New York, NY: Springer Science+Business Media, 2006.

    Google Scholar 

  27. Westbrook JI, et al. Evaluating the impact of information communication technologies on complex organizational systems: a multi-disciplinary, multi-method framework. Stud Health Technol Inform. 2004;107(Pt 2):1323–7.

    PubMed  Google Scholar 

  28. Anderson JG. Evaluation in health informatics: social network analysis. Comput Biol Med. 2002;32(3):179.

    Google Scholar 

  29. Merrill J, et al. Description of a method to support public health information management: organizational network analysis. J Biomed Inform. 2007;40(4):422.

    PubMed  CrossRef  Google Scholar 

  30. Donabedian A. Evaluating the quality of medical care. Milbank Mem Fund Q. 1966;44(3):166–206.

    PubMed  CrossRef  Google Scholar 

  31. DeLone WH, McLean ER. Information systems success: the quest for the dependent variable. Inf Syst Res. 1992;3(1):60–95.

    CrossRef  Google Scholar 

  32. Delone WH. The DeLone and McLean model of information systems success: a ten-year update. J Manage Inf Syst. 2003;19(4):9–30.

    Google Scholar 

  33. Fehrenbach N, Ross D, Hastings T, Renahan-White A. Towards measuring value: an evaluation framework for public health information systems. Public Health Informatics Institute. 2005. Web. Last accessed 1 Apr 2013.

    Google Scholar 

  34. Hebert M. Telehealth success: evaluation framework development. Stud Health Technol Inform. 2001;2:1145–9.

    Google Scholar 

  35. Cronholm S. Information Systems Evaluation-adding process descriptions to six evaluation types. European Conference on Information Technology Evaluation ECITE, Amsterdam, The Netherlands: Academic Conferences Limited; 2004.

    Google Scholar 

  36. “tool, n.”. OED Online. March 2013. Oxford University Press. Accessible from: http://www.oed.com/view/Entry/203258?rskey=WzPh27&result=1. Cited 8 Apr 2013.

  37. Health Resource and Services Administration. Health IT Adoption Toolbox. Accessed from: http://www.hrsa.gov/healthit/toolbox/. Cited 8 Apr 2013.

  38. Cusack CM, Byrne C, Hook JM, McGowan J, Poon EG, Zafar A. Health Information Technology Evaluation Toolkit: 2009 Update (Prepared for the AHRQ National Resource Center for Health Information Technology under Contract No. 290-04-0016.) AHRQ Publication No. 09-0083-EF. Rockville: Agency for Healthcare Research and Quality. 2009.

    Google Scholar 

  39. Centers for Disease Control and Prevention. Framework for program evaluation in public health. MMWR. 1999;48 (No. RR-11). URL: http://www.cdc.gov/eval/framework/index.htm. Last accessed 12 Mar 2013.

  40. Friedman CP. “Smallball” evaluation: a prescription for studying community-based information interventions. J Med Libr Assoc. 2005;93(4 Suppl):S43.

    PubMed  Google Scholar 

  41. Brender J. Handbook of evaluation methods for health informatics. Burlington, MA: Academic Press, 2006.

    Google Scholar 

  42. Creswell JW. Research design: qualitative, quantitative, and mixed methods approaches. Thousand Oaks: SAGE Publications, Incorporated; 2008.

    Google Scholar 

  43. Ericsson KA, Ralf TK, Clemens TR. The role of deliberate practice in the acquisition of expert performance. Psychol Rev. 1993;100(3):363.

    CrossRef  Google Scholar 

  44. Butler MA, Bender AD. Intensive care unit bedside documentation systems. Realizing cost savings and quality improvements. Comput Nurs. 1999;17(1):32.

    PubMed  CAS  Google Scholar 

  45. Heathfield H, Pitty D, Hanka R. Evaluating information technology in health care: barriers and challenges. BMJ. 1998;316:1959–61.

    PubMed  CrossRef  CAS  Google Scholar 

  46. Ammenwerth E, Mansmann U, Mahler C, Kandert M, Eichstadter R. Are quantitative methods sufficient to show why wards react differently to computer-based nursing document? In: Surjan G, Engelbrecht R, McNair P, editors. Proceedings of the 17th international congress of the European Federation for Medical Informatics (Medical Informatics Europe 2002 – health Data in the Information Society), 25–29 Aug 2002, Budapest, Studies in Health Technology and Informatics. Vol. 90. Amsterdam: IOS Press; 2002, p. 377–81.

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Paul C. Fu Jr. MD, MPH .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2014 Springer-Verlag London

About this chapter

Cite this chapter

Fu, P.C., Tolentino, H., Franzke, L.H. (2014). Evaluation for Public Health Informatics. In: Magnuson, J., Fu, Jr., P. (eds) Public Health Informatics and Information Systems. Health Informatics. Springer, London. https://doi.org/10.1007/978-1-4471-4237-9_13

Download citation

  • DOI: https://doi.org/10.1007/978-1-4471-4237-9_13

  • Published:

  • Publisher Name: Springer, London

  • Print ISBN: 978-1-4471-4236-2

  • Online ISBN: 978-1-4471-4237-9

  • eBook Packages: MedicineMedicine (R0)