The role of formative evaluation in implementation research and the QUERI experience

  • Cheryl B. Stetler
  • Marcia W. Legro
  • Carolyn M. Wallace
  • Candice Bowman
  • Marylou Guihan
  • Hildi Hagedorn
  • Barbara Kimmel
  • Nancy D. Sharp
  • Jeffrey L. Smith


This article describes the importance and role of 4 stages of formative evaluation in our growing understanding of how to implement research findings into practice in order to improve the quality of clinical care. It reviews limitations of traditional approaches to implementation research and presents a rationale for new thinking and use of new methods. Developmental, implementation-focused, progress-focused, and interpretive evaluations are then defined and illustrated with examples from Veterans Health Administration Quality Enhancement Research Initiative projects. This article also provides methodologic details and highlights challenges encountered in actualizing formative evaluation within implementation research.

Key words

process assessment (health care) evaluation methodology evaluation studies 


  1. 1.
    QUERI. HSRD special solicitation for service directed projects (SDP) on implementation of research into practice (posted 2003). Available at: Accessed August 10, 2005.Google Scholar
  2. 2.
    Demakis JG, McQueen L, Kizer KW, Feussner JR. Quality enhancement research initiative (QUERI): a collaboration between research and clinical practice. Med Care. 2000;38(suppl 1):117–25.Google Scholar
  3. 3.
    McQueen L, Mittman BS, Demakis JG. Overview of the Veterans Health Administration (VHA) Quality Enhancement Research Initiative (QUERI). J Am Med Inform Assoc. 2004;11:339–43.CrossRefPubMedGoogle Scholar
  4. 4.
    QUERI. QUERI program description (posted July 2003). Available at: Accessed August 10, 2005.Google Scholar
  5. 5.
    Cook TD, Campbell DT. Quasi-Experimentation: Design & Analysis Issues for Field Settings. Boston: Houghton Mifflin Company; 1979.Google Scholar
  6. 6.
    Campbell DT, Stanley JC. Experimental and Quasi-Experimental Designs for Research. Boston: Houghton Mifflin Company; 1963.Google Scholar
  7. 7.
    Isaac S, Michael W. Handbook in Research and Evaluation: For Education and the Behavioral Sciences. San Diego: EdITS Publishers; 1982.Google Scholar
  8. 8.
    Dzewaltowski D, Estabrooks P, Glasgow R, Klesges L. Workgoup to evaluate and enhance the reach and dissemination of health promotion interventions (RE-AIM). Available at: Accessed July 9, 2005.Google Scholar
  9. 9.
    Glasgow RE, Lichtenstein E, Marcus A. Why don’t we see more translation of health promotion research to practice? Rethinking the efficacy-to-effectiveness transition. Am J Pub Health. 2003; 93:1261–7.CrossRefGoogle Scholar
  10. 10.
    Stead M, Hastings G, Eadie D. The challenge of evaluating complex interventions: a framework for evaluating media advocacy. Health Educ Res. 2002;17:351–64.CrossRefPubMedGoogle Scholar
  11. 11.
    Zapka J, Goins KV, Pbert L, Ockene JK. Translating efficacy research to effectiveness studies in practice: lessons from research to promote smoking cessation in community health centers. Health Promot Pract. 2004;5:245–55.CrossRefPubMedGoogle Scholar
  12. 12.
    Ovretveit J, Gustafson D. Using research to inform quality programmes. BMJ. 2003;326:759–61.CrossRefPubMedGoogle Scholar
  13. 13.
    Hulscher M, Laurant M, Grol R. Process evaluation on quality improvement interventions. Qual Saf Health Care. 2002;12:40–6.CrossRefGoogle Scholar
  14. 14.
    Solberg LI. Guideline implementation: what the literature doesn’t tell us. Jt Comm J Qual Improv. 2000;26:525–37.PubMedGoogle Scholar
  15. 15.
    Mittman B. Creating the evidence base for quality improvement collaboratives. Ann Int Med. 2004;140:897–901.PubMedGoogle Scholar
  16. 16.
    Dehar MA, Casswell S, Duignan P. Formative and process evaluation of health promotion and disease prevention programs. Eval Rev. 1993;17:204–20.CrossRefGoogle Scholar
  17. 17.
    Rossi P, Freeman H. Evaluation: A Systematic Approach. Newbury Park: Sage Publications; 1993.Google Scholar
  18. 18.
    Bhola HS. Evaluating “Literacy for Development” Projects, Programs and Campaigns: Evaluation Planning, Design and Implementation, and Utilization of Evaluation Results. Hamburg, Germany: UNESCO Institute for Education; DSE (German Foundation for International Development), xii; 1990.Google Scholar
  19. 19.
    Patton MQ. Evaluation of program implementation. Eval Stud Rev Annu. 1979;4:318–45.Google Scholar
  20. 20.
    Altman DG. A framework for evaluating community-based heart disease prevention programs. Soc Sci Med. 1986;22:479–87.CrossRefPubMedGoogle Scholar
  21. 21.
    Havas S, Anliker J, Damron D, Feldman R, Langenberg P. Uses of process evaluation in the Maryland WIC 5-a-day promotion program. Health Educ Behav. 2000;27:254–63.CrossRefPubMedGoogle Scholar
  22. 22.
    Evans RI, Raines BE, Owen AE. Formative evaluation in school-based health promotion investigations. Prev Med. 1989;18:229–34.CrossRefPubMedGoogle Scholar
  23. 23.
    Patton MQ. Utilization-Focused Evaluation: The New Century Text. 3rd edn. Thousand Oaks, CA: Sage Publications; 1997.Google Scholar
  24. 24.
    Patton MQ. Qualitative Research & Evaluation Methods. 3rd edn. Newbury Park, CA: Sage Publications; 2001.Google Scholar
  25. 25.
    Walshe K, Freeman T. Effectiveness of quality improvement: learning from evaluations. Qual Saf Health Care. 2002;11:85–7.CrossRefPubMedGoogle Scholar
  26. 26.
    Wholey J, Hatry H, Newcomer K, eds. Handbook of Practical Program Evaluation. San Francisco: Jossey-Bass Publishers; 1994.Google Scholar
  27. 27.
    Forsetlund L, Talseth KO, Bradley P, Nordheim L, Bjorndal A. Many a slip between cup and lip. Process evaluation of a program to promote and support evidence-based public health practice. Eval Rev. 2003;27:179–209.CrossRefPubMedGoogle Scholar
  28. 28.
    Guide for Implementing Evidence-Based Clinical Practice and Conducting Implementation Research. Available at: Accessed August 10, 2005.Google Scholar
  29. 29.
    van Bokhoven MA, Kok G, van der Weijden T. Designing a quality improvement intervention: a systematic approach. Qual Saf Health Care. 2003;3:215–20.CrossRefGoogle Scholar
  30. 30.
    Rogers EM. Diffusion of Innovations. 4th edn. New York: Free Press; 1995.Google Scholar
  31. 31.
    Stetler C, Corrigan B, Sander-Buscemi K, Burns M. Integration of evidence into practice and the change process: a fall prevention program as a model. Outcomes Manag Nurs Practice. 1999;3:102–11.Google Scholar
  32. 32.
    Sales A. Organizational readiness for evidence-based health care interventions. Available at:;instrument_reviews.asp?detail=53. Accessed August 10, 2005.Google Scholar
  33. 33.
    Luther SL, Nelson A, Powell-Cope G. Provider attitudes and beliefs about clinical practice guidelines. SCI Nurs. 2004;21:206–12.PubMedGoogle Scholar
  34. 34.
    Brown K, Gerhardt M. Formative evaluation: an integrative practice model and case study. Personnel Psychol. 2002;55:951f.Google Scholar
  35. 35.
    Legro M, Wallace C, Hatzakis M, Goldstein B. Barriers to optimal use of computerized clinical reminders: the SCI QUERI experience. VA QUERI Quart. 2003;4:2.Google Scholar
  36. 36.
    Wallace C, Hatzakis M, Legro M, Goldstein B. Understanding a VA preventive care clinical reminder: lessons learned. SCI Nurs. 2004;21:149–52.PubMedGoogle Scholar
  37. 37.
    Stetler C. The role of the organization in translating research into evidence based practice. Outcomes Manag Nurs Practice. 2003;7:97–103.Google Scholar
  38. 38.
    Rycroft-Malone J, Kitson A, Harvey G, et al. Ingredients for change: revisiting a conceptual framework. Qual Saf Health Care. 2002;11:174–80.CrossRefPubMedGoogle Scholar
  39. 39.
    McCormack B, Kitson A, Harvey G, et al. Getting evidence into practice: the meaning of ‘context’. J Adv Nurs. 2002;38:94–104.CrossRefPubMedGoogle Scholar
  40. 40.
    Bradley E, Holmboe E, Mattera J, et al. A qualitative study of increasing β-blocker use after myocardial infarction: why do some hospitals succeed? J Am Med Assoc. 2001;285:2604–11.CrossRefGoogle Scholar
  41. 41.
    Kimmel B. Bridging the gap between knowledge and practice — the veterans administration pathway, Newsletter of the National Council of the University Research Administrators 2003-4;35.Google Scholar
  42. 42.
    Dunlap M, Beyth R, Deswal A, Massie B, Saleh J, Kimmel B. VA practice matters on treating chronic heart failure. VA Practice Matters. 2004;9:1–8. Available at: Accessed September 9, 2005.Google Scholar
  43. 43.
    Hawe P, Shiell A, Riley T. Complex interventions: how “out of control” can a randomized controlled trial be? BMJ. 2004;328:1561–3.CrossRefPubMedGoogle Scholar
  44. 44.
    Kirchhoff K, Dille C. Issues in intervention research: maintaining integrity. Appl Nurs Res. 1994;7:32–8.CrossRefPubMedGoogle Scholar
  45. 45.
    Santacroce SJ, Maccarelli LM, Grey M. Intervention fidelity. Nurs Res. 2004;53:63–6.CrossRefPubMedGoogle Scholar
  46. 46.
    Boyd NR, Windsor RA. A formative evaluation in maternal and child health practice: the partners for life nutrition education program for pregnant women. Matern Child Health J. 2003;7:137–43.CrossRefPubMedGoogle Scholar
  47. 47.
    Stetler C, Creer E, Effken J. Evaluating a redesign program: challenges and opportunities. In Kelly K., ed. Series on Nursing Administration, Vol. 8. St. Louis: Mosby Year Book; 1996.Google Scholar
  48. 48.
    Effken E, Stetler C. Impact of organizational redesign. J Nurs Admin. 1997;27:23–32.Google Scholar
  49. 49.
    Krumholz H, Herrin J. Quality improvement: the need is there but so are the challenges. Am J Med. 2000;109:501–3.CrossRefPubMedGoogle Scholar
  50. 50.
    Willenbring ML, Hagedorn H. Implementing evidence-based practices in opioid agonist therapy clinics. In: Roberts A., Yeager K., eds. Evidence-Based Practice Manual: Research and Outcome Measures in Health and Human Services 2004. New York: Oxford University Press; 2004:340–7.Google Scholar
  51. 51.
    Willenbring ML, Hagedorn H, Poster AC, Kenny M. Variations in evidence-based clinical practices in nine United States Veterans Administration opioid agonist therapy clinics. Drug Alcohol Dependence. 2004;75:97–106.CrossRefGoogle Scholar
  52. 52.
    LaVela S, Legro M, Weaver F, Smith B. Staff influenza vaccination: lessons learned. SCI Nurs. 2004;21:153–7.PubMedGoogle Scholar
  53. 53.
    La Vela SL, Legro MW, Weaver FM, Goldstein B, Smith B. Do Patient Intentions Predict Vaccination Behavior Over Time? Poster. Academy Health Annual Conference June 2004. San Diego, CA.Google Scholar
  54. 54.
    Nazareth I, Freemantle N, Duggan C, Mason J, Haines A. Evaluation of a complex intervention for changing professional behaviour: the evidence based out reach (EBOR) Trial. J Health Serv Res Policy. 2002;7:230–8.CrossRefPubMedGoogle Scholar
  55. 55.
    Kukafka R, Johnson SB, Linfante A, Allegrante JP. Grounding a new information technology implementation framework in behavioral science: a systematic analysis of the literature on IT use. J Biomed Inform. 2003;36:218–27.CrossRefPubMedGoogle Scholar
  56. 56.
    Sanson-Fisher RW, Grimshaw JM, Eccles MP. The science of changing providers’ behaviour: the missing link in evidence-based practice. Med J Aust. 2004;180:205–6.PubMedGoogle Scholar
  57. 57.
    Walker AE, Grimshaw J, Johnston M, Pitts N, Steen N, Eccles M. PRIME—PRocess modelling in ImpleMEntation research: selecting a theoretical basis for interventions to change clinical practice. BMC Health Serv Res. 2003;3:22.CrossRefPubMedGoogle Scholar
  58. 58.
    Yin R. Case Study Research: Design and Methods. Thousand Oaks: Sage; 1994.Google Scholar
  59. 59.
    Patterson ES, Nguyen AD, Halloran JM, Asch SM. Human factors barriers to the effective use of ten HIV clinical reminders. J Am Med Inform Assoc. 2004;11:50–9.CrossRefPubMedGoogle Scholar
  60. 60.
    Sharp ND, Pineros SL, Hsu C, Starks H, Sales AE. A qualitative study to identify barriers and facilitators to implementation of pilot interventions in the Veterans Health Administration (VHA) Northwest Network. Worldviews Evidence-Based Nurs. 2004;1:129–39.CrossRefGoogle Scholar
  61. 61.
    Devers KJ, Sofaer S, Rundall TG (Guest eds.). Qualitative methods in health services research: a special supplement to HSR. Health Services Res. 1999;34(part II):1083–263.Google Scholar
  62. 62.
    Litchman J, Roumanis S, Radford M, et al. Can practice guidelines be transported effectively to different settings? Results from a multi-center interventional study. J Comm J Qual Improv. 2001;27:42–53.Google Scholar

Copyright information

© Society of General Internal Medicine 2006

Authors and Affiliations

  • Cheryl B. Stetler
    • 1
  • Marcia W. Legro
    • 2
    • 3
  • Carolyn M. Wallace
    • 2
  • Candice Bowman
    • 4
  • Marylou Guihan
    • 5
  • Hildi Hagedorn
    • 6
  • Barbara Kimmel
    • 7
    • 8
  • Nancy D. Sharp
    • 2
  • Jeffrey L. Smith
    • 9
  1. 1.Independent ConsultantAmherstUSA
  2. 2.VA Puget Sound Health Care SystemSeattleUSA
  3. 3.University of WashingtonSeattleUSA
  4. 4.VA San Diego Healthcare SystemSan DiegoUSA
  5. 5.Edward Hines, Jr. VA Healthcare SystemHinesUSA
  6. 6.Minneapolis VA Medical CenterMinneapolisUSA
  7. 7.Baylor College of MedicineHoustonUSA
  8. 8.Houston Veterans Affairs Medical CenterHoustanUSA
  9. 9.Central Arkansas Veterans Healthcare SystemLittle RockUSA

Personalised recommendations