Translational Behavioral Medicine

, Volume 7, Issue 3, pp 478–485 | Cite as

Engaging multilevel stakeholders in an implementation trial of evidence-based quality improvement in VA women’s health primary care

  • Alison B. HamiltonEmail author
  • Julian Brunner
  • Cindy Cain
  • Emmeline Chuang
  • Tana M. Luger
  • Ismelda Canelo
  • Lisa Rubenstein
  • Elizabeth M. Yano
Original Research


The Veterans Health Administration (VHA) has undertaken primary care transformation based on patient-centered medical home (PCMH) tenets. VHA PCMH models are designed for the predominantly male Veteran population, and require tailoring to meet women Veterans’ needs. We used evidence-based quality improvement (EBQI), a stakeholder-driven implementation strategy, in a cluster randomized controlled trial across 12 sites (eight EBQI, four control) that are members of a Practice-Based Research Network. EBQI involves engaging multilevel, inter-professional leaders and staff as stakeholders in reviewing evidence and setting QI priorities. The goal of this analysis was to examine processes of engaging stakeholders in early implementation of EBQI to tailor VHA’s medical home for women. Four inter-professional regional stakeholder planning meetings were conducted; these meetings engaged stakeholders by providing regional data about gender disparities in Veterans’ care experiences. Subsequent to each meeting, qualitative interviews were conducted with 87 key stakeholders (leaders and staff). Stakeholders were asked to describe QI efforts and the use of data to change aspects of care, including women’s health care. Interview transcripts were summarized and coded using a hybrid deductive/inductive analytic approach. The presentation of regional-level data about gender disparities resulted in heightened awareness and stakeholder buy-in and decision-making related to women’s health-focused QI. Interviews revealed that stakeholders were familiar with QI, with regional and facility leaders aware of inter-disciplinary committees and efforts to foster organizational change, including PCMH transformation. These efforts did not typically focus on women’s health, though some informal efforts had been undertaken. Barriers to engaging in QI included lack of communication across clinical service lines, fluidity in staffing, and lack of protected time. Inter-professional, multilevel stakeholders need to be engaged in implementation early, with data and discussion that convey the importance and relevance of a new initiative. Stakeholder perspectives on institutional norms (e.g., gender norms) and readiness for population-specific QI are useful drivers of clinical initiatives designed to transform care for clinical subpopulations.


Stakeholder engagement Implementation Evidence-based quality improvement women’s health Primary care Veterans health administration 



We would like to thank Dr. Susan Frayne for her WH-PBRN leadership and Ms. Diane Carney for her management of the WH-PBRN; Ms. Selene Mak for data analysis support; Dr. Amy Cohen for editorial input; and Ms. Kristina Oishi and Ms. Anneka Oishi for logistics support. We would also like to thank Ms. Angela Cohen for her overall management of the Women’s Health CREATE, and our Executive Steering Committee members and Women Veteran Council members for their valuable contributions to this project.

Compliance with ethical standards

Study funding sources

Support for this research was provided by VA HSR&D Service, Veterans Health Administration through the CREATE initiative (CRE 12-026) and the VA HSR&D Center for the Study of Healthcare Innovation, Implementation, and Policy (CIN 13-417). Development and implementation of the EBQI intervention is being supported through annual Memoranda of Understanding with funding from VA Women’s Health Services (WHS) in the Office of Patient Care Services, Veterans Health Administration. Dr. Yano’s effort was funded by a VA HSR&D Senior Research Career Scientist Award (Project #RCS 05-195).

Conflicts of interest

The authors declare that they have no conflicts of interest.

Reporting and control of data

The findings reported have not been previously published and the manuscript is not being simultaneously submitted elsewhere. The data has not been previously reported. The authors have full control of all primary data and agree to allow the journal to review their data if requested. The views expressed within this study are solely those of the authors and do not necessarily represent the views of the Department of Veterans Affairs or the U.S. Government.

Ethics approval and consent to participate

This study protocol has been approved by the VA Greater Los Angeles Healthcare System IRB-B Subcommittee (IRB Chair, Paul Shekelle, MD, PhD) (approval number 2013-040589). Individual study components, including key stakeholder interviews, were reviewed and approved by IRB-B as serial project modifications to an umbrella IRB application under the study title. For each study component, we obtained IRB approval for waivers of documentation of consent for all data collections. Consent was provided verbally and recorded in the case of interviews. The VA Central IRB (Washington DC) also determined that the participating VA medical centers (n = 12) were not engaged in research and that the intervention (or implementation strategy, evidence-based quality improvement [EBQI]) was QI. Documentation of IRB approval (and in VA settings, Research and Development Committee approval as well) and Central IRB determination of non-research for the implementation strategy are available on request. The UCLA Institutional Review Board also reviewed and approved the key stakeholder interview component. No individual person’s data in any form are or will be included in this or subsequent manuscripts. Only aggregated data will be reported. For key stakeholder data, non-attributed or identifiable quotes may be reported in addition to thematic summaries.

Statement on welfare of animals

No animals are or will be involved in this study protocol. Similarly, use of animal or human data or tissue is not applicable to this study protocol.

Helsinki statement

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards.


  1. 1.
    National Center for Veterans Analysis and Statistics. Projected Veteran population 2013 to 2043. October 13, 2014. Available at: Accessibility verified December 31, 2016.
  2. 2.
    de Kleijn, M., Lagro-Janssen, A. L., Canelo, I., & Yano, E. M. (2015). Creating a roadmap for delivering gender-sensitive comprehensive care for women Veterans: results of a national expert panel. Medical Care, 53, S156–S164.CrossRefGoogle Scholar
  3. 3.
    Rosland, A. M., Nelson, K., Sun, H., et al. (2013). The patient-centered medical home in the Veterans Health Administration. The American Journal of Managed Care, 19, e263–e272.PubMedGoogle Scholar
  4. 4.
    Yano, E. M., Bair, M. J., Carrasquillo, O., Krein, S. L., & Rubenstein, L. V. (2014). Patient Aligned Care Teams (PACT): VA’s journey to implement patient-centered medical homes. Journal of General Internal Medicine, 29, S547–S549.CrossRefPubMedGoogle Scholar
  5. 5.
    O’Toole, T. P., Johnson, E. E., Aiello, R., Kane, V., & Pape, L. (2016). Tailoring care to vulnerable populations by incorporating social determinants of health: the Veterans Health Administration’s “Homeless Patient Aligned Care Team” program. Preventing Chronic Disease, 13, E44.PubMedPubMedCentralGoogle Scholar
  6. 6.
    Hernandez, S. E., Taylor, L., Grembowski, D., et al. (2016). A first look at PCMH implementation for minority Veterans: room for improvement. Medical Care, 54, 253–261.CrossRefPubMedGoogle Scholar
  7. 7.
    Engel, P. A., Spencer, J., Paul, T., & Boardman, J. B. (2016). The geriatrics in primary care demonstration: integrating comprehensive geriatric care into the medical home: preliminary data. Journal of the American Geriatrics Society, 64, 875–879.CrossRefPubMedGoogle Scholar
  8. 8.
    Yano, E. M., Haskell, S., & Hayes, P. (2014). Delivery of gender-sensitive comprehensive primary care to women veterans: implications for VA Patient Aligned Care Teams. Journal of General Internal Medicine, 29, S703–S707.CrossRefPubMedGoogle Scholar
  9. 9.
    National Center for Veterans Analysis and Statistics. Profile of women Veterans: 2015. December, 2016. Available at: Accessibility verified December 31, 2016.
  10. 10.
    Wilson LC. The prevalence of military sexual trauma: a meta-analysis. Trauma Viol Abuse, first published date: December 16, 2016.Google Scholar
  11. 11.
    Mattocks, K. M. (2015). Care coordination for women veterans: bridging the gap between systems of care. Medical Care, 53, S8–S9.CrossRefPubMedGoogle Scholar
  12. 12.
    Chuang E, Brunner J, Mak S, et al. Challenges with implementing a patient-centered medical home model for women Veterans. Women’s Health Issues, 2017.Google Scholar
  13. 13.
    Hamilton, A. B., Frayne, S. M., Cordasco, K. M., & Washington, D. L. (2013). Factors related to attrition from VA healthcare use: findings from the National Survey of women Veterans. Journal of General Internal Medicine, 28, S510–S516.CrossRefPubMedGoogle Scholar
  14. 14.
    Kirchner, J. E., Parker, L. E., Bonner, L. M., Fickel, J. J., Yano, E. M., & Ritchie, M. J. (2012). Roles of managers, frontline staff and local champions, in implementing quality improvement: stakeholders’ perspectives. Journal of Evaluation in Clinical Practice, 18, 63–69.CrossRefPubMedGoogle Scholar
  15. 15.
    Yano, E. M., Green, L. W., Glanz, K., et al. (2012). Implementation and spread of interventions into the multilevel context of routine practice and policy: implications for the cancer care continuum. Journal of the National Cancer Institute. Monographs, 2012, 86–99.CrossRefPubMedPubMedCentralGoogle Scholar
  16. 16.
    Solimeo, S. L., Stewart, K. R., Stewart, G. L., & Rosenthal, G. (2014). Implementing a patient centered medical home in the Veterans health administration: perspectives of primary care providers. Healthc (Amst), 2, 245–250.CrossRefGoogle Scholar
  17. 17.
    Rubenstein, L. V., Chaney, E. F., Ober, S., et al. (2010). Using evidence-based quality improvement methods for translating depression collaborative care research into practice. Families, Systems & Health, 28, 91–113.CrossRefGoogle Scholar
  18. 18.
    Curran, G. M., Pyne, J., Fortney, J. C., et al. (2011). Development and implementation of collaborative care for depression in HIV clinics. AIDS Care, 23, 1626–1636.CrossRefPubMedGoogle Scholar
  19. 19.
    Fortney, J., Enderle, M., McDougall, S., et al. (2012). Implementation outcomes of evidence-based quality improvement for depression in VA community based outpatient clinics. Implementation Science, 11, 30.CrossRefGoogle Scholar
  20. 20.
    Hamilton, A. B., Cohen, A. N., Glover, D. L., et al. (2013). Implementation of evidence-based employment services in specialty mental health. Health Services Research, 48, 2224–2244.CrossRefPubMedPubMedCentralGoogle Scholar
  21. 21.
    Rubenstein, L. V., Stockdale, S. E., Sapir, N., et al. (2014). A patient-centered primary care practice approach using evidence-based quality improvement: rationale, methods, and early assessment of implementation. Journal of General Internal Medicine, 29, S589–S597.CrossRefPubMedGoogle Scholar
  22. 22.
    Yoon, J., Chow, A., & Rubenstein, L. V. (2016). Impact of medical home implementation through evidence-based quality improvement on utilization and costs. Medical Care, 54, 118–125.CrossRefPubMedGoogle Scholar
  23. 23.
    Nadeem, E., Olin, S. S., Hill, L. C., Hoagwood, K. E., & Horwitz, S. M. (2013). Understanding the components of quality improvement collaboratives: a systematic literature review. The Milbank Quarterly, 91, 354–394.CrossRefPubMedPubMedCentralGoogle Scholar
  24. 24.
    Yano, E. M., Darling, J. E., Hamilton, A. B., et al. (2016). Cluster randomized trial of a multilevel evidence-based quality improvement approach to tailoring VA Patient Aligned Care Teams to the needs of women Veterans. Implementation Science, 11, 101.CrossRefPubMedPubMedCentralGoogle Scholar
  25. 25.
    Frayne, S. M., Carney, D. V., Bastian, L., et al. (2013). The VA Women’s Health Practice-Based Research Network: amplifying women veterans’ voices in VA research. Journal of General Internal Medicine, 28, S504–S509.CrossRefPubMedGoogle Scholar
  26. 26.
    Kwan, B. M., Sills, M. R., Graham, D., et al. (2016). Stakeholder engagement in a patient-reported outcomes (PRO) measure implementation: a report from the SAFTINet Practice-Based Research Network (PBRN). Journal of American Board of Family Medicine, 29, 102–115.CrossRefGoogle Scholar
  27. 27.
    Curran, G. M., Bauer, M., Mittman, B., Pyne, J. M., & Stetler, C. (2012). Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Medical Care, 50, 217–226.CrossRefPubMedPubMedCentralGoogle Scholar
  28. 28.
    Stetler, C. B., Legro, M. W., Wallace, C. M., et al. (2006). The role of formative evaluation in implementation research and the QUERI experience. Journal of General Internal Medicine, 21, S1–S8.CrossRefPubMedPubMedCentralGoogle Scholar
  29. 29.
    Rubenstein, L. V., Danz, M. S., Crain, A. L., Glasgow, R. E., Whitebird, R. R., & Solberg, L. I. (2014). Assessing organizational readiness for depression care quality improvement: relative commitment and implementation capability. Implementation Science, 9, 173.CrossRefPubMedPubMedCentralGoogle Scholar
  30. 30.
    Concannon, T. W., Meissner, P., Grunbaum, J. A., et al. (2012). A new taxonomy for stakeholder engagement in patient-centered outcomes research. Journal of General Internal Medicine, 27, 985–991.CrossRefPubMedPubMedCentralGoogle Scholar
  31. 31.
    Patton, M. Q. (2002). Qualitative Research and Evaluation Methods (3rd ed.). Thousand Oaks, CA: Sage.Google Scholar
  32. 32.
    Averill, J. B. (2002). Matrix analysis as a complementary analytic strategy in qualitative inquiry. Qualitative Health Research, 12, 855–866.CrossRefPubMedGoogle Scholar
  33. 33.
    Miles, M. B., & Huberman, A. M. (1994). Qualitative Data Analysis: an Expanded Sourcebook. Thousand Oaks, CA: Sage.Google Scholar
  34. 34.
    Scoville, R., & Little, K. (2014). Comparing Lean and Quality Improvement. IHI White Paper. Cambridge, Massachusetts: Institute for Healthcare Improvement (Available at Scholar
  35. 35.
    Smith, J. L., Williams Jr., J. W., Owen, R. R., Rubenstein, L. V., & Chaney, E. (2008). Developing a national dissemination plan for collaborative care for depression: QUERI series. Implementation Science, 3, 59.CrossRefPubMedPubMedCentralGoogle Scholar
  36. 36.
    McCormack, B., Rycroft-Malone, J., Decorby, K., et al. (2013). A realist review of interventions and strategies to promote evidence-informed healthcare: a focus on change agency. Implementation Science, 8, 107.CrossRefPubMedPubMedCentralGoogle Scholar
  37. 37.
    Mendel, P., Ngo, V. K., Dixon, E., et al. (2011). Partnered evaluation of a community engagement intervention: use of a kickoff conference in a randomized trial for depression care improvement in underserved communities. Ethnicity & Disease, 21, S1 78-88.Google Scholar
  38. 38.
    Chen, Q., Shin, M. H., Chan, J. A., et al. (2016). Partnering with VA stakeholders to develop a comprehensive patient safety data display: lessons learned from the field. American Journal of Medical Quality, 31, 178–186.CrossRefPubMedGoogle Scholar
  39. 39.
    Salyers, M. P., Godfrey, J. L., McGuire, A. B., Gearhart, T., Rollins, A. L., & Boyle, C. (2009). Implementing the illness management and recovery program for consumers with severe mental illness. Psychiatric Services, 60, 483–490.CrossRefPubMedGoogle Scholar
  40. 40.
    Ahmad, R., Kyratsis, Y., & Holmes, A. (2012). When the user is not the chooser: learning from stakeholder involvement in technology adoption decisions in infection control. The Journal of Hospital Infection, 81, 163–168.CrossRefPubMedGoogle Scholar
  41. 41.
    Berry, C. A., Mijanovich, T., Albert, S., et al. (2013). Patient-centered medical home among small urban practices serving low-income and disadvantaged patients. Annals of Family Medicine, 11, S82–S89.CrossRefPubMedPubMedCentralGoogle Scholar
  42. 42.
    Martinson, B. C., Nelson, D., Hagel-Campbell, E., et al. (2016). Initial results from the Survey of Organizational Research Climates (SOuRCe) in the U.S. Department of Veterans Affairs Healthcare System. PloS One, 11, e0151571.CrossRefPubMedPubMedCentralGoogle Scholar
  43. 43.
    MacGregor, C., Hamilton, A. B., Oishi, S. M., & Yano, E. M. (2011). Description, development, and philosophies of mental health service delivery for female veterans in the VA: a qualitative study. Women’s Health Issues, 21, S138–S144.CrossRefPubMedGoogle Scholar
  44. 44.
    Tannenbaum, C., Greaves, L., & Graham, I. D. (2016). Why sex and gender matter in implementation research. BMC Medical Research Methodology, 16, 145.CrossRefPubMedPubMedCentralGoogle Scholar
  45. 45.
    Lukas, C. V., Holmes, S. K., Cohen, A. B., et al. (2007). Transformational change in health care systems: an organizational model. Health Care Management Review, 32, 309–320.CrossRefPubMedGoogle Scholar
  46. 46.
    Mold, J. W., & Peterson, K. A. (2005). Primary care practice-based research networks: working at the interface between research and quality improvement. Annals of Family Medicine, 3, S12–S20.CrossRefPubMedPubMedCentralGoogle Scholar
  47. 47.
    Jeffs, L. P., Lo, J., Beswick, S., & Campbell, H. (2013). Implementing an organization-wide quality improvement initiative: insights from project leads, managers, and frontline nurses. Nursing Administration Quarterly, 37, 222–230.CrossRefPubMedGoogle Scholar
  48. 48.
    Johnson Faherty, L., Mate, K. S., & Moses, J. M. (2016). Leveraging trainees to improve quality and safety at the point of care: three models for engagement. Academic Medicine, 91, 503–509.CrossRefPubMedGoogle Scholar
  49. 49.
    Spears, W., Tsoh, J. Y., Potter, M. B., et al. (2014). Use of community engagement strategies to increase research participation in practice-based research networks (PBRNs). Journal of American Board of Family Medicine, 27, 763–771.CrossRefGoogle Scholar
  50. 50.
    Irvine, A. (2011). Duration, dominance and depth in telephone and face-to-face interviews: a comparative exploration. Intl J Qual Methods., 10, 202–220.CrossRefGoogle Scholar
  51. 51.
    Brandrud, A. S., Schreiner, A., Hjortdahl, P., Helljesen, G. S., Nyen, B., & Nelson, E. C. (2011). Three success factors for continual improvement in healthcare: an analysis of the reports of improvement team members. BMJ Quality and Safety, 20, 251–259.CrossRefPubMedGoogle Scholar
  52. 52.
    Kaplan, H. C., Brady, P. W., Dritz, M. C., et al. (2010). The influence of context on quality improvement success in health care: a systematic review of the literature. The Milbank Quarterly, 88, 500–559.CrossRefPubMedPubMedCentralGoogle Scholar
  53. 53.
    Franx, G., Dixon, L., Wensing, M., & Pincus, H. (2013). Implementation strategies for collaborative primary care-mental health models. Current Opinion in Psychiatry, 26, 502–510.CrossRefPubMedGoogle Scholar

Copyright information

© Society of Behavioral Medicine (outside the US) 2017

Authors and Affiliations

  1. 1.VA HSR&D Center for the Study of Healthcare Innovation, Implementation, and PolicyVA Greater Los Angeles Healthcare SystemLos AngelesUSA
  2. 2.Department of Psychiatry and Biobehavioral SciencesUCLA Geffen School of MedicineLos AngelesUSA
  3. 3.Department of Health Policy & ManagementUCLA Fielding School of Public HealthLos AngelesUSA
  4. 4.Pitzer College Psychology Field GroupClaremontUSA
  5. 5.RAND HealthSanta MonicaUSA
  6. 6.Department of MedicineUCLA Geffen School of MedicineLos AngelesUSA

Personalised recommendations