The Veterans Health Administration (VHA) has undertaken primary care transformation based on patient-centered medical home (PCMH) tenets. VHA PCMH models are designed for the predominantly male Veteran population, and require tailoring to meet women Veterans’ needs. We used evidence-based quality improvement (EBQI), a stakeholder-driven implementation strategy, in a cluster randomized controlled trial across 12 sites (eight EBQI, four control) that are members of a Practice-Based Research Network. EBQI involves engaging multilevel, inter-professional leaders and staff as stakeholders in reviewing evidence and setting QI priorities. The goal of this analysis was to examine processes of engaging stakeholders in early implementation of EBQI to tailor VHA’s medical home for women. Four inter-professional regional stakeholder planning meetings were conducted; these meetings engaged stakeholders by providing regional data about gender disparities in Veterans’ care experiences. Subsequent to each meeting, qualitative interviews were conducted with 87 key stakeholders (leaders and staff). Stakeholders were asked to describe QI efforts and the use of data to change aspects of care, including women’s health care. Interview transcripts were summarized and coded using a hybrid deductive/inductive analytic approach. The presentation of regional-level data about gender disparities resulted in heightened awareness and stakeholder buy-in and decision-making related to women’s health-focused QI. Interviews revealed that stakeholders were familiar with QI, with regional and facility leaders aware of inter-disciplinary committees and efforts to foster organizational change, including PCMH transformation. These efforts did not typically focus on women’s health, though some informal efforts had been undertaken. Barriers to engaging in QI included lack of communication across clinical service lines, fluidity in staffing, and lack of protected time. Inter-professional, multilevel stakeholders need to be engaged in implementation early, with data and discussion that convey the importance and relevance of a new initiative. Stakeholder perspectives on institutional norms (e.g., gender norms) and readiness for population-specific QI are useful drivers of clinical initiatives designed to transform care for clinical subpopulations.
This is a preview of subscription content, access via your institution.
National Center for Veterans Analysis and Statistics. Projected Veteran population 2013 to 2043. October 13, 2014. Available at: http://www.va.gov/vetdata/docs/quickfacts/population_slideshow.pdf. Accessibility verified December 31, 2016.
de Kleijn, M., Lagro-Janssen, A. L., Canelo, I., & Yano, E. M. (2015). Creating a roadmap for delivering gender-sensitive comprehensive care for women Veterans: results of a national expert panel. Medical Care, 53, S156–S164.
Rosland, A. M., Nelson, K., Sun, H., et al. (2013). The patient-centered medical home in the Veterans Health Administration. The American Journal of Managed Care, 19, e263–e272.
Yano, E. M., Bair, M. J., Carrasquillo, O., Krein, S. L., & Rubenstein, L. V. (2014). Patient Aligned Care Teams (PACT): VA’s journey to implement patient-centered medical homes. Journal of General Internal Medicine, 29, S547–S549.
O’Toole, T. P., Johnson, E. E., Aiello, R., Kane, V., & Pape, L. (2016). Tailoring care to vulnerable populations by incorporating social determinants of health: the Veterans Health Administration’s “Homeless Patient Aligned Care Team” program. Preventing Chronic Disease, 13, E44.
Hernandez, S. E., Taylor, L., Grembowski, D., et al. (2016). A first look at PCMH implementation for minority Veterans: room for improvement. Medical Care, 54, 253–261.
Engel, P. A., Spencer, J., Paul, T., & Boardman, J. B. (2016). The geriatrics in primary care demonstration: integrating comprehensive geriatric care into the medical home: preliminary data. Journal of the American Geriatrics Society, 64, 875–879.
Yano, E. M., Haskell, S., & Hayes, P. (2014). Delivery of gender-sensitive comprehensive primary care to women veterans: implications for VA Patient Aligned Care Teams. Journal of General Internal Medicine, 29, S703–S707.
National Center for Veterans Analysis and Statistics. Profile of women Veterans: 2015. December, 2016. Available at: https://www.va.gov/vetdata/docs/SpecialReports/Women_Veterans_Profile_12_22_2016.pdf. Accessibility verified December 31, 2016.
Wilson LC. The prevalence of military sexual trauma: a meta-analysis. Trauma Viol Abuse, first published date: December 16, 2016.
Mattocks, K. M. (2015). Care coordination for women veterans: bridging the gap between systems of care. Medical Care, 53, S8–S9.
Chuang E, Brunner J, Mak S, et al. Challenges with implementing a patient-centered medical home model for women Veterans. Women’s Health Issues, 2017.
Hamilton, A. B., Frayne, S. M., Cordasco, K. M., & Washington, D. L. (2013). Factors related to attrition from VA healthcare use: findings from the National Survey of women Veterans. Journal of General Internal Medicine, 28, S510–S516.
Kirchner, J. E., Parker, L. E., Bonner, L. M., Fickel, J. J., Yano, E. M., & Ritchie, M. J. (2012). Roles of managers, frontline staff and local champions, in implementing quality improvement: stakeholders’ perspectives. Journal of Evaluation in Clinical Practice, 18, 63–69.
Yano, E. M., Green, L. W., Glanz, K., et al. (2012). Implementation and spread of interventions into the multilevel context of routine practice and policy: implications for the cancer care continuum. Journal of the National Cancer Institute. Monographs, 2012, 86–99.
Solimeo, S. L., Stewart, K. R., Stewart, G. L., & Rosenthal, G. (2014). Implementing a patient centered medical home in the Veterans health administration: perspectives of primary care providers. Healthc (Amst), 2, 245–250.
Rubenstein, L. V., Chaney, E. F., Ober, S., et al. (2010). Using evidence-based quality improvement methods for translating depression collaborative care research into practice. Families, Systems & Health, 28, 91–113.
Curran, G. M., Pyne, J., Fortney, J. C., et al. (2011). Development and implementation of collaborative care for depression in HIV clinics. AIDS Care, 23, 1626–1636.
Fortney, J., Enderle, M., McDougall, S., et al. (2012). Implementation outcomes of evidence-based quality improvement for depression in VA community based outpatient clinics. Implementation Science, 11, 30.
Hamilton, A. B., Cohen, A. N., Glover, D. L., et al. (2013). Implementation of evidence-based employment services in specialty mental health. Health Services Research, 48, 2224–2244.
Rubenstein, L. V., Stockdale, S. E., Sapir, N., et al. (2014). A patient-centered primary care practice approach using evidence-based quality improvement: rationale, methods, and early assessment of implementation. Journal of General Internal Medicine, 29, S589–S597.
Yoon, J., Chow, A., & Rubenstein, L. V. (2016). Impact of medical home implementation through evidence-based quality improvement on utilization and costs. Medical Care, 54, 118–125.
Nadeem, E., Olin, S. S., Hill, L. C., Hoagwood, K. E., & Horwitz, S. M. (2013). Understanding the components of quality improvement collaboratives: a systematic literature review. The Milbank Quarterly, 91, 354–394.
Yano, E. M., Darling, J. E., Hamilton, A. B., et al. (2016). Cluster randomized trial of a multilevel evidence-based quality improvement approach to tailoring VA Patient Aligned Care Teams to the needs of women Veterans. Implementation Science, 11, 101.
Frayne, S. M., Carney, D. V., Bastian, L., et al. (2013). The VA Women’s Health Practice-Based Research Network: amplifying women veterans’ voices in VA research. Journal of General Internal Medicine, 28, S504–S509.
Kwan, B. M., Sills, M. R., Graham, D., et al. (2016). Stakeholder engagement in a patient-reported outcomes (PRO) measure implementation: a report from the SAFTINet Practice-Based Research Network (PBRN). Journal of American Board of Family Medicine, 29, 102–115.
Curran, G. M., Bauer, M., Mittman, B., Pyne, J. M., & Stetler, C. (2012). Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Medical Care, 50, 217–226.
Stetler, C. B., Legro, M. W., Wallace, C. M., et al. (2006). The role of formative evaluation in implementation research and the QUERI experience. Journal of General Internal Medicine, 21, S1–S8.
Rubenstein, L. V., Danz, M. S., Crain, A. L., Glasgow, R. E., Whitebird, R. R., & Solberg, L. I. (2014). Assessing organizational readiness for depression care quality improvement: relative commitment and implementation capability. Implementation Science, 9, 173.
Concannon, T. W., Meissner, P., Grunbaum, J. A., et al. (2012). A new taxonomy for stakeholder engagement in patient-centered outcomes research. Journal of General Internal Medicine, 27, 985–991.
Patton, M. Q. (2002). Qualitative Research and Evaluation Methods (3rd ed.). Thousand Oaks, CA: Sage.
Averill, J. B. (2002). Matrix analysis as a complementary analytic strategy in qualitative inquiry. Qualitative Health Research, 12, 855–866.
Miles, M. B., & Huberman, A. M. (1994). Qualitative Data Analysis: an Expanded Sourcebook. Thousand Oaks, CA: Sage.
Scoville, R., & Little, K. (2014). Comparing Lean and Quality Improvement. IHI White Paper. Cambridge, Massachusetts: Institute for Healthcare Improvement (Available at ihi.org).
Smith, J. L., Williams Jr., J. W., Owen, R. R., Rubenstein, L. V., & Chaney, E. (2008). Developing a national dissemination plan for collaborative care for depression: QUERI series. Implementation Science, 3, 59.
McCormack, B., Rycroft-Malone, J., Decorby, K., et al. (2013). A realist review of interventions and strategies to promote evidence-informed healthcare: a focus on change agency. Implementation Science, 8, 107.
Mendel, P., Ngo, V. K., Dixon, E., et al. (2011). Partnered evaluation of a community engagement intervention: use of a kickoff conference in a randomized trial for depression care improvement in underserved communities. Ethnicity & Disease, 21, S1 78-88.
Chen, Q., Shin, M. H., Chan, J. A., et al. (2016). Partnering with VA stakeholders to develop a comprehensive patient safety data display: lessons learned from the field. American Journal of Medical Quality, 31, 178–186.
Salyers, M. P., Godfrey, J. L., McGuire, A. B., Gearhart, T., Rollins, A. L., & Boyle, C. (2009). Implementing the illness management and recovery program for consumers with severe mental illness. Psychiatric Services, 60, 483–490.
Ahmad, R., Kyratsis, Y., & Holmes, A. (2012). When the user is not the chooser: learning from stakeholder involvement in technology adoption decisions in infection control. The Journal of Hospital Infection, 81, 163–168.
Berry, C. A., Mijanovich, T., Albert, S., et al. (2013). Patient-centered medical home among small urban practices serving low-income and disadvantaged patients. Annals of Family Medicine, 11, S82–S89.
Martinson, B. C., Nelson, D., Hagel-Campbell, E., et al. (2016). Initial results from the Survey of Organizational Research Climates (SOuRCe) in the U.S. Department of Veterans Affairs Healthcare System. PloS One, 11, e0151571.
MacGregor, C., Hamilton, A. B., Oishi, S. M., & Yano, E. M. (2011). Description, development, and philosophies of mental health service delivery for female veterans in the VA: a qualitative study. Women’s Health Issues, 21, S138–S144.
Tannenbaum, C., Greaves, L., & Graham, I. D. (2016). Why sex and gender matter in implementation research. BMC Medical Research Methodology, 16, 145.
Lukas, C. V., Holmes, S. K., Cohen, A. B., et al. (2007). Transformational change in health care systems: an organizational model. Health Care Management Review, 32, 309–320.
Mold, J. W., & Peterson, K. A. (2005). Primary care practice-based research networks: working at the interface between research and quality improvement. Annals of Family Medicine, 3, S12–S20.
Jeffs, L. P., Lo, J., Beswick, S., & Campbell, H. (2013). Implementing an organization-wide quality improvement initiative: insights from project leads, managers, and frontline nurses. Nursing Administration Quarterly, 37, 222–230.
Johnson Faherty, L., Mate, K. S., & Moses, J. M. (2016). Leveraging trainees to improve quality and safety at the point of care: three models for engagement. Academic Medicine, 91, 503–509.
Spears, W., Tsoh, J. Y., Potter, M. B., et al. (2014). Use of community engagement strategies to increase research participation in practice-based research networks (PBRNs). Journal of American Board of Family Medicine, 27, 763–771.
Irvine, A. (2011). Duration, dominance and depth in telephone and face-to-face interviews: a comparative exploration. Intl J Qual Methods., 10, 202–220.
Brandrud, A. S., Schreiner, A., Hjortdahl, P., Helljesen, G. S., Nyen, B., & Nelson, E. C. (2011). Three success factors for continual improvement in healthcare: an analysis of the reports of improvement team members. BMJ Quality and Safety, 20, 251–259.
Kaplan, H. C., Brady, P. W., Dritz, M. C., et al. (2010). The influence of context on quality improvement success in health care: a systematic review of the literature. The Milbank Quarterly, 88, 500–559.
Franx, G., Dixon, L., Wensing, M., & Pincus, H. (2013). Implementation strategies for collaborative primary care-mental health models. Current Opinion in Psychiatry, 26, 502–510.
We would like to thank Dr. Susan Frayne for her WH-PBRN leadership and Ms. Diane Carney for her management of the WH-PBRN; Ms. Selene Mak for data analysis support; Dr. Amy Cohen for editorial input; and Ms. Kristina Oishi and Ms. Anneka Oishi for logistics support. We would also like to thank Ms. Angela Cohen for her overall management of the Women’s Health CREATE, and our Executive Steering Committee members and Women Veteran Council members for their valuable contributions to this project.
Study funding sources
Support for this research was provided by VA HSR&D Service, Veterans Health Administration through the CREATE initiative (CRE 12-026) and the VA HSR&D Center for the Study of Healthcare Innovation, Implementation, and Policy (CIN 13-417). Development and implementation of the EBQI intervention is being supported through annual Memoranda of Understanding with funding from VA Women’s Health Services (WHS) in the Office of Patient Care Services, Veterans Health Administration. Dr. Yano’s effort was funded by a VA HSR&D Senior Research Career Scientist Award (Project #RCS 05-195).
Conflicts of interest
The authors declare that they have no conflicts of interest.
Reporting and control of data
The findings reported have not been previously published and the manuscript is not being simultaneously submitted elsewhere. The data has not been previously reported. The authors have full control of all primary data and agree to allow the journal to review their data if requested. The views expressed within this study are solely those of the authors and do not necessarily represent the views of the Department of Veterans Affairs or the U.S. Government.
Ethics approval and consent to participate
This study protocol has been approved by the VA Greater Los Angeles Healthcare System IRB-B Subcommittee (IRB Chair, Paul Shekelle, MD, PhD) (approval number 2013-040589). Individual study components, including key stakeholder interviews, were reviewed and approved by IRB-B as serial project modifications to an umbrella IRB application under the study title. For each study component, we obtained IRB approval for waivers of documentation of consent for all data collections. Consent was provided verbally and recorded in the case of interviews. The VA Central IRB (Washington DC) also determined that the participating VA medical centers (n = 12) were not engaged in research and that the intervention (or implementation strategy, evidence-based quality improvement [EBQI]) was QI. Documentation of IRB approval (and in VA settings, Research and Development Committee approval as well) and Central IRB determination of non-research for the implementation strategy are available on request. The UCLA Institutional Review Board also reviewed and approved the key stakeholder interview component. No individual person’s data in any form are or will be included in this or subsequent manuscripts. Only aggregated data will be reported. For key stakeholder data, non-attributed or identifiable quotes may be reported in addition to thematic summaries.
Statement on welfare of animals
No animals are or will be involved in this study protocol. Similarly, use of animal or human data or tissue is not applicable to this study protocol.
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards.
Research: Researchers interested in engaging organizational stakeholders in implementation and improvement efforts could consider tailoring data reports to subpopulations of interest and using that data to educate, raise awareness, and promote buy-in. Researchers could also make efforts to align new initiatives with institutional norms and existing practices.
Practice: Practitioners interested in engaging in implementation and improvement efforts could, with institutional support, (1) seek out local researchers who are conducting studies on topics of interest to learn about opportunities for involvement, (2) seek out inter-disciplinary improvement initiatives to become involved in them or use them as a model for new initiatives, or (3) identify relevant training opportunities that might create linkages to individuals who are conducting interesting projects.
Policy: Policymakers interested in engaging in implementation and improvement efforts could identify ways to obtain data—analyzed and available in approachable ways—that would support decision-making around care targets that may warrant policy changes or innovations, and they could also partner with researchers to guide the design of studies that would generate policy-relevant results.
About this article
Cite this article
Hamilton, A.B., Brunner, J., Cain, C. et al. Engaging multilevel stakeholders in an implementation trial of evidence-based quality improvement in VA women’s health primary care. Behav. Med. Pract. Policy Res. 7, 478–485 (2017). https://doi.org/10.1007/s13142-017-0501-5
- Stakeholder engagement
- Evidence-based quality improvement
- women’s health
- Primary care
- Veterans health administration