This study aimed to validate the factor structure of the expanded Primary Care Behavioral Health Provider Adherence Questionnaire (PPAQ-2), which is designed to assess provider fidelity to both the Primary Care Behavioral Health (PCBH) and collaborative care management (CCM) models of integrated primary care. Two-hundred fifty-three integrated care providers completed self-reports of professional background, perceptions of clinic integration and related practice barriers, and the PPAQ-2. Confirmatory factor analyses were conducted to assess the theorized factor structure and criterion validity was assessed through correlational analysis. Factor analyses demonstrated adequate fit with the data and acceptable to excellent composite reliabilities across five PCBH domains and five CCM domains. Validity was demonstrated by correlations between adherence scores and measures of clinic integration and barriers to fidelity. The PPAQ-2 is a psychometrically sound measure that can be used in future integrated care dismantling studies to identify provider behaviors that best predict patient outcomes.
This is a preview of subscription content, log in to check access.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
Tax calculation will be finalised during checkout.
Afthanorhan, W. M. A. B. W. (2013). A comparison of partial least square structural equation modeling (PLS-SEM) and covariance based structural equation modeling (CB-SEM) for confirmatory factor analysis. International Journal of Engineering Science and Innovation Technology,2, 198–205.
Atkins, D., Kilbourne, A. M., & Shulkin, D. (2017). Moving from discovery to system-wide change: The role of research in a learning health care system: Experience from three decades of health systems research in the Veterans Health Administration. Annual Review of Public Health,38, 467–487.
Awang, Z., Ahmad, J. H., & Zin, N. M. (2010). Modelling job satisfaction and work commitment among lecturers: a case of UITM kelantan. Journal of Statistical Modeling and Analytics,1, 45–59.
Bacon, L. D. (1999). Using LISREL and PLS to Measure Customer Satisfaction. Paper presented at the Sawtooth Software Conference Proceeding, La Jolla, CA.
Beehler, G. P., Funderburk, J. S., King, P. R., Wade, M., & Possemato, K. (2015). Using the Primary Care Behavioral Health Provider Adherence Questionnaire (PPAQ) to identify variation in practice patterns. Translational Behavioral Medicine,5, 384–392.
Beehler, G. P., Funderburk, J. S., Possemato, K., & Dollar, K. M. (2013). Psychometric assessment of the Primary Care Behavioral Health Provider Adherence Questionnaire (PPAQ). Translational Behavioral Medicine,3, 379–391.
Beehler, G. P., Funderburk, J. S., Possemato, K., & Vair, C. L. (2013). Developing a measure of provider adherence to improve the implementation of behavioral health services in primary care: A Delphi study. Implementation Science,8, 19.
Beehler, G. P., & Lilienthal, K. R. (2017). Provider perceptions of an integrated primary care quality improvement strategy: The PPAQ Toolkit. Psychological Services,14, 50–56.
Beehler, G. P., & Wray, L. O. (2012). Behavioral health providers’ perspectives of delivering behavioral health services in primary care: a qualitative analysis. BMC Health Services Research,12, 337. https://doi.org/10.1186/1472-6963-12-337.
Bellg, A. J., Borrelli, B., Resnick, B., Hecht, J., Minicucci, D. S., Ory, M., … Czajkowski, S. (2004). Enhancing treatment fidelity in health behavior change studies: Best practices and recommendations from the NIH Behavior Change Consortium. Health Psychology,23(5), 443–451. https://doi.org/10.1037/0278-6184.108.40.2063.
Bower, P., Gilbody, S., Richards, D., Fletcher, J., & Sutton, A. (2006). Collaborative care for depression in primary care: Making sense of a complex intervention: Systematic review and meta-regression. The British Journal of Psychiatry,189, 484–493.
Brawer, P. A., Brugh, A. M. T., Martielli, R. P., O’Connor, S. P., Mastnak, J., Scherrer, J. F., & Day, T. E. (2011). Enhancing entrance into PTSD treatment for post-deployment veterans through collaborative/integrative care. Translational Behavioral Medicine,1(4), 609–614.
Brawer, P. A., Martielli, R. P., Pye, P. L., Manwaring, J., & Tierney, A. (2010). St. Louis Initiative for Integrated Care Excellence (SLI2CE): Integrated collaborative care on a large scale model Families. Systems, & Health,28, 175–187.
Bryan, C. J., Corso, M. L., Corso, K. A., Morrow, C. E., Kanzler, K. E., & Ray-Sannerud, B. (2012). Severity of mental health impairment and trajectories of improvement in an integrated primary care clinic. Journal of Consulting and Clinical Psychology,80(3), 396–403.
Bryan, C. J., Morrow, C. E., & Appolonio, K. K. (2009). Impact of behavioral health consultant interventions on patient symptoms and functioning in an integrated family medicine clinic. Journal of Clinical Psychology,65, 281–293.
Butler, M., Kane, R. L., McAlpine, D., Kathol, R. G., Fu, S. S., Hagedorn, H., & Wilt, T. J. (2008). Integration of Mental Health/Substance Abuse and Primary Care No. 173. (AHRQ Publication No. 09-E003.). Rockville: Agency for Healthcare Research and Quality.
Carroll, C., Patterson, M., Wood, S., Booth, B., Rick, J., & Balain, S. (2007). A conceptual framework for implementation fidelity. Implementation Science,2, 40.
Chaney, E. F., Rubenstein, L. V., Liu, C. F., Yano, E. M., Bolkan, C., Lee, M., … Uman, J. (2011). Implementing collaborative care for depression treatment in primary care: A cluster randomized evaluation of a quality improvement practice design. Implementation Science,6, 121.
Cigrang, J. A., Rauch, S. A., Mintz, J., Brundige, A. R., Mitchell, J. A., Najera, E., … STRONG STAR Consortium. (2017). Moving effective treatment for posttraumatic stress disorder to primary care: A randomized controlled trial with active duty military. Families, Systems & Health,35, 450–462.
Fauth, J., & Tremblay, G. (2011). The integrated care evaluation project: Antioch University.
Fornell, C., & Larcker, D. F. (1981). Evaluating structural equation models with unobservable variables and measurement error. Journal of Marketing Research,18, 39–50.
Fortney, J., Enderle, M., McDougall, S., Clothier, J., Otero, J., Altman, L., & Curran, G. (2012). Implementation outcomes of evidence-based quality improvement for depression in VA community based outpatient clinics. Implementation Science,7, 30. https://doi.org/10.1186/1748-5908-7-30.
Gilbody, S., Bower, P., Fletcher, J., Richards, D., & Sutton, A. J. (2006). Collaborative care for depression: A cumulative meta-analysis and review of longer-term outcomes. Archive of Internal Medicine,166, 2314–2321.
Hasson, H. (2010). Systematic evaluation of implementation fidelity of complex interventions in health and social care. Implementation Science,5, 67. https://doi.org/10.1186/1748-5908-5-67.
Hrisos, S., Eccles, M. P., Francis, J. J., Dickinson, H. O., Kaner, E. F., Beyer, F., & Johnston, M. (2009). Are there valid proxy measures of clinical behaviour? A systematic review. Implementation Science,4, 37. https://doi.org/10.1186/1748-5908-4-37.
Hunter, C. L., & Goodie, J. L. (2010). Operational and clinical components for integrated-collaborative behavioral healthcare in the patient-centered medical home. Families, Systems & Health,28, 308–321. https://doi.org/10.1037/a0021761.
Hunter, C. L., Goodie, J. L., Oordt, M. S., & Dobmeyer, A. C. (2017). Integrated behavioral health in primary care: Step-by-step guidance for assessment and intervention (2nd ed.). Washington, DC: American Psychological Association.
Kearney, L. K., Dollar, K. M., Beehler, G. P., Goldstein, W. R., Grasso, J., Wray, L. O., & Pomerantz, A. S. (in press). Creation and implementation of a national interprofessional integrated primary care competency training program: Preliminary findings and lessons learned. Training and Education in Professional Psychology.
Kearney, L. K., Post, E. P., Zeiss, A., Goldstein, M. G., & Dundon, M. (2011). The role of mental and behavioral health in the application of the patient-centered medical home in the Department of Veterans Affairs. Translational Behavioral Medicine,1, 624–628.
Mowbray, C. T., Holter, M. C., Teague, G. B., & Bybee, D. (2003). Fidelity criteria: Development, measurement, and validation. American Journal of Evaluation,24, 315–340.
Netemeyer, R. G., Bearden, W. O., & Sharma, S. (2003). Scaling procedures: Issues and applications. Thousand Oaks, CA: Sage.
Newman, I., Lim, J., & Pineda, F. (2013). Content validity using mixed methods approach: Its application and development through the use of a Table of Specifications Methodology. Journal of Mixed Methods Research,7, 243–260.
Nunally, J. C., & Bernstien, I. H. (1994). Psychometric theory (3rd ed.). New York: Mcfgraw-Hill.
Oslin, D. W., Ross, J., Sayers, S., Murphy, J., Kane, V., & Katz, I. R. (2006). Screening, assessment, and management of depression in VA primary care clinics. The Behavioral Health Laboratory. Journal of General Internal Medicine,21, 46–50.
Oxman, T. E., Schulberg, H. C., Greenberg, R. L., Dietrich, A. J., Williams, J. W. J., Nutting, P. A., & Bruce, M. L. (2006). A fidelity measure for integrated management of depression in primary care. Medical Care,44, 1030–1037.
Possemato, K., Johnson, E. M., Beehler, G. P., Shepardson, R. L., King, P., Vair, C. L., … Wray, L. O. (2018). Patient outcomes associated with primary care behavioral health services: A systematic review. General Hospital Psychiatry,53, 1–11.
Ray-Sannerud, B. N., Dolan, D. C., Morrow, C. E., Corso, K. A., Kanzler, K. E., Corso, M. L., & Bryan, C. J. (2012). Longitudinal outcomes after brief behavioral health intervention in an integrated primary care clinic. Families, Systems & Health,30(1), 60–71. https://doi.org/10.1037/a0027029.
Reiter, J. T., Dobmeyer, A. C., & Hunter, C. L. (2018). The Primary Care Behavioral Health (PCBH) Model: An overview and operational definition. Journal of Clinical Psychology in Medical Settings,25, 109–126.
Robinson, P. J., & Reiter, J. T. (2016). Behavioral consultation and primary care: A guide to integrating services (2nd ed.). New York, NY: Springer.
Ross, J. T., TenHave, T., Eakin, A. C., Difillippo, S., & Oslin, D. W. (2008). A randomized controlled trial of a close monitoring program for minor depression and distress. Journal of General Internal Medicine,23, 1379–1385.
Rubenstein, L. V., Chaney, E. F., Ober, S., Felker, B., Sherman, S. E., Lanto, A., & Vivell, S. (2010). Using evidence-based quality improvement methods for translating depression collaborative care research into practice. Families, Systems, & Health,28, 91–113.
Rubenstein, L. V., Williams, J. W., Danz, M., Shekelle, P., Suttorp, M., & Johnsen, B. (2009). Determining key features of effective depression interventions. Washington, DC: Department of Veterans Affairs.
Shiner, B., Watts, B. V., Pomerantz, A., Groft, A., Scott, D., Street, B., & Young-Xu, Y. (2009). Access to what? An evaluation of the key ingredients to effective advanced mental health access at a VA Medical Center and its affiliated community-based outreach clinics. Military Medicine,174, 1024–1032.
Strosahl, K. (1998). Integrating behavioral health and primary care services: The primary mental health care model. In A. Blount (Ed.), Integrated primary care: The future of medical and mental health collaboration (pp. 139–166). New York: Norton.
VA Health Services Research and Development Service. (2008). Collaborative Care for Depression in the Primary Care Setting: A Primer on VA’s Translating Initiatives for Depression into Effective Solutions (TIDES) Project, Boston, MA.
Williams, J. W., Jr., Gerrity, M., Holsinger, T., Dobscha, S., Gaynes, B., & Dietrich, A. (2007). Systematic review of multifaceted interventions to improve depression care. General Hospital Psychiatry,29(2), 91–116. https://doi.org/10.1016/j.genhosppsych.2006.12.003.
Woltmann, E., Grogan-Kaylor, A., Perron, B., Georges, H., Kilbourne, A. M., & Bauer, M. S. (2012). Comparative effectiveness of collaborative chronic care models for mental health conditions across primary, specialty, and behavioral health care settings: Systematic review and meta-analysis. American Journal of Psychiatry,169, 790–804.
Wong, K. K. (2010). Handling small survey sample size and skewed dataset with partial least square path modelling (pp. 20–23). Vue: The Magazine of the Marketing Research and Intelligence Association.
Zanjani, F., Miller, B., Turiano, N., Ross, J., & Oslin, D. (2008). Effectiveness of telephone-based referral care management: A brief intervention to improve psychiatric treatment engagement. Psychiatric Services,59, 776–781.
This study was funded by the VA Center for Integrated Healthcare at the VA Western New York Healthcare System. The information provided in this study does not represent the views of the Department of Veterans Affairs or the United States Government.
Conflict of interest
Gregory P. Beehler, Jennifer S. Funderburk, Paul R. King, Kyle Possemato, John A. Maddoux, Wade R. Goldstein, Michael Wade have no conflict of interest.
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.
Human and Animal Rights and Informed Consent
All procedures followed were in accordance with the ethical standard of the responsible committee on human experimentation of the VA Western New York Healthcare System and with the Helsinki Declaration of 1975, as revised in 2000. As an anonymous provider survey, this study was deemed exempt from Institutional Review Board review.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Beehler, G.P., Funderburk, J.S., King, P.R. et al. Validation of an Expanded Measure of Integrated Care Provider Fidelity: PPAQ-2. J Clin Psychol Med Settings 27, 158–172 (2020). https://doi.org/10.1007/s10880-019-09628-0
- Guideline adherence
- Integrated primary care
- Mental health services
- Primary health care
- Program evaluation