Validation of an Expanded Measure of Integrated Care Provider Fidelity: PPAQ-2

  • Gregory P. BeehlerEmail author
  • Jennifer S. Funderburk
  • Paul R. King
  • Kyle Possemato
  • John A. Maddoux
  • Wade R. Goldstein
  • Michael Wade


This study aimed to validate the factor structure of the expanded Primary Care Behavioral Health Provider Adherence Questionnaire (PPAQ-2), which is designed to assess provider fidelity to both the Primary Care Behavioral Health (PCBH) and collaborative care management (CCM) models of integrated primary care. Two-hundred fifty-three integrated care providers completed self-reports of professional background, perceptions of clinic integration and related practice barriers, and the PPAQ-2. Confirmatory factor analyses were conducted to assess the theorized factor structure and criterion validity was assessed through correlational analysis. Factor analyses demonstrated adequate fit with the data and acceptable to excellent composite reliabilities across five PCBH domains and five CCM domains. Validity was demonstrated by correlations between adherence scores and measures of clinic integration and barriers to fidelity. The PPAQ-2 is a psychometrically sound measure that can be used in future integrated care dismantling studies to identify provider behaviors that best predict patient outcomes.


Guideline adherence Integrated primary care Mental health services Primary health care Program evaluation 



This study was funded by the VA Center for Integrated Healthcare at the VA Western New York Healthcare System. The information provided in this study does not represent the views of the Department of Veterans Affairs or the United States Government.

Compliance with Ethical Standards

Conflict of interest

Gregory P. Beehler, Jennifer S. Funderburk, Paul R. King, Kyle Possemato, John A. Maddoux, Wade R. Goldstein, Michael Wade have no conflict of interest.

Ethical Approval

 All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Human and Animal Rights and Informed Consent

All procedures followed were in accordance with the ethical standard of the responsible committee on human experimentation of the VA Western New York Healthcare System and with the Helsinki Declaration of 1975, as revised in 2000. As an anonymous provider survey, this study was deemed exempt from Institutional Review Board review.


  1. Afthanorhan, W. M. A. B. W. (2013). A comparison of partial least square structural equation modeling (PLS-SEM) and covariance based structural equation modeling (CB-SEM) for confirmatory factor analysis. International Journal of Engineering Science and Innovation Technology, 2, 198–205.Google Scholar
  2. Atkins, D., Kilbourne, A. M., & Shulkin, D. (2017). Moving from discovery to system-wide change: The role of research in a learning health care system: Experience from three decades of health systems research in the Veterans Health Administration. Annual Review of Public Health, 38, 467–487.CrossRefGoogle Scholar
  3. Awang, Z., Ahmad, J. H., & Zin, N. M. (2010). Modelling job satisfaction and work commitment among lecturers: a case of UITM kelantan. Journal of Statistical Modeling and Analytics, 1, 45–59.Google Scholar
  4. Bacon, L. D. (1999). Using LISREL and PLS to Measure Customer Satisfaction. Paper presented at the Sawtooth Software Conference Proceeding, La Jolla, CA.Google Scholar
  5. Beehler, G. P., Funderburk, J. S., King, P. R., Wade, M., & Possemato, K. (2015). Using the Primary Care Behavioral Health Provider Adherence Questionnaire (PPAQ) to identify variation in practice patterns. Translational Behavioral Medicine, 5, 384–392.CrossRefGoogle Scholar
  6. Beehler, G. P., Funderburk, J. S., Possemato, K., & Dollar, K. M. (2013). Psychometric assessment of the Primary Care Behavioral Health Provider Adherence Questionnaire (PPAQ). Translational Behavioral Medicine, 3, 379–391.CrossRefGoogle Scholar
  7. Beehler, G. P., Funderburk, J. S., Possemato, K., & Vair, C. L. (2013). Developing a measure of provider adherence to improve the implementation of behavioral health services in primary care: A Delphi study. Implementation Science, 8, 19.CrossRefGoogle Scholar
  8. Beehler, G. P., & Lilienthal, K. R. (2017). Provider perceptions of an integrated primary care quality improvement strategy: The PPAQ Toolkit. Psychological Services, 14, 50–56.CrossRefGoogle Scholar
  9. Beehler, G. P., & Wray, L. O. (2012). Behavioral health providers’ perspectives of delivering behavioral health services in primary care: a qualitative analysis. BMC Health Services Research, 12, 337. Scholar
  10. Bellg, A. J., Borrelli, B., Resnick, B., Hecht, J., Minicucci, D. S., Ory, M., … Czajkowski, S. (2004). Enhancing treatment fidelity in health behavior change studies: Best practices and recommendations from the NIH Behavior Change Consortium. Health Psychology, 23(5), 443–451. Scholar
  11. Bower, P., Gilbody, S., Richards, D., Fletcher, J., & Sutton, A. (2006). Collaborative care for depression in primary care: Making sense of a complex intervention: Systematic review and meta-regression. The British Journal of Psychiatry, 189, 484–493.CrossRefGoogle Scholar
  12. Brawer, P. A., Brugh, A. M. T., Martielli, R. P., O’Connor, S. P., Mastnak, J., Scherrer, J. F., & Day, T. E. (2011). Enhancing entrance into PTSD treatment for post-deployment veterans through collaborative/integrative care. Translational Behavioral Medicine, 1(4), 609–614.CrossRefGoogle Scholar
  13. Brawer, P. A., Martielli, R. P., Pye, P. L., Manwaring, J., & Tierney, A. (2010). St. Louis Initiative for Integrated Care Excellence (SLI2CE): Integrated collaborative care on a large scale model Families. Systems, & Health, 28, 175–187.CrossRefGoogle Scholar
  14. Bryan, C. J., Corso, M. L., Corso, K. A., Morrow, C. E., Kanzler, K. E., & Ray-Sannerud, B. (2012). Severity of mental health impairment and trajectories of improvement in an integrated primary care clinic. Journal of Consulting and Clinical Psychology, 80(3), 396–403.CrossRefGoogle Scholar
  15. Bryan, C. J., Morrow, C. E., & Appolonio, K. K. (2009). Impact of behavioral health consultant interventions on patient symptoms and functioning in an integrated family medicine clinic. Journal of Clinical Psychology, 65, 281–293.CrossRefGoogle Scholar
  16. Butler, M., Kane, R. L., McAlpine, D., Kathol, R. G., Fu, S. S., Hagedorn, H., & Wilt, T. J. (2008). Integration of Mental Health/Substance Abuse and Primary Care No. 173. (AHRQ Publication No. 09-E003.). Rockville: Agency for Healthcare Research and Quality.Google Scholar
  17. Carroll, C., Patterson, M., Wood, S., Booth, B., Rick, J., & Balain, S. (2007). A conceptual framework for implementation fidelity. Implementation Science, 2, 40.CrossRefGoogle Scholar
  18. Chaney, E. F., Rubenstein, L. V., Liu, C. F., Yano, E. M., Bolkan, C., Lee, M., … Uman, J. (2011). Implementing collaborative care for depression treatment in primary care: A cluster randomized evaluation of a quality improvement practice design. Implementation Science, 6, 121.CrossRefGoogle Scholar
  19. Cigrang, J. A., Rauch, S. A., Mintz, J., Brundige, A. R., Mitchell, J. A., Najera, E., … STRONG STAR Consortium. (2017). Moving effective treatment for posttraumatic stress disorder to primary care: A randomized controlled trial with active duty military. Families, Systems & Health, 35, 450–462.CrossRefGoogle Scholar
  20. Fauth, J., & Tremblay, G. (2011). The integrated care evaluation project: Antioch University.Google Scholar
  21. Fornell, C., & Larcker, D. F. (1981). Evaluating structural equation models with unobservable variables and measurement error. Journal of Marketing Research, 18, 39–50.CrossRefGoogle Scholar
  22. Fortney, J., Enderle, M., McDougall, S., Clothier, J., Otero, J., Altman, L., & Curran, G. (2012). Implementation outcomes of evidence-based quality improvement for depression in VA community based outpatient clinics. Implementation Science, 7, 30. Scholar
  23. Gilbody, S., Bower, P., Fletcher, J., Richards, D., & Sutton, A. J. (2006). Collaborative care for depression: A cumulative meta-analysis and review of longer-term outcomes. Archive of Internal Medicine, 166, 2314–2321.CrossRefGoogle Scholar
  24. Hasson, H. (2010). Systematic evaluation of implementation fidelity of complex interventions in health and social care. Implementation Science, 5, 67. Scholar
  25. Hrisos, S., Eccles, M. P., Francis, J. J., Dickinson, H. O., Kaner, E. F., Beyer, F., & Johnston, M. (2009). Are there valid proxy measures of clinical behaviour? A systematic review. Implementation Science, 4, 37. Scholar
  26. Hunter, C. L., & Goodie, J. L. (2010). Operational and clinical components for integrated-collaborative behavioral healthcare in the patient-centered medical home. Families, Systems & Health, 28, 308–321. Scholar
  27. Hunter, C. L., Goodie, J. L., Oordt, M. S., & Dobmeyer, A. C. (2017). Integrated behavioral health in primary care: Step-by-step guidance for assessment and intervention (2nd ed.). Washington, DC: American Psychological Association.CrossRefGoogle Scholar
  28. Kearney, L. K., Dollar, K. M., Beehler, G. P., Goldstein, W. R., Grasso, J., Wray, L. O., & Pomerantz, A. S. (in press). Creation and implementation of a national interprofessional integrated primary care competency training program: Preliminary findings and lessons learned. Training and Education in Professional Psychology.Google Scholar
  29. Kearney, L. K., Post, E. P., Zeiss, A., Goldstein, M. G., & Dundon, M. (2011). The role of mental and behavioral health in the application of the patient-centered medical home in the Department of Veterans Affairs. Translational Behavioral Medicine, 1, 624–628.CrossRefGoogle Scholar
  30. Mowbray, C. T., Holter, M. C., Teague, G. B., & Bybee, D. (2003). Fidelity criteria: Development, measurement, and validation. American Journal of Evaluation, 24, 315–340.CrossRefGoogle Scholar
  31. Netemeyer, R. G., Bearden, W. O., & Sharma, S. (2003). Scaling procedures: Issues and applications. Thousand Oaks, CA: Sage.CrossRefGoogle Scholar
  32. Newman, I., Lim, J., & Pineda, F. (2013). Content validity using mixed methods approach: Its application and development through the use of a Table of Specifications Methodology. Journal of Mixed Methods Research, 7, 243–260.CrossRefGoogle Scholar
  33. Nunally, J. C., & Bernstien, I. H. (1994). Psychometric theory (3rd ed.). New York: Mcfgraw-Hill.Google Scholar
  34. Oslin, D. W., Ross, J., Sayers, S., Murphy, J., Kane, V., & Katz, I. R. (2006). Screening, assessment, and management of depression in VA primary care clinics. The Behavioral Health Laboratory. Journal of General Internal Medicine, 21, 46–50.CrossRefGoogle Scholar
  35. Oxman, T. E., Schulberg, H. C., Greenberg, R. L., Dietrich, A. J., Williams, J. W. J., Nutting, P. A., & Bruce, M. L. (2006). A fidelity measure for integrated management of depression in primary care. Medical Care, 44, 1030–1037.CrossRefGoogle Scholar
  36. Possemato, K., Johnson, E. M., Beehler, G. P., Shepardson, R. L., King, P., Vair, C. L., … Wray, L. O. (2018). Patient outcomes associated with primary care behavioral health services: A systematic review. General Hospital Psychiatry, 53, 1–11.CrossRefGoogle Scholar
  37. Ray-Sannerud, B. N., Dolan, D. C., Morrow, C. E., Corso, K. A., Kanzler, K. E., Corso, M. L., & Bryan, C. J. (2012). Longitudinal outcomes after brief behavioral health intervention in an integrated primary care clinic. Families, Systems & Health, 30(1), 60–71. Scholar
  38. Reiter, J. T., Dobmeyer, A. C., & Hunter, C. L. (2018). The Primary Care Behavioral Health (PCBH) Model: An overview and operational definition. Journal of Clinical Psychology in Medical Settings, 25, 109–126.CrossRefGoogle Scholar
  39. Robinson, P. J., & Reiter, J. T. (2016). Behavioral consultation and primary care: A guide to integrating services (2nd ed.). New York, NY: Springer.CrossRefGoogle Scholar
  40. Ross, J. T., TenHave, T., Eakin, A. C., Difillippo, S., & Oslin, D. W. (2008). A randomized controlled trial of a close monitoring program for minor depression and distress. Journal of General Internal Medicine, 23, 1379–1385.CrossRefGoogle Scholar
  41. Rubenstein, L. V., Chaney, E. F., Ober, S., Felker, B., Sherman, S. E., Lanto, A., & Vivell, S. (2010). Using evidence-based quality improvement methods for translating depression collaborative care research into practice. Families, Systems, & Health, 28, 91–113.CrossRefGoogle Scholar
  42. Rubenstein, L. V., Williams, J. W., Danz, M., Shekelle, P., Suttorp, M., & Johnsen, B. (2009). Determining key features of effective depression interventions. Washington, DC: Department of Veterans Affairs.Google Scholar
  43. Shiner, B., Watts, B. V., Pomerantz, A., Groft, A., Scott, D., Street, B., & Young-Xu, Y. (2009). Access to what? An evaluation of the key ingredients to effective advanced mental health access at a VA Medical Center and its affiliated community-based outreach clinics. Military Medicine, 174, 1024–1032.CrossRefGoogle Scholar
  44. Strosahl, K. (1998). Integrating behavioral health and primary care services: The primary mental health care model. In A. Blount (Ed.), Integrated primary care: The future of medical and mental health collaboration (pp. 139–166). New York: Norton.Google Scholar
  45. VA Health Services Research and Development Service. (2008). Collaborative Care for Depression in the Primary Care Setting: A Primer on VA’s Translating Initiatives for Depression into Effective Solutions (TIDES) Project, Boston, MA.Google Scholar
  46. Williams, J. W., Jr., Gerrity, M., Holsinger, T., Dobscha, S., Gaynes, B., & Dietrich, A. (2007). Systematic review of multifaceted interventions to improve depression care. General Hospital Psychiatry, 29(2), 91–116. Scholar
  47. Woltmann, E., Grogan-Kaylor, A., Perron, B., Georges, H., Kilbourne, A. M., & Bauer, M. S. (2012). Comparative effectiveness of collaborative chronic care models for mental health conditions across primary, specialty, and behavioral health care settings: Systematic review and meta-analysis. American Journal of Psychiatry, 169, 790–804.CrossRefGoogle Scholar
  48. Wong, K. K. (2010). Handling small survey sample size and skewed dataset with partial least square path modelling (pp. 20–23). Vue: The Magazine of the Marketing Research and Intelligence Association.Google Scholar
  49. Zanjani, F., Miller, B., Turiano, N., Ross, J., & Oslin, D. (2008). Effectiveness of telephone-based referral care management: A brief intervention to improve psychiatric treatment engagement. Psychiatric Services, 59, 776–781.CrossRefGoogle Scholar

Copyright information

© This is a U.S. Government work and not under copyright protection in the US; foreign copyright protection may apply 2019

Authors and Affiliations

  • Gregory P. Beehler
    • 1
    • 2
    Email author
  • Jennifer S. Funderburk
    • 3
    • 4
    • 5
  • Paul R. King
    • 1
    • 6
  • Kyle Possemato
    • 3
    • 4
    • 5
  • John A. Maddoux
    • 7
  • Wade R. Goldstein
    • 1
  • Michael Wade
    • 3
  1. 1.VA Center for Integrated Healthcare (116N)VA WNY Healthcare SystemBuffaloUSA
  2. 2.School of Public Health and Health Professions, University at BuffaloThe State University of New YorkBuffaloUSA
  3. 3.VA Center for Integrated HealthcareSyracuse VA Medical CenterSyracuseUSA
  4. 4.Department of PsychologySyracuse UniversitySyracuseUSA
  5. 5.Department of PsychiatryUniversity of RochesterRochesterUSA
  6. 6.Department of Counseling, School, and Educational Psychology, University at BuffaloThe State University of New YorkBuffaloUSA
  7. 7.Harry S. Truman Memorial Veterans’ HospitalColumbiaUSA

Personalised recommendations