Validation of an Expanded Measure of Integrated Care Provider Fidelity: PPAQ-2

Abstract

This study aimed to validate the factor structure of the expanded Primary Care Behavioral Health Provider Adherence Questionnaire (PPAQ-2), which is designed to assess provider fidelity to both the Primary Care Behavioral Health (PCBH) and collaborative care management (CCM) models of integrated primary care. Two-hundred fifty-three integrated care providers completed self-reports of professional background, perceptions of clinic integration and related practice barriers, and the PPAQ-2. Confirmatory factor analyses were conducted to assess the theorized factor structure and criterion validity was assessed through correlational analysis. Factor analyses demonstrated adequate fit with the data and acceptable to excellent composite reliabilities across five PCBH domains and five CCM domains. Validity was demonstrated by correlations between adherence scores and measures of clinic integration and barriers to fidelity. The PPAQ-2 is a psychometrically sound measure that can be used in future integrated care dismantling studies to identify provider behaviors that best predict patient outcomes.

This is a preview of subscription content, log in to check access.

Fig. 1

References

  1. Afthanorhan, W. M. A. B. W. (2013). A comparison of partial least square structural equation modeling (PLS-SEM) and covariance based structural equation modeling (CB-SEM) for confirmatory factor analysis. International Journal of Engineering Science and Innovation Technology,2, 198–205.

    Google Scholar 

  2. Atkins, D., Kilbourne, A. M., & Shulkin, D. (2017). Moving from discovery to system-wide change: The role of research in a learning health care system: Experience from three decades of health systems research in the Veterans Health Administration. Annual Review of Public Health,38, 467–487.

    Article  Google Scholar 

  3. Awang, Z., Ahmad, J. H., & Zin, N. M. (2010). Modelling job satisfaction and work commitment among lecturers: a case of UITM kelantan. Journal of Statistical Modeling and Analytics,1, 45–59.

    Google Scholar 

  4. Bacon, L. D. (1999). Using LISREL and PLS to Measure Customer Satisfaction. Paper presented at the Sawtooth Software Conference Proceeding, La Jolla, CA.

  5. Beehler, G. P., Funderburk, J. S., King, P. R., Wade, M., & Possemato, K. (2015). Using the Primary Care Behavioral Health Provider Adherence Questionnaire (PPAQ) to identify variation in practice patterns. Translational Behavioral Medicine,5, 384–392.

    Article  Google Scholar 

  6. Beehler, G. P., Funderburk, J. S., Possemato, K., & Dollar, K. M. (2013). Psychometric assessment of the Primary Care Behavioral Health Provider Adherence Questionnaire (PPAQ). Translational Behavioral Medicine,3, 379–391.

    Article  Google Scholar 

  7. Beehler, G. P., Funderburk, J. S., Possemato, K., & Vair, C. L. (2013). Developing a measure of provider adherence to improve the implementation of behavioral health services in primary care: A Delphi study. Implementation Science,8, 19.

    Article  Google Scholar 

  8. Beehler, G. P., & Lilienthal, K. R. (2017). Provider perceptions of an integrated primary care quality improvement strategy: The PPAQ Toolkit. Psychological Services,14, 50–56.

    Article  Google Scholar 

  9. Beehler, G. P., & Wray, L. O. (2012). Behavioral health providers’ perspectives of delivering behavioral health services in primary care: a qualitative analysis. BMC Health Services Research,12, 337. https://doi.org/10.1186/1472-6963-12-337.

    Article  PubMed  PubMed Central  Google Scholar 

  10. Bellg, A. J., Borrelli, B., Resnick, B., Hecht, J., Minicucci, D. S., Ory, M., … Czajkowski, S. (2004). Enhancing treatment fidelity in health behavior change studies: Best practices and recommendations from the NIH Behavior Change Consortium. Health Psychology,23(5), 443–451. https://doi.org/10.1037/0278-6133.23.5.443.

    Article  PubMed  Google Scholar 

  11. Bower, P., Gilbody, S., Richards, D., Fletcher, J., & Sutton, A. (2006). Collaborative care for depression in primary care: Making sense of a complex intervention: Systematic review and meta-regression. The British Journal of Psychiatry,189, 484–493.

    Article  Google Scholar 

  12. Brawer, P. A., Brugh, A. M. T., Martielli, R. P., O’Connor, S. P., Mastnak, J., Scherrer, J. F., & Day, T. E. (2011). Enhancing entrance into PTSD treatment for post-deployment veterans through collaborative/integrative care. Translational Behavioral Medicine,1(4), 609–614.

    Article  Google Scholar 

  13. Brawer, P. A., Martielli, R. P., Pye, P. L., Manwaring, J., & Tierney, A. (2010). St. Louis Initiative for Integrated Care Excellence (SLI2CE): Integrated collaborative care on a large scale model Families. Systems, & Health,28, 175–187.

    Article  Google Scholar 

  14. Bryan, C. J., Corso, M. L., Corso, K. A., Morrow, C. E., Kanzler, K. E., & Ray-Sannerud, B. (2012). Severity of mental health impairment and trajectories of improvement in an integrated primary care clinic. Journal of Consulting and Clinical Psychology,80(3), 396–403.

    Article  Google Scholar 

  15. Bryan, C. J., Morrow, C. E., & Appolonio, K. K. (2009). Impact of behavioral health consultant interventions on patient symptoms and functioning in an integrated family medicine clinic. Journal of Clinical Psychology,65, 281–293.

    Article  Google Scholar 

  16. Butler, M., Kane, R. L., McAlpine, D., Kathol, R. G., Fu, S. S., Hagedorn, H., & Wilt, T. J. (2008). Integration of Mental Health/Substance Abuse and Primary Care No. 173. (AHRQ Publication No. 09-E003.). Rockville: Agency for Healthcare Research and Quality.

  17. Carroll, C., Patterson, M., Wood, S., Booth, B., Rick, J., & Balain, S. (2007). A conceptual framework for implementation fidelity. Implementation Science,2, 40.

    Article  Google Scholar 

  18. Chaney, E. F., Rubenstein, L. V., Liu, C. F., Yano, E. M., Bolkan, C., Lee, M., … Uman, J. (2011). Implementing collaborative care for depression treatment in primary care: A cluster randomized evaluation of a quality improvement practice design. Implementation Science,6, 121.

    Article  Google Scholar 

  19. Cigrang, J. A., Rauch, S. A., Mintz, J., Brundige, A. R., Mitchell, J. A., Najera, E., … STRONG STAR Consortium. (2017). Moving effective treatment for posttraumatic stress disorder to primary care: A randomized controlled trial with active duty military. Families, Systems & Health,35, 450–462.

    Article  Google Scholar 

  20. Fauth, J., & Tremblay, G. (2011). The integrated care evaluation project: Antioch University.

  21. Fornell, C., & Larcker, D. F. (1981). Evaluating structural equation models with unobservable variables and measurement error. Journal of Marketing Research,18, 39–50.

    Article  Google Scholar 

  22. Fortney, J., Enderle, M., McDougall, S., Clothier, J., Otero, J., Altman, L., & Curran, G. (2012). Implementation outcomes of evidence-based quality improvement for depression in VA community based outpatient clinics. Implementation Science,7, 30. https://doi.org/10.1186/1748-5908-7-30.

    Article  PubMed  Google Scholar 

  23. Gilbody, S., Bower, P., Fletcher, J., Richards, D., & Sutton, A. J. (2006). Collaborative care for depression: A cumulative meta-analysis and review of longer-term outcomes. Archive of Internal Medicine,166, 2314–2321.

    Article  Google Scholar 

  24. Hasson, H. (2010). Systematic evaluation of implementation fidelity of complex interventions in health and social care. Implementation Science,5, 67. https://doi.org/10.1186/1748-5908-5-67.

    Article  PubMed  Google Scholar 

  25. Hrisos, S., Eccles, M. P., Francis, J. J., Dickinson, H. O., Kaner, E. F., Beyer, F., & Johnston, M. (2009). Are there valid proxy measures of clinical behaviour? A systematic review. Implementation Science,4, 37. https://doi.org/10.1186/1748-5908-4-37.

    Article  PubMed  Google Scholar 

  26. Hunter, C. L., & Goodie, J. L. (2010). Operational and clinical components for integrated-collaborative behavioral healthcare in the patient-centered medical home. Families, Systems & Health,28, 308–321. https://doi.org/10.1037/a0021761.

    Article  Google Scholar 

  27. Hunter, C. L., Goodie, J. L., Oordt, M. S., & Dobmeyer, A. C. (2017). Integrated behavioral health in primary care: Step-by-step guidance for assessment and intervention (2nd ed.). Washington, DC: American Psychological Association.

    Google Scholar 

  28. Kearney, L. K., Dollar, K. M., Beehler, G. P., Goldstein, W. R., Grasso, J., Wray, L. O., & Pomerantz, A. S. (in press). Creation and implementation of a national interprofessional integrated primary care competency training program: Preliminary findings and lessons learned. Training and Education in Professional Psychology.

  29. Kearney, L. K., Post, E. P., Zeiss, A., Goldstein, M. G., & Dundon, M. (2011). The role of mental and behavioral health in the application of the patient-centered medical home in the Department of Veterans Affairs. Translational Behavioral Medicine,1, 624–628.

    Article  Google Scholar 

  30. Mowbray, C. T., Holter, M. C., Teague, G. B., & Bybee, D. (2003). Fidelity criteria: Development, measurement, and validation. American Journal of Evaluation,24, 315–340.

    Article  Google Scholar 

  31. Netemeyer, R. G., Bearden, W. O., & Sharma, S. (2003). Scaling procedures: Issues and applications. Thousand Oaks, CA: Sage.

    Google Scholar 

  32. Newman, I., Lim, J., & Pineda, F. (2013). Content validity using mixed methods approach: Its application and development through the use of a Table of Specifications Methodology. Journal of Mixed Methods Research,7, 243–260.

    Article  Google Scholar 

  33. Nunally, J. C., & Bernstien, I. H. (1994). Psychometric theory (3rd ed.). New York: Mcfgraw-Hill.

    Google Scholar 

  34. Oslin, D. W., Ross, J., Sayers, S., Murphy, J., Kane, V., & Katz, I. R. (2006). Screening, assessment, and management of depression in VA primary care clinics. The Behavioral Health Laboratory. Journal of General Internal Medicine,21, 46–50.

    Article  Google Scholar 

  35. Oxman, T. E., Schulberg, H. C., Greenberg, R. L., Dietrich, A. J., Williams, J. W. J., Nutting, P. A., & Bruce, M. L. (2006). A fidelity measure for integrated management of depression in primary care. Medical Care,44, 1030–1037.

    Article  Google Scholar 

  36. Possemato, K., Johnson, E. M., Beehler, G. P., Shepardson, R. L., King, P., Vair, C. L., … Wray, L. O. (2018). Patient outcomes associated with primary care behavioral health services: A systematic review. General Hospital Psychiatry,53, 1–11.

    Article  Google Scholar 

  37. Ray-Sannerud, B. N., Dolan, D. C., Morrow, C. E., Corso, K. A., Kanzler, K. E., Corso, M. L., & Bryan, C. J. (2012). Longitudinal outcomes after brief behavioral health intervention in an integrated primary care clinic. Families, Systems & Health,30(1), 60–71. https://doi.org/10.1037/a0027029.

    Article  Google Scholar 

  38. Reiter, J. T., Dobmeyer, A. C., & Hunter, C. L. (2018). The Primary Care Behavioral Health (PCBH) Model: An overview and operational definition. Journal of Clinical Psychology in Medical Settings,25, 109–126.

    Article  Google Scholar 

  39. Robinson, P. J., & Reiter, J. T. (2016). Behavioral consultation and primary care: A guide to integrating services (2nd ed.). New York, NY: Springer.

    Google Scholar 

  40. Ross, J. T., TenHave, T., Eakin, A. C., Difillippo, S., & Oslin, D. W. (2008). A randomized controlled trial of a close monitoring program for minor depression and distress. Journal of General Internal Medicine,23, 1379–1385.

    Article  Google Scholar 

  41. Rubenstein, L. V., Chaney, E. F., Ober, S., Felker, B., Sherman, S. E., Lanto, A., & Vivell, S. (2010). Using evidence-based quality improvement methods for translating depression collaborative care research into practice. Families, Systems, & Health,28, 91–113.

    Article  Google Scholar 

  42. Rubenstein, L. V., Williams, J. W., Danz, M., Shekelle, P., Suttorp, M., & Johnsen, B. (2009). Determining key features of effective depression interventions. Washington, DC: Department of Veterans Affairs.

    Google Scholar 

  43. Shiner, B., Watts, B. V., Pomerantz, A., Groft, A., Scott, D., Street, B., & Young-Xu, Y. (2009). Access to what? An evaluation of the key ingredients to effective advanced mental health access at a VA Medical Center and its affiliated community-based outreach clinics. Military Medicine,174, 1024–1032.

    Article  Google Scholar 

  44. Strosahl, K. (1998). Integrating behavioral health and primary care services: The primary mental health care model. In A. Blount (Ed.), Integrated primary care: The future of medical and mental health collaboration (pp. 139–166). New York: Norton.

    Google Scholar 

  45. VA Health Services Research and Development Service. (2008). Collaborative Care for Depression in the Primary Care Setting: A Primer on VA’s Translating Initiatives for Depression into Effective Solutions (TIDES) Project, Boston, MA.

  46. Williams, J. W., Jr., Gerrity, M., Holsinger, T., Dobscha, S., Gaynes, B., & Dietrich, A. (2007). Systematic review of multifaceted interventions to improve depression care. General Hospital Psychiatry,29(2), 91–116. https://doi.org/10.1016/j.genhosppsych.2006.12.003.

    Article  PubMed  Google Scholar 

  47. Woltmann, E., Grogan-Kaylor, A., Perron, B., Georges, H., Kilbourne, A. M., & Bauer, M. S. (2012). Comparative effectiveness of collaborative chronic care models for mental health conditions across primary, specialty, and behavioral health care settings: Systematic review and meta-analysis. American Journal of Psychiatry,169, 790–804.

    Article  Google Scholar 

  48. Wong, K. K. (2010). Handling small survey sample size and skewed dataset with partial least square path modelling (pp. 20–23). Vue: The Magazine of the Marketing Research and Intelligence Association.

    Google Scholar 

  49. Zanjani, F., Miller, B., Turiano, N., Ross, J., & Oslin, D. (2008). Effectiveness of telephone-based referral care management: A brief intervention to improve psychiatric treatment engagement. Psychiatric Services,59, 776–781.

    Article  Google Scholar 

Download references

Funding

This study was funded by the VA Center for Integrated Healthcare at the VA Western New York Healthcare System. The information provided in this study does not represent the views of the Department of Veterans Affairs or the United States Government.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Gregory P. Beehler.

Ethics declarations

Conflict of interest

Gregory P. Beehler, Jennifer S. Funderburk, Paul R. King, Kyle Possemato, John A. Maddoux, Wade R. Goldstein, Michael Wade have no conflict of interest.

Ethical Approval

 All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Human and Animal Rights and Informed Consent

All procedures followed were in accordance with the ethical standard of the responsible committee on human experimentation of the VA Western New York Healthcare System and with the Helsinki Declaration of 1975, as revised in 2000. As an anonymous provider survey, this study was deemed exempt from Institutional Review Board review.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Beehler, G.P., Funderburk, J.S., King, P.R. et al. Validation of an Expanded Measure of Integrated Care Provider Fidelity: PPAQ-2. J Clin Psychol Med Settings 27, 158–172 (2020). https://doi.org/10.1007/s10880-019-09628-0

Download citation

Keywords

  • Guideline adherence
  • Integrated primary care
  • Mental health services
  • Primary health care
  • Program evaluation