Mechanisms of Change in the ARC Organizational Strategy: Increasing Mental Health Clinicians’ EBP Adoption Through Improved Organizational Culture and Capacity

  • Nathaniel J. Williams
  • Charles Glisson
  • Anthony Hemmelgarn
  • Philip Green
Original Article

Abstract

The development of efficient and scalable implementation strategies in mental health is restricted by poor understanding of the change mechanisms that increase clinicians’ evidence-based practice (EBP) adoption. This study tests the cross-level change mechanisms that link an empirically-supported organizational strategy for supporting implementation (labeled ARC for Availability, Responsiveness, and Continuity) to mental health clinicians’ EBP adoption and use. Four hundred seventy-five mental health clinicians in 14 children’s mental health agencies were randomly assigned to the ARC intervention or a control condition. Measures of organizational culture, clinicians’ intentions to adopt EBPs, and job-related EBP barriers were collected before, during, and upon completion of the three-year ARC intervention. EBP adoption and use were assessed at 12-month follow-up. Multilevel mediation analyses tested changes in organizational culture, clinicians’ intentions to adopt EBPs, and job-related EBP barriers as linking mechanisms explaining the effects of ARC on clinicians’ EBP adoption and use. ARC increased clinicians’ EBP adoption (OR = 3.19, p = .003) and use (81 vs. 56 %, d = .79, p = .003) at 12-month follow-up. These effects were mediated by improvement in organizational proficiency culture leading to increased clinician intentions to adopt EBPs and by reduced job-related EBP barriers. A combined mediation analysis indicated the organizational culture-EBP intentions mechanism was the primary carrier of ARC’s effects on clinicians’ EBP adoption and use. ARC increases clinicians’ EBP adoption and use by creating proficient organizational cultures that increase clinicians’ intentions to adopt EBPs.

Keywords

Mechanism Mediation Organizational implementation strategy Organizational culture Intervention Mental health Evidence-based practice ARC 

Notes

Funding

This study was funded by grants from the National Institute of Mental Health to NJW (F31MH099846) and CG (R01MH084855). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institute of Mental Health.

Compliance with Ethical Standards

Conflict of interest

Nathaniel Williams declares that he has no conflict of interest. Charles Glisson declares that he has no conflict of interest. Anthony Hemmelgarn declares that he has no conflict of interest. Philip Green declares that he has no conflict of interest.

Ethical approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed consent

Informed consent was obtained from all individual participants included in the study.

References

  1. Aarons, G. A., Ehrhart, M. G., Farahnak, L. R., & Sklar, M. (2014). Aligning leadership across systems and organizations to develop a strategic climate for evidence-based practice implementation. Annual Review of Public Health, 35, 255–274.CrossRefPubMedPubMedCentralGoogle Scholar
  2. Aarons, G. A., Glisson, C., Green, P. D., Hoagwood, K., Kelleher, K. J., Landsverk, J. A., & The Research Networkon Youth Mental Health. (2012). The organizational social context of mental health services and clinician attitudes toward evidence-based practice: A United States national study. Implementation Science, 7(1), 56.CrossRefPubMedPubMedCentralGoogle Scholar
  3. Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011). Advancing a conceptual model of evidence-based practice implementation in public sectors. Administration and Policy in Mental Health, 38, 4–23.CrossRefPubMedGoogle Scholar
  4. Aarons, G. A., Sommerfeld, D. H., & Walrath-Greene, C. M. (2009). Evidence-based practice implementation: The impact of public versus private sector organization type on organizational support, provider attitudes, and adoption of evidence-based practice. Implementation Science, 4(1), 83.CrossRefPubMedPubMedCentralGoogle Scholar
  5. Ajzen, I. (1991). The theory of planned behavior. Organizational Behavior and Human Decision Processes, 50, 179–211.CrossRefGoogle Scholar
  6. Baer, J. S., Wells, E. A., Rosengren, D. B., Hartzler, B., Beadnell, B., & Dunn, C. (2009). Agency context and tailored training in technology transfer: A pilot evaluation of motivational interviewing training for community counselors. Journal of Substance Abuse Treatment, 37, 191–202.CrossRefPubMedPubMedCentralGoogle Scholar
  7. Bartholomew, N. G., Joe, G. W., Rowan-Szal, G. A., & Simpson, D. D. (2007). Counselor assessments of training and adoption barriers. Journal of Substance Abuse Treatment, 33, 193–199.CrossRefPubMedPubMedCentralGoogle Scholar
  8. Chan, D. (1998). Functional relations among constructs in the same content domain at different levels of analysis: A typology of composition models. Journal of Applied Psychology, 83, 234–246.CrossRefGoogle Scholar
  9. Chaudoir, S. R., Dugan, A. G., & Barr, C. H. (2013). Measuring factors affecting implementation of health innovations: A systematic review of structural, organizational, provider, patient, and innovation level measures. Implementation Science, 8(1), 22.CrossRefPubMedPubMedCentralGoogle Scholar
  10. Cohen, J., & Cohen, P. (1983). Applied multiple regression/correlation analysis for the behavioral sciences. Hillsdale: Erlbaum.Google Scholar
  11. Collins, P. Y., Patel, V., Joestl, S. S., March, D., Insel, T. R., Daar, A. S., et al. (2011). Grand challenges in global mental health. Nature, 475, 27–30.CrossRefPubMedPubMedCentralGoogle Scholar
  12. Cooke, R. A., & Rousseau, D. M. (1988). Behavioral norms and expectations: A quantitative approach to the assessment of organizational culture. Group & Organization Studies, 13, 245–273.CrossRefGoogle Scholar
  13. Ehrhart, M. G., Aarons, G. A., & Farahnak, L. R. (2014). Assessing the organizational context for EBP implementation: The development and validity testing of the Implementation Climate Scale (ICS). Implementation Science, 9, 157.CrossRefPubMedPubMedCentralGoogle Scholar
  14. Fishbein, M., Triandis, H. C., Kanfer, F. H., Becker, M., Middlestadt, S. E., & Eichler, A. (2001). Factors influencing behavior and behavior change. In A. Baum, T. Revenson, & J. Singer (Eds.), Handbook of health psychology (pp. 3–17). Mahwah: Lawrence Erlbaum Associates.Google Scholar
  15. Gioia, D., & Dziadosz, G. (2008). Adoption of evidence-based practices in community mental health: A mixed methods study of practitioner experience. Community Mental Health Journal, 44, 347–357.CrossRefPubMedGoogle Scholar
  16. Glisson, C., Dukes, D., & Green, P. (2006). The effects of the ARC organizational intervention on caseworker turnover, climate, and culture in children’s service systems. Child Abuse and Neglect, 30, 855–880.CrossRefPubMedGoogle Scholar
  17. Glisson, C., Green, P., & Williams, N. J. (2012a). Assessing the organizational social context (OSC) of child welfare systems: Implications for research and practice. Child Abuse and Neglect, 36, 621–632.CrossRefPubMedGoogle Scholar
  18. Glisson, C., Hemmelgarn, A., Green, P., Dukes, D., Atkinson, S., & Williams, N. J. (2012b). Randomized trial of the availability, responsiveness, and continuity (ARC) organizational intervention with community-based mental health programs and clinicians serving youth. Journal of the American Academy of Child and Adolescent Psychiatry, 51, 780–787.CrossRefPubMedGoogle Scholar
  19. Glisson, C., Hemmelgarn, A., Green, P., & Williams, N. J. (2013). Randomized trial of the availability, responsiveness and continuity (ARC) organizational intervention for improving youth outcomes in community mental health programs. Journal of the American Academy of Child and Adolescent Psychiatry, 52, 493–500.CrossRefPubMedGoogle Scholar
  20. Glisson, C., Landsverk, J., Schoenwald, S., Kelleher, K., Hoagwood, K. E., Mayberg, S., et al. (2008a). Assessing the organizational social context (OSC) of mental health services: Implications for research and practice. Administration and Policy in Mental Health, 35, 98–113.CrossRefPubMedGoogle Scholar
  21. Glisson, C., Schoenwald, S. K., Hemmelgarn, A., Green, P., Dukes, D., Armstrong, K. S., & Chapman, J. E. (2010). Randomized trial of MST and ARC in a two-level evidence-based treatment implementation strategy. Journal of Consulting and Clinical Psychology, 78, 537–550.CrossRefPubMedPubMedCentralGoogle Scholar
  22. Glisson, C., Schoenwald, S. K., Kelleher, K., Landsverk, J., Hoagwood, K. E., Mayberg, S., et al. (2008b). Therapist turnover and new program sustainability in mental health clinics as a function of organizational culture, climate, and service structure. Administration and Policy In Mental Health, 35, 124–133.CrossRefPubMedGoogle Scholar
  23. Glisson, C., Williams, N. J., Green, P., Hemmelgarn, A., & Hoagwood, K. (2014). The organizational social context of mental health Medicaid waiver programs with family support services: Implications for research and practice. Administration and Policy In Mental Health, 41, 32–42.CrossRefPubMedPubMedCentralGoogle Scholar
  24. Hartnell, C. A., Ou, A. Y., & Kinicki, A. (2011). Organizational culture and organizational effectiveness: A meta-analytic investigation of the competing values framework’s theoretical suppositions. Journal of Applied Psychology, 96, 677–694.CrossRefPubMedGoogle Scholar
  25. Hayes, A. F., & Scharkow, M. (2013). The relative trustworthiness of inferential tests of the indirect effect in statistical mediation analysis: Does method really matter? Psychological Science, 24, 1918–1927.CrossRefPubMedGoogle Scholar
  26. Hofstede, G., Neuijen, B., Ohayv, D. D., & Sanders, G. (1990). Measuring organizational cultures: A qualitative and quantitative study across twenty cases. Administrative Science Quarterly, 35, 286–316.CrossRefGoogle Scholar
  27. Hogue, A., Dauber, S., Henderson, C. E., & Liddle, H. A. (2014). Reliability of therapist self-report on treatment targets and focus in family-based intervention. Administration and Policy In Mental Health, 41, 697–705.CrossRefPubMedPubMedCentralGoogle Scholar
  28. Hurlburt, M. S., Garland, A. F., Nguyen, K., & Brookman-Frazee, L. (2010). Child and family therapy process: Concordance of therapist and observational perspectives. Administration and Policy in Mental Health, 37, 230–244.CrossRefPubMedGoogle Scholar
  29. Insel, T. R. (2009). Translating scientific opportunity into public health impact. Archives of General Psychiatry, 66, 128–133.CrossRefPubMedGoogle Scholar
  30. Jaccard, J., Litardo, H. A., & Wan, C. K. (1999). Subjective culture and social behavior. In J. Adamopoulos & Y. Kashima (Eds.), Social psychology and cultural context (pp. 95–106). Thousand Oaks: Sage publications.CrossRefGoogle Scholar
  31. James, L. R., Demaree, R. G., & Wolf, G. (1993). Rwg: An assessment of within-group agreement. Journal of Applied Psychology, 78, 306–309.CrossRefGoogle Scholar
  32. Jensen-Doss, A., Hawley, K. M., Lopez, M., & Osterberg, L. D. (2009). Using evidence-based treatments: The experiences of youth providers working under a mandate. Professional Psychology: Research and Practice, 40, 417–424.CrossRefGoogle Scholar
  33. Larson, E. L., Early, E., Cloonan, P., Sugrue, S., & Parides, M. (2000). An organizational climate intervention associated with increased handwashing and decreased nosocomial infections. Behavioral Medicine, 26, 14–22.CrossRefPubMedGoogle Scholar
  34. LeBreton, J. M., & Senter, J. L. (2008). Answers to 20 questions about interrater reliability and interrater agreement. Organizational Research Methods, 11, 815–852.CrossRefGoogle Scholar
  35. MacKinnon, D. P., Lockwood, C. M., Hoffman, J. M., West, S. G., & Sheets, V. (2002). A comparison of methods to test mediation and other intervening variable effects. Psychological Methods, 7, 83–104.CrossRefPubMedPubMedCentralGoogle Scholar
  36. MacKinnon, D. P., Lockwood, C. M., & Williams, J. (2004). Confidence limits for the indirect effect: Distribution of the product and resampling methods. Multivariate Behavioral Research, 39, 99–128.CrossRefPubMedPubMedCentralGoogle Scholar
  37. Michie, S., van Stralen, M. M., & West, R. (2011). The behavior change wheel: A new method for characterizing and designing behavior change interventions. Implementation Science. doi:10.1186/1748-5908-6-42.Google Scholar
  38. National Institute of Mental Health. (2008). National Institute of Mental Health strategic plan. Washington, DC: U. S. Department of Health and Human Services.Google Scholar
  39. Novins, D. K., Green, A. E., Legha, R. K., & Aarons, G. A. (2013). Dissemination and implementation of evidence-based practices for child and adolescent mental health: A systematic review. Journal of the American Academy of Child and Adolescent Psychiatry, 52, 1009–1025.CrossRefPubMedPubMedCentralGoogle Scholar
  40. Olin, S. S., Williams, N., Pollock, M., Armusewicz, K., Kutash, K., Glisson, C., & Hoagwood, K. E. (2014). Quality indicators for family support services and their relationship to organizational social context. Administration and Policy in Mental Health and Mental Health Services Research, 41, 43–54.CrossRefPubMedGoogle Scholar
  41. Parmelli, E., Flodgren, G., Beyer, F., Baillie, N., Schaafsma, M. E., & Eccles, M. P. (2011). The effectiveness of strategies to change organizational culture to improve healthcare performance: A systematic review. Implementation Science, 6(1), 33.CrossRefPubMedPubMedCentralGoogle Scholar
  42. Podsakoff, P. M., MacKenzie, S. B., Lee J., & Podsakoff, N. P. (2003). Common method bias in behavioral research: a critical review of the literature and recommended remedies. Journal of Applied Psychology, 88, 879–903.CrossRefPubMedGoogle Scholar
  43. Powell, B. J., McMillen, J. C., Proctor, E. K., Carpenter, C. R., Griffey, R. T., Bunger, A. C., et al. (2012). A compilation of strategies for implementing clinical innovations in health and mental health. Medical Care Research and Review, 69, 123–157.CrossRefPubMedGoogle Scholar
  44. Powell, B. J., Proctor, E. K., & Glass, J. E. (2014). A systematic review of strategies for implementing empirically supported mental health interventions. Research on Social Work Practice, 24, 192–212.CrossRefPubMedGoogle Scholar
  45. Preacher, K. J., & Kelley, K. (2011). Effect size measures for mediation models: Quantitative strategies for communicating indirect effects. Psychological Methods, 16, 93–115.CrossRefPubMedGoogle Scholar
  46. Preacher, K. J., & Selig, J. P. (2012). Advantages of Monte Carlo confidence intervals for indirect effects. Communication Methods and Measures, 6, 77–98.CrossRefGoogle Scholar
  47. Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., & Hensley, M. (2010). Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health, 38, 65–76.CrossRefPubMedCentralGoogle Scholar
  48. Raghavan, R., Inkelas, M., Franke, T., & Halfon, N. (2007). Administrative barriers to the adoption of high-quality mental health services for children in foster care: A national study. Administration and Policy in Mental Health and Mental Health Services Research, 34, 191–201.CrossRefPubMedGoogle Scholar
  49. Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis (2nd ed.). Thousand Oaks: Sage publications.Google Scholar
  50. Rogers, E. M. (2003). Diffusion of innovations (5th ed.). New York: Free Press.Google Scholar
  51. Schein, E. H. (2004). Organizational culture and leadership. San Francisco, CA: Jossey-Bass.Google Scholar
  52. Schoenwald, S. K., Chapman, J. E., Kelleher, K., Hoagwood, K. E., Landsverk, J., Stevens, J., et al. (2008). A survey of the infrastructure for children’s mental health services: Implications for the implementation of empirically supported treatments (ESTs). Administration and Policy In Mental Health, 35, 84–97.CrossRefPubMedGoogle Scholar
  53. Schulte, M., Ostroff, C., Shmulyian, S., & Kinicki, A. (2009). Organizational climate configurations: Relationships to collective attitudes, customer satisfaction, and financial performance. Journal of Applied Psychology, 94, 618–634.CrossRefPubMedGoogle Scholar
  54. Sheeran, P. (2002). Intention-behavior relations: A conceptual and empirical review. In W. Stroebe & M. Hewstone (Eds.), European review of social psychology (Vol. 12, pp. 1–36). London: Wiley.Google Scholar
  55. Taylor, A. B., MacKinnon, D. P., & Tein, J. Y. (2008). Tests of the three-path mediated effect. Organizational Research Methods, 11, 241–269.CrossRefGoogle Scholar
  56. Trice, H. M., & Beyer, J. M. (1993). The cultures of work organizations. Englewood Cliffs: Prentice Hall.Google Scholar
  57. Walrath, C. M., Sheehan, A. K., Holden, E. W., Hernandez, M., & Blau, G. (2006). Evidence-based treatments in the field: A brief report on provider knowledge, implementation, and practice. Journal of Behavioral Health Services Research, 33, 244–253.CrossRefPubMedGoogle Scholar
  58. Williams, N. J. (2015). Multilevel mechanisms of implementation strategies in mental health: Integrating theory, research, and practice. Administration and Policy in Mental Health and Mental Health Services Research,. doi:10.1007/s10488-015-0693-2.PubMedCentralGoogle Scholar
  59. Williams, N. J., & Glisson, C. (2013). Reducing turnover is not enough: The need for proficient organizational cultures to support positive youth outcomes in child welfare. Children and Youth Services Review, 35, 1871–1877.CrossRefGoogle Scholar
  60. Williams, N. J., & Glisson, C. (2014a). Testing a theory of organizational culture, climate, and youth outcomes in child welfare systems: A United States national study. Child Abuse and Neglect, 38, 757–767.CrossRefPubMedGoogle Scholar
  61. Williams, N. J., & Glisson, C. (2014b). The role of organizational culture and climate in the dissemination and implementation of empirically-supported treatments for youth. In R. S. Beidas & P. C. Kendall (Eds.), Dissemination and implementation of evidence based practices in child and adolescent mental health (pp. 61–81). New York: Oxford University Press.Google Scholar
  62. Zhang, Z., Zyphur, M. J., & Preacher, K. J. (2009). Testing multilevel mediation using hierarchical linear models: Problems and solutions. Organizational Research Methods, 12, 695–719.CrossRefGoogle Scholar
  63. Zohar, D., & Polachek, T. (2014). Discourse-based intervention for modifying supervisory communication as leverage for safety climate and performance improvement: A randomized field study. Journal of Applied Psychology, 99, 113–124.CrossRefPubMedGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  • Nathaniel J. Williams
    • 1
  • Charles Glisson
    • 2
  • Anthony Hemmelgarn
    • 2
  • Philip Green
    • 2
  1. 1.School of Social WorkBoise State UniversityBoiseUSA
  2. 2.Children’s Mental Health Services Research CenterUniversity of TennesseeKnoxvilleUSA

Personalised recommendations