Advertisement

Discrepancies and Agreement in Perceptions of Implementation Leadership: Associations with Dosage of School-Based Evidence-Based Practices for Children with Autism

  • Rosemary D. MezaEmail author
  • Rinad S. Beidas
  • Mark G. Ehrhart
  • David S. Mandell
  • Shannon Dorsey
  • Lindsay Frederick
  • Cristine Oh
  • Jill Locke
Original Article
  • 43 Downloads

Abstract

Studies demonstrate that discrepancies among leader–follower perceptions of leadership are related to organizational processes that may impact evidence-based practice (EBP) implementation. However, it is unknown whether discrepancies in leadership perceptions also predict EBP use. This study examined the association of principal–staff alignment and discrepancy in perceptions of implementation leadership with the dosage of three Autism-focused school-based EBPs, Discrete Trial Training, Pivotal Response Training (PRT), and Visual Schedules. PRT dosage was higher when principals under-rated their implementation leadership and when it was agreed upon that implementation leadership was low. Findings have implications for leaders support of EBPs implemented in school-based settings.

Keywords

Implementation leadership Discrepancy Evidence-based practice dosage School mental health Autism 

Notes

Funding

This study was funded by the National Institute of Mental Health (K01 MH100199).

Compliance with Ethical Standards

Conflicts of interest

The authors have no conflicts of interest to declare.

Ethical Approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

References

  1. Aarons, G. A., Ehrhart, M. G., & Farahnak, L. R. (2014). The implementation leadership scale (ILS): Development of a brief measure of unit level implementation leadership. Implementation Science, 9(1), 1–10.Google Scholar
  2. Aarons, G. A., Ehrhart, M. G., Farahnak, L. R., Sklar, M., & Horowitz, J. (2015). Discrepancies in leader and follower ratings of transformational leadership: Relationship with organizational culture in mental health. Administration and Policy in Mental Health and Mental Health Services Research, 44(4), 480–491.  https://doi.org/10.1007/s10488-015-0672-7.Google Scholar
  3. Aarons, G. A., Ehrhart, M. G., Moullin, J. C., Torres, E. M., & Green, A. E. (2017a). Testing the leadership and organizational change for implementation (LOCI) intervention in substance abuse treatment: A cluster randomized trial study protocol. Implementation Science, 12(1), 1–11.  https://doi.org/10.1186/s13012-017-0562-3.Google Scholar
  4. Aarons, G. A., Ehrhart, M. G., Torres, E. M., Finn, N. K., & Beidas, R. S. (2017b). The humble leader: Association of discrepancies in leader and follower ratings of implementation leadership with organizational climate in mental health. Psychiatric Services, 68(2), 115–122.  https://doi.org/10.1176/appi.ps.201600062.Google Scholar
  5. Aarons, G. A., Green, A. E., Trott, E., Willging, C. E., Torres, E. M., Ehrhart, M. G., et al. (2016). The roles of system and organizational leadership in system-wide evidence-based intervention sustainment: A mixed-method study. Administration and Policy in Mental Health and Mental Health Services Research, 43(6), 991–1008.  https://doi.org/10.1007/s10488-016-0751-4.Google Scholar
  6. Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health and Mental Health Services Research, 28(1), 4–23.  https://doi.org/10.1007/s10488-010-0327-7.Google Scholar
  7. Aarons, G., & Sommerfeld, D. (2012). Leadership, innovation climate, and attitudes toward evidence-based practice during a statewide implementation. Journal of the American Academy of Child and Adolescent Psychiatry, 51(4), 423–431.  https://doi.org/10.1016/j.jaac.2012.01.018.Google Scholar
  8. American Psychological Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed.). Arlington, VA: American Psychiatric Publishing.Google Scholar
  9. Arick, J. R., Young, H. E., Falco, R. A., Loos, L. M., Krug, D. A., Gense, M. H., et al. (2003). Designing an outcome study to monitor the progress of students with autism spectrum disorders. Focus on Autism and Other Developmental Disabilities, 18(2), 75–87.  https://doi.org/10.1177/108835760301800201.Google Scholar
  10. Atwater, L., Roush, P., & Fischthal, A. (1995). The influence of upward feedback on self- and follower ratings of leadership. Personnel Psychology, 48(1), 35–59.  https://doi.org/10.1111/j.1744-6570.1995.tb01745.x.Google Scholar
  11. Atwater, L., Waldman, D., Ostroff, C., Robie, C., Waldman, D., & Johnson, K. M. (2005). Self–other agreement: Comparing its relationship with performance in the U.S. and Europe. International Journal of Selection and Assessment, 13(1), 25–40.  https://doi.org/10.1111/j.0965-075X.2005.00297.x.Google Scholar
  12. Atwater, L. E., & Yammarino, F. J. (1992). Does self-other agreement on leadership perceptions moderate the validity of leadership and performance predictions? Personnel Psychology, 45(1), 141–164.  https://doi.org/10.1111/j.1744-6570.1992.tb00848.x.Google Scholar
  13. Bass, B. M., & Yammarino, F. J. (1991). Congruence of self and others’ leadership ratings of naval officers for understanding successful performance. Applied Psychology, 40(4), 437–454.Google Scholar
  14. Bliese, P. D. (2000). Within-group agreement, non-independence, and reliability. In K. J. Klein & S. W. Kozlowski (Eds.), Multilevel theory research, and methods in organizations (pp. 349–381). San Francisco: Jossey-Bass.Google Scholar
  15. Bradley, E. H., Curry, L. A., & Devers, K. J. (2007). Qualitative data analysis for health services research: Developing taxonomy, themes, and theory. Health Services Research, 42(4), 1758–1772.  https://doi.org/10.1111/j.1475-6773.2006.00684.x.Google Scholar
  16. Brookman-Frazee, L., & Stahmer, A. C. (2018). Effectiveness of a multi-level implementation strategy for ASD interventions: Study protocol for two linked cluster randomized trials. Implementation Science, 13(66), 1–14.  https://doi.org/10.1186/s13012-018-0757-2.Google Scholar
  17. Burns, B. J., Costello, E. J., Angold, A., Tweed, D., Stangl, D., Farmer, E. M. Z., et al. (1995). Children’s mental health service use across service sectors. Health Affairs, 14(3), 147–159.Google Scholar
  18. Cook, B. G., & Odom, S. L. (2013). Evidence-based practices and implementation science in special education. Exceptional Children, 79(2), 135–144.  https://doi.org/10.1177/001440291307900201.Google Scholar
  19. Cook, R. D., & Weisberg, S. (1982). Residuals and influence in regression. New York: Chapman and Hall.Google Scholar
  20. Cummings, G. G., Estabrooks, C. A., Midodzi, W. K., Wallin, L., & Hayduk, L. (2007). Influence of organizational characteristics and context on research utilization. Entrepreneurship and Regional Development, 55(4), S24–S39.  https://doi.org/10.1080/08985626.2017.1328905.Google Scholar
  21. Dettmer, S., Simpson, R. L., Smith Myles, B., & Ganz, J. G. (2000). The use of visual supports to facilitate transition of students with autism. Focus on Autism and Other Developmental Disabilities, 15(3), 163–169.Google Scholar
  22. Edmondson, A. (1999). Psychological safety and learning behavior in work teams. Administrative Science Quarterly, 44(2), 350.  https://doi.org/10.2307/2666999.Google Scholar
  23. Farmer, E. M. Z., Burns, B. J., Phillips, S. D., Angold, A., & Costello, E. J. (2003). Pathways into and through mental health services for children and adolescents. Psychiatric Services, 54(1), 60–66.  https://doi.org/10.1176/appi.ps.54.1.60.Google Scholar
  24. Fleenor, J. W., McCauley, C. D., & Brutus, S. (1996). Self-other rating agreement and leader effectiveness. Leadership Quarterly, 7(4), 487–506.  https://doi.org/10.1016/S1048-9843(96)90003-X.Google Scholar
  25. Forman, S. G., Olin, S. S., Hoagwood, K. E., Crowe, M., & Saka, N. (2009). Evidence-based interventions in schools: Developers’ views of implementation barriers and facilitators. School Mental Health, 1(1), 26–36.  https://doi.org/10.1007/s12310-008-9002-5.Google Scholar
  26. Gottfredson, D. C., & Gottfredson, G. D. (2002). Quality of school-based prevention programs: Results from a national survey. Journal of Research in Crime and Delinquency, 39(1), 3–35.Google Scholar
  27. Henggeler, S. W., Chapman, J. E., Rowland, M. D., Halliday-Boykins, C. A., Randall, J., Shackelford, J., et al. (2008). Statewide adoption and initial implementation of contingency management for substance-abusing adolescents. Journal of Consulting and Clinical Psychology, 76(4), 556–567.  https://doi.org/10.1037/0022-006X.76.4.556.Google Scholar
  28. James, L. R. (1982). Aggregation bias in estimates of perceptual agreement. Journal of Applied Psychology, 67(2), 219–229.Google Scholar
  29. Koegel, R. L., Schreibman, L., Good, A., Cerniglia, L., Murphy, C., & Koegel, L. K. (1988). How to teach pivotal behaviors to children with autism: A training manual. Santa Barbara: University of California.Google Scholar
  30. Langley, A. K., Nadeem, E., Kataoka, S. H., Stein, B. D., & Jaycox, L. H. (2010). Evidence-based mental health programs in schools: Barriers and facilitators of successful implementation. School Mental Health, 2(3), 105–113.  https://doi.org/10.1007/s12310-010-9038-1.Google Scholar
  31. Lebreton, J. M., & Senter, J. L. (2008). Answers to 20 questions about interrater reliability and interrater agreement. Organizational Research Methods, 11(4), 815–852.Google Scholar
  32. Locke, J., Beidas, R. S., Marcus, S., Stahmer, A., Aarons, G. A., Lyon, A. R., et al. (2016). A mixed methods study of individual and organizational factors that affect implementation of interventions for children with autism in public schools. Implementation Science, 11(1), 1–9.  https://doi.org/10.1186/s13012-016-0501-8.Google Scholar
  33. Locke, J., Olsen, A., Wideman, R., Downey, M. M., Kretzmann, M., Kasari, C., et al. (2015). A tangled web: The challenges of implementing an evidence-based social engagement intervention for children with autism in urban public school settings. Behavior Therapy, 46(1), 54–67.  https://doi.org/10.1016/j.beth.2014.05.001.Google Scholar
  34. Locke, J., Violante, S., Pullmann, M. D., Kerns, S. E. U., Jungbluth, N., & Dorsey, S. (2017). Agreement and discrepancy between supervisor and clinician alliance: Associations with clinicians’ perceptions of psychological climate and emotional exhaustion. Administration and Policy in Mental Health and Mental Health Services Research, 0(0), 1–13.  https://doi.org/10.1007/s10488-017-0841-y.Google Scholar
  35. Lyon, A. R., Cook, C. R., Brown, E. C., Locke, J., Davis, C., Ehrhart, M., et al. (2018). Assessing organizational implementation context in the education sector: Confirmatory factor analysis of measures of implementation leadership, climate, and citizenship. Implementation Science, 13(1), 1–14.  https://doi.org/10.1186/s13012-017-0705-6.Google Scholar
  36. Mandell, D. S., Stahmer, A. C., Shin, S., Xie, M., Reisinger, E., & Marcus, S. C. (2013). The role of treatment fidelity on outcomes during a randomized field trial of an autism intervention. Autism, 17(3), 281–295.  https://doi.org/10.1177/1362361312473666.Google Scholar
  37. Mihalic, S. F., Fagan, A. A., & Argamaso, S. (2008). Implementing the LifeSkills training drug prevention program: Factors related to implementation fidelity. Implementation Science, 3(1), 1–16.  https://doi.org/10.1186/1748-5908-3-5.Google Scholar
  38. National Research Council (U.S.). (2001). Educating children with autism. Washington, DC: National Academy Press.Google Scholar
  39. Niemiec, C. P., & Ryan, R. M. (2009). Autonomy, competence, and relatedness in the classroom: Applying self-determination theory to educational practice. Theory and Research in Education, 7(2), 133–144.  https://doi.org/10.1177/1477878509104318.Google Scholar
  40. Ostroff, C., Atwater, L. E., & Feinberg, B. J. (2004). Understanding self-other agreement: A look at rater and ratee characteristics, context, and outcomes. Personnel Psychology, 57(2), 333–375.  https://doi.org/10.1111/j.1744-6570.2004.tb02494.x.Google Scholar
  41. Owens, B. P., & Hekman, D. R. (2012). Modeling how to grow: An inductive examination of humble leader behaviors, contingencies, and outcomes. The Academy of Management Journal, 55(4), 787–818.Google Scholar
  42. Owens, B. P., & Hekman, D. R. (2016). How does leader humility influence team performance? Exploring the mechanisms of contagion and collective promotion focus. Academy of Management Journal, 59(3), 1088–1111.  https://doi.org/10.5465/amj.2013.0660.Google Scholar
  43. Ozer, E. J. (2006). Contextual effects in school-based violence prevention programs: A conceptual framework and empirical review. Journal of Primary Prevention, 27(3), 315–340.  https://doi.org/10.1007/s10935-006-0036-x.Google Scholar
  44. Palinkas, L. A., Aarons, G. A., Horwitz, S., Chamberlain, P., Hurlburt, M., & Landsverk, J. (2011). Mixed method designs in implementation research. Administration and Policy in Mental Health and Mental Health Services Research, 38(1), 44–53.  https://doi.org/10.1007/s10488-010-0314-z.Google Scholar
  45. Pellecchia, M., Connell, J. E., Beidas, R. S., Xie, M., Marcus, S. C., & Mandell, D. S. (2015). Dismantling the active ingredients of an intervention for children with autism. Journal of Autism and Developmental Disorders, 45(9), 2917–2927.  https://doi.org/10.1007/s10803-015-2455-0.Google Scholar
  46. Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., et al. (2011). Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health and Mental Health Services Research, 38(2), 65–76.  https://doi.org/10.1007/s10488-010-0319-7.Google Scholar
  47. Ryan, R., & Deci, E. (2000). Self-determination theory and the facilitation of intrinsic motivation. American Psychologist, 55(1), 68–78.  https://doi.org/10.1037/0003-066X.55.1.68.Google Scholar
  48. Shanock, L. R., Baran, B. E., Gentry, W. A., Clever Pattison, S., & Heggestad, E. D. (2014). Erratum to: Polynomial regression with response surface analysis: A powerful approach for examining moderation and overcoming limitations of difference scores. Journal of Business and Psychology, 29(1), 161.  https://doi.org/10.1007/s10869-013-9317-6.Google Scholar
  49. Shanock, L. R., Baran, B. E., Gentry, W. A., Pattison, S. C., & Heggestad, E. D. (2010). Polynomial regression with response surface analysis: A powerful approach for examining moderation and overcoming limitations of difference scores. Journal of Business and Psychology, 25(4), 543–554.  https://doi.org/10.1007/s10869-010-9183-4.Google Scholar
  50. Smith, T., Groen, A. D., & Wynn, J. W. (2000). Randomized trial of intensive early intervention for children with pervasive developmental disorder. American Journal on Mental Retardation, 105(4), 269–285.Google Scholar
  51. Stahmer, A. C., Reith, S., Lee, E., Reisingger, E. M., Mandell, D. S., & Connell, J. E. (2015). Training teachers to use evidence-based practices for autism: Examining procedural implementation fidelity. Psychology in Schools, 52(2), 181–195.  https://doi.org/10.1002/pits.Google Scholar
  52. Suhrheinrich, J., Stahmer, A. C., Reed, S., Schreibman, L., Reisinger, E., & Mandell, D. (2013). Implementation challenges in translating pivotal response training into community settings. Journal of Autism and Developmental Disorders, 43(12), 2970–2976.  https://doi.org/10.1007/s10803-013-1826-7.Google Scholar
  53. Weiner, B. J., Belden, C. M., Bergmire, D. M., & Johnston, M. (2011). The meaning and measurement of implementation climate. Implementation Science, 6(1), 78.  https://doi.org/10.1186/1748-5908-6-78.Google Scholar
  54. Whitaker, K., Fortier, A., Bruns, E. J., Nicodimos, S., Ludwig, K., Lyon, A. R., et al. (2018). How do school mental health services vary across contexts? Lessons learned from two efforts to implement a research-based strategy. School Mental Health, 10(2), 134–146.  https://doi.org/10.1007/s12310-017-9243-2.Google Scholar
  55. Wilson, S. J., Lipsey, M. W., & Derzon, J. H. (2003). The effects of school-based intervention programs on aggressive behavior: A meta-analysis. Journal of Consulting and Clinical Psychology, 71(1), 136–149.  https://doi.org/10.1037/0022-006X.71.1.136.Google Scholar
  56. Yammarino, F. J., & Atwater, L. E. (1993). Understanding self-perception accuracy: Implications for human resource management. Human Resource Management, 32(2–3), 231–247.  https://doi.org/10.1002/hrm.3930320204.Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.University of WashingtonSeattleUSA
  2. 2.University of PennsylvaniaPhiladelphiaUSA
  3. 3.University of Central FloridaOrlandoUSA
  4. 4.University of PittsburghPittsburghUSA
  5. 5.Department of PsychologyUniversity of WashingtonSeattleUSA

Personalised recommendations