In the Same Ballpark or a Whole New Ball Game? Staff as Raters of Youth Behavior
There is a lack of psychometrically sound tools for measuring youth outcomes in out-of-school time (OST) settings. Consequently, behavior ratings completed by OST staff are being scored as though the raters were teachers, even though cross-informant correlations are notoriously low (meta-analysis r = .27). Across 26 schools, 227 students were assessed by both teachers and OST staff using the Devereux Student Strengths Assessment (DESSA) to measure Social Emotional Competence. These 4th and 5th grade students were 50% male; 53% 5th graders; and 51% Latino, 20% Mixed/Other, 11% Black, 11% Asian, and 7% White. In the full sample, OST staff rated children’s behavior more harshly than teachers (p < .001; d = .32), although the scores were associated (r = .31, p < .001). Among the ratings completed within the same week, teacher and staff distributions were not statistically different. Teacher and staff ratings had a “medium” correlation (r = .42; p = .01) and a classification consistency (88%) that exceeded chance by a “moderate” amount (κ = .43). Few, if any, studies have previously compared the ratings of the same children by teachers and OST providers. Cross-informant inter-rater reliability between teachers and OST staff was higher than expected on the DESSA.
KeywordsSocial emotional competence Out-of-school time After-school DESSA Cross-informant inter-rater reliability Assessment and outcome measures
This project was supported by the Kaiser Permanente Northern California Community Benefit Program. The authors wish to acknowledge Jennette Claassen and all of the Playworks staff who participated in this project, and specifically the coaches, program managers, program associates, and program directors from the Silicon Valley and San Francisco offices. The authors also wish to thank Paul LeBuffe (Devereux Center for Resilient Children), Jack Naglieri & Chavaughn Brown (George Mason University), Jessica Adamson and Scott Marshall (Apperson Evo Social & Emotional), and Mechelle Timmons (CASO, Inc.) who were each generous and creative enablers of this work.
Compliance with Ethical Standards
Conflicts of Interest
Two of the authors are employees of the Devereux Center for Resilient Children, the non-profit organization that developed the Devereux Student Strengths Assessment (DESSA). Although one author of this paper had a role in the development of the DESSA, no author receives any direct financial remuneration from the sale of the DESSA or any other tool or resource mentioned within this manuscript.
Data analyzed in this paper were collected for evaluation purposes at a non-profit agency and de-identified data were obtained by the university through a data-share agreement, using procedures in accordance with the ethical standards of the Committee for Protection of Human Subjects (CPHS) at the University of California, Berkeley.
A consent waiver was granted by the Committee for the Protection of Human Subjects (CPHS) at the University of California, Berkeley.
- Afterschool Alliance. (2014). America After 3PM. http://www.afterschoolalliance.org/documents/AA3PM-2014/AA3PM_National_Report.pdf.
- Atlas, J. A. (2010). Test review of the Devereux Student Strengths Assessment. In R. A. Spies, J. F. Carlson, K. F. Geisinger (Eds.), The eighteenth mental measurements yearbook (pp. 178–180). Lincoln, NE: Buros Center for Testing.Google Scholar
- Chain J., Shapiro, V.B., LeBuffe, P.A., & Bryson, A.M. (2012, August). Indigenous cultural differences in social emotional competence and academic achievement. Poster session presented at the meeting of the Society for the Psychological Study of Ethnic Minority Issues of the American Psychological Association, Orlando, FL.Google Scholar
- Cohen, J. (1988). Statistical power analysis for the behavioral sciences. 2nd edn. Hillsdale, NJ: Erlbaum.Google Scholar
- Denham, S. A., Ji, P., & Hamre, B. (2010). Compendium of preschool through elementary social-emotional learning and associated assessment measures. Chicago, IL: Collaborative for Academic, Social, and Emotional Learning.Google Scholar
- Dowdy, E., Furlong, M., Eklund, K., Saeki, E., & Ritchey, K. (2010). Screening for mental health and wellness: Current school-based practices and emerging possibilities. In B. Doll, W. Phofl, J. Yoon (Eds.), Handbook of youth prevention science (pp. 70–95). New York, NY: Routledge.Google Scholar
- Gullotta, T. P. (2015). After-school programming and SEL. In J. A. Durlak, C. E. Domitrovich, R. P. Weissberg, T. P. Gullotta (Eds.), Handbook of social and emotional learning: Research and practice. New York, NY: Guilford.Google Scholar
- Haggerty, K., Elgin, J., & Woolley, A. (2011). Social-emotional learning and school climate assessment measures for middle school youth. Retrieved from the Raikes Foundation website: http://www.raikesfoundation.org/
- Huebner, E. S., Gilman, R., & Furlong, M. J. (2009). A conceptual model for research in positive psychology in children and youth. Handbook of positive psychology in schools. In M. J. Furlong, R. Gilman, E. S. Huebner (Eds.), Handbook of positive psychology in schools (pp. 3–8). New York, NY: Routledge.Google Scholar
- Johnson, S., Hollis, C., Marlow, N., Simms, V., & Wolke, D. (2014). Screening for childhood mental health disorders using the Strengths and Difficulties Questionnaire: The validity of multi-informant reports. Developmental Medicine & Child Neurology, 56, 453–459. doi: 10.1111/dmcn.12360.CrossRefGoogle Scholar
- LeBuffe, P. A., Shapiro, V. B., & Naglieri, J. A. (2014). The Devereux Student Strengths Assessment (DESSA): Assessment, technical manual, and user’s guide. Charlotte, NC: Apperson, Inc.(Original work published 2009).Google Scholar
- Malcomb, K. K. (2010). Test review of the Devereux Student Strengths Assessment. In R. A. Spies, J. F. Carlson, K. F. Geisinger (Eds.), The eighteenth mental measurements yearbook (pp. 180–182). Lincoln, NE: Buros Center for Testing.Google Scholar
- McKown, C. (2015). Challenges and opportunities in the direct assessment of children’s social and emotional comprehension. In J. A. Durlak, C. E. Domitrovich, R. P. Weissberg, T. P. Gullotta (Eds.), Handbook of social and emotional learning: Research and practice (pp.320–335). New York, NY: Guilford.Google Scholar
- Merrell, K. W., & Gueldner, B. A. (2010). Social and emotional learning in the classroom: promoting mental health and academic success. New York, NY: Guilford Press.Google Scholar
- Rosas, S., Chaiken, L., & Case, J. (2008). Cross-setting consistent and setting-specific strength behaviors in preschoolers: Influence on reported concerns. In C. Newman, C. Liberton, K. Kutash, R. Friedman (Eds.). A system of care for children’s mental health: Expanding the research base, 20 (pp. 201–204). Tampa: University of South Florida, Louis de la Parte Florida Mental Health Institute, Research and Training Center for Children’s Mental Health.Google Scholar
- Shapiro, V. B., Accomazzo, S., Claassen, J., & Robitaille, J. L. (2015). The choices, challenges, and lessons learned from a multi-method social-emotional/character assessment in an out of school time setting. Journal of Youth Development: Bridging Research and Practice, 10(3), 32–45.Google Scholar
- Shapiro, V.B., Kim, B.K.E., Robitaille, J.L., & LeBuffe, P.A. (2016). Protective factor screening for prevention practice: Sensitivity and specificity of the DESSA-Mini. School Psychology Quarterly. Advance online publication. http://dx.doi.org/10.1037/spq0000181.
- Simmons, C. A., Shapiro, V. B., Accomazzo, S., & Manthey, T. J. (2016). Strengths-based social work: A meta-theory to guide social work research and practice. In N. Coady, P. Lehmann (Eds.), Theoretical perspectives for direct social work practice. 3rd edn. (pp. 131–154). New York, NY: Springer Publishing Company.Google Scholar
- StataCorp. (2013). Stata statistical software: Release 13. College Station, TX: StataCorp LP.Google Scholar
- Tsang, K. L. V., Wong, P. Y. H., & Lo, S. K. (2012). Assessing psychosocial well-being of adolescents: A systematic review of measurement instruments. Child: Care, Health and Development, 38(35), 629–646.Google Scholar