School Mental Health

, Volume 9, Issue 3, pp 273–283 | Cite as

Screening for Student Mental Health Risk: Diagnostic Accuracy, Measurement Invariance, and Predictive Validity of the Social, Academic, and Emotional Behavior Risk Screener-Student Rating Scale (SAEBRS-SRS)

  • Nathaniel P. von der Embse
  • Stephen P. Kilgus
  • Stephanie Iaccarino
  • Shana Levi-Nielsen
Original Paper
First Online:

Abstract

Schools have become the largest provider of mental health services to children, thus increasing the need to identify children at risk before problems worsen. The Social, Academic, and Emotional Behavioral Risk Screener-Student Rating Scale (SAEBRS-SRS) student version is a universal screener that assesses pre-symptomology of internalizing and externalizing behaviors. Prior research on the SAEBRS-SRS has supported a psychometrically defensible factor structure; however, additional research is necessary to support applied used in schools. In this study, 1102 middle grades students completed the SAEBRS-SRS and the Strengths and Difficulties Questionnaire. Data were analyzed to evaluate the diagnostic accuracy of the screener via receiver operating characteristic curve analyses. Multi-group confirmatory factor analysis was conducted to identify measurement invariance across gender. Results indicated that (1) the SAEBRS-SRS has adequate diagnostic accuracy statistics, particularly for the SAEBRS Total score, and (2) the SAEBRS-SRS is invariant across gender at the metric and scalar levels of measurement invariance. Implications and recommendations for future research are further discussed.

Keywords

Universal screening Rating scale Mental health Behavioral assessment 
This is a preview of subscription content, log in to check access.

Notes

Compliance with Ethical Standards

Conflict of interest

Nathaniel P. von der Embse and Stephen P. Kilgus have a financial conflict of interest due to receipt of royalties on net sales for one of the measures discussed.

Ethical Approval

The American Psychological Association ethical standards were completely adhered to throughout the development of this manuscript. All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. All consent and assent procedures were approved by both the University Institutional Research Review Board and the local school district superintendent.

References

  1. Atkins, M. S., Hoagwood, K. E., Kutash, K., & Seidman, E. (2010). Toward the integration of education and mental health in schools. Administration and Policy in Mental Health and Mental Health Services Research, 37(1–2), 40–47.CrossRefPubMedPubMedCentralGoogle Scholar
  2. Barrett, S., Eber, L., & Weist, M. D. (2013). Advancing education effectiveness: An interconnected systems framework for positive behavioral interventions and supports (PBIS) and school mental health. Center for positive behavioral interventions and supports. Eugene, OR: University of Oregon Press.Google Scholar
  3. Borsboom, D. (2006). When does measurement invariance matter? Medical care, 44(11), S176−S181.CrossRefPubMedGoogle Scholar
  4. Bradley, R., Doolittle, J., & Bartolotta, R. (2008). Building on the data and adding to the discussion: The experiences and outcomes of students with emotional disturbance. Journal of Behavioral Education, 17(1), 4–23.CrossRefGoogle Scholar
  5. Bruhn, A. L., Woods-Groves, S., & Huddle, S. (2014). A preliminary investigation of emotional and behavioral screening practices in K-12 schools. Education and Treatment of Children, 37, 611–634.CrossRefGoogle Scholar
  6. Burns, M. K., Riley-Tillman, T. C., & Van Der Heyden, A. M. (2012). RTI applications, volume 1: Academic and behavioral interventions (Vol. 1). New York City: Guilford Press.Google Scholar
  7. Cappella, E., Frazier, S. L., Atkins, M. S., Schoenwald, S. K., & Glisson, C. (2008). Enhancing schools’ capacity to support children in poverty: An ecological model of school-based mental health services. Administration and Policy in Mental Health and Mental Health Services Research, 35(5), 395–409.CrossRefPubMedPubMedCentralGoogle Scholar
  8. Carran, D. T., & Scott, K. G. (1992). Risk assessment in preschool children: Research implications for the early detection of educational handicaps. Topics in Early Childhood Special Education, 12, 196–211. doi: 10.1177/027112149201200205.CrossRefGoogle Scholar
  9. Chen, F. F. (2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling, 14, 464–504. doi: 10.1080/10705510701301834.CrossRefGoogle Scholar
  10. Cheung, G. W., & Rensvold, R. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 9(2), 233–255. doi: 10.1207/S15328007SEM0902_5.CrossRefGoogle Scholar
  11. Cohen, J. (1968). Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological Bulletin, 70, 213–220. doi: 10.1037/h0026256.CrossRefPubMedGoogle Scholar
  12. Cook, C. R., Volpe, R. J., & Livanis, A. (2010). Constructing a roadmap for future universal screening research beyond academics. Assessment for Effective Intervention, 35(4), 197–205.CrossRefGoogle Scholar
  13. Cortina, J. M. (1993). What is coefficient alpha? An examination of theory and applications. Journal or Applied Psychology, 78, 98–104.CrossRefGoogle Scholar
  14. De Los Reyes, A., Augenstein, T. M., Wang, M., Thomas, S. A., Drabick, D. A., Burgers, D. E., et al. (2015). The validity of the multi-informant approach to assessing child and adolescent mental health. Psychological Bulletin, 141, 858–900. doi: 10.1037/a0038498.CrossRefPubMedPubMedCentralGoogle Scholar
  15. Dever, B. V., Kamphaus, R. W., Dowdy, E., Raines, T. C., & DiStefano, C. (2013). Surveillance of middle and high school mental health risk by student self-report screener. Western Journal of Emergency Medicine: Integrating Emergency Care with Population Health, 14(4), 384–390. doi: 10.5811/westjem.2013.2.15349.CrossRefGoogle Scholar
  16. Dowdy, E., Doane, K., Eklund, K., & Dever, B. V. (2011). A comparison of teacher nomination and screening to identify behavioral and emotional risk within a sample of underrepresented students. Journal of Emotional and Behavioral Disorders. doi: 10.1177/1063426611417627.Google Scholar
  17. Dowdy, E., & Kim, E. (2012). Choosing informants when conducting a universal screening for behavioral and emotional risk. School Psychology Forum, 6, 1–10.Google Scholar
  18. Drummond, T. (1994). The student risk screening scale (SRSS). Grants Pass, OR: Josephine County Mental Health Program.Google Scholar
  19. Flanagan, K. S., Bierman, K. L., & Kam, C. M. (2003). Identifying at-risk children at school entry: The usefulness of multibehavioral problem profiles. Journal of Clinical Child and Adolescent Psychology, 32(3), 396–407.CrossRefPubMedPubMedCentralGoogle Scholar
  20. Glover, T. A., & Albers, C. A. (2007). Considerations for evaluating universal screening assessments. Journal of School Psychology, 45(2), 117–135.CrossRefGoogle Scholar
  21. Goodman, R. (1997). The Strengths and Difficulties Questionnaire: A research note. Journal of Child Psychology and Psychiatry, 38, 581–586. doi: 10.1111/j.1469-7610.1997.tb01545.x.CrossRefPubMedGoogle Scholar
  22. Goodman, R. (2001). Psychometric properties of the Strengths and Difficulties Questionnaire. Child and Adolescent Psychiatry, 40(11), 1137–1145.CrossRefGoogle Scholar
  23. Goodman, A., & Goodman, R. (2009). Strengths and Difficulties Questionnaire as a dimensional measure of child mental health. Journal of the American Academy of Child and Adolescent Psychiatry, 48, 400–403.CrossRefPubMedGoogle Scholar
  24. Goodman, R., & Scott, S. (1999). Comparing the Strengths and Difficulties Questionnaire and the child behavior checklist: Is small beautiful? Journal of Abnormal Child Psychology, 27, 17–24.CrossRefPubMedGoogle Scholar
  25. Hemphill, J. F. (2003). Interpreting the magnitude of correlation coefficients. American Psychologist, 58, 78–79.CrossRefPubMedGoogle Scholar
  26. Hintze, J. M., & Silberglitt, B. (2005). A longitudinal examination of the diagnostic accuracy and predictive validity of R-CBM and high-stakes testing. School Psychology Review, 34, 372–386.Google Scholar
  27. Jenkins, L. N., Demaray, M. K., Wren, N. S., Secord, S. M., Lyell, K. M., Magers, A. M., et al. (2014). A critical review of five commonly used social-emotional and behavioral screeners for elementary or secondary schools. Contemporary School Psychology, 18, 241–254. doi: 10.1007/s40688-014-0026-6.CrossRefGoogle Scholar
  28. Kamphaus, R. W. (2012). Screening for behavioral and emotional risk: Constructs and practicalities. School Psychology Forum, 6, 89–97.Google Scholar
  29. Kamphaus, R. W., & Reynolds, C. R. (2007). BASC-2 behavioral and emotional screening system manual. Bloomington: Pearson.Google Scholar
  30. Kilgus, S. P., Chafouleas, S. M., & Riley-Tillman, T. C. (2013). Development and initial validation of the social and academic behavior risk screener for elementary grades. School Psychology Quarterly, 28, 210–226. doi: 10.1037/spq0000024.CrossRefPubMedGoogle Scholar
  31. Kilgus, S. P., Eklund, K., von der Embse, N. P., Taylor, C. N., & Sims, W. A. (2016). Psychometric defensibility of the Social, Academic, and Emotional Behavior Risk Screener (SAEBRS) teacher rating scale and multiple gating procedure within elementary and middle school samples. Journal of School Psychology, 58, 21–39. doi: 10.1016/j.jsp.2016.07.001 CrossRefPubMedGoogle Scholar
  32. Kilgus, S. P., Riley-Tillman, T. C., Chafouleas, S. M., Christ, T. J., & Welsh, M. E. (2014). Direct behavior rating as a school-based behavior universal screener: Replication across sites. Journal of School Psychology, 52, 63–82.CrossRefPubMedGoogle Scholar
  33. Kilgus, S. P., Sims, W., von der Embse, N. P., & Riley-Tillman, T. C. (2015). Confirmation of models for interpretation and use of the social and academic behavior risk screener (SABRS). School Psychology Quarterly, 30, 335–352. doi: 10.1037/spq0000087.CrossRefPubMedGoogle Scholar
  34. Kilgus, S. P., Sims, W., von der Embse, N. P., & Taylor, C. (2016). Technical adequacy of the social, academic, and emotional behavior risk screener in an elementary sample. Assessment for Effective Intervention, 42(1), 46–59. doi: 10.1177/153450841562326.CrossRefGoogle Scholar
  35. Kilgus, S. P., & von der Embse, N. P. (2015). Social, Academic, and Emotional Behavior Risk Screener (SAEBRS). Minneapolis, MN: Theodore J. Christ & Colleagues.Google Scholar
  36. Kline, R. B. (2011). Principles and practice of structural equation modeling (3rd ed.). New York, NY: Guilford Press.Google Scholar
  37. Lane, K. L., Little, M. A., Casey, A. M., Lambert, W., Wehby, J., Weisenbach, J. L., et al. (2009). A comparison of systematic screening tools for emotional and behavioral disorders. Journal of Emotional and Behavioral Disorders, 17, 93–105. doi: 10.1177/1063426608326203.CrossRefGoogle Scholar
  38. Lane, K. L., Oakes, W. P., Ennis, R. P., & Hirsch, S. E. (2014). Identifying students for secondary and tertiary prevention efforts: How do we determine which students have Tier 2 and Tier 3 needs? Preventing School Failure: Alternative Education for Children and Youth, 58(3), 171–182.CrossRefGoogle Scholar
  39. Levitt, J. M., Saka, N., Hunter Romanelli, L., & Hoagwood, K. (2007). Early identification of mental health problems in schools: The status of instrumentation. Journal of School Psychology, 45, 163–191. doi: 10.1016/j.jsp.2006.11.005.CrossRefGoogle Scholar
  40. McIntosh, K., Ty, S. V., & Miller, L. D. (2014). Effects of school-wide positive behavioral interventions and supports on internalizing problems current evidence and future directions. Journal of Positive Behavior Interventions, 16(4), 209–218. doi: 10.1177/1098300713491980.CrossRefGoogle Scholar
  41. Merikangas, K. R., He, J. P., Burstein, M., Swanson, S. A., Avenevoli, S., Cui, L., et al. (2010). Lifetime prevalence of mental disorders in US adolescents: Results from the national comorbidity survey replication-adolescent supplement (NCS-A). Journal of the American Academy of Child and Adolescent Psychiatry, 49(10), 980–989. doi: 10.1016/j.jaac.2010.05.017.CrossRefPubMedPubMedCentralGoogle Scholar
  42. Merikangas, K. R., Nakamura, E. F., & Kessler, R. C. (2009). Epidemiology of mental disorders in children and adolescents. Dialogues in Clinical Neuroscience, 11(1), 7–20.PubMedPubMedCentralGoogle Scholar
  43. Metz, C. E. (1978). Basic principles of ROC analysis. Seminars in Nuclear Medicine, 8, 283–298.CrossRefPubMedGoogle Scholar
  44. Miller, F. G., Cohen, D., Chafouleas, S. M., Riley- Tillman, T. C., Welsh, M. E., & Fabiano, G. A. (2015). A comparison of measures to screen for social, emotional, and behavioral risk. School Psychology Quarterly, 30, 184–196.CrossRefPubMedGoogle Scholar
  45. Mills, C., Stephan, S. H., Moore, E., Weist, M. D., Daly, B. P., & Edwards, M. (2006). The President’s New Freedom Commission: Capitalizing on opportunities to advance school-based mental health services. Clinical Child and Family Psychology Review, 9, 149–161.CrossRefPubMedGoogle Scholar
  46. Muthén, L. K. (2012). Model fit index WRMR. Retrieved November 23, 2012, from http://www.statmodel.com/discussion/messages/9/5096.html?1321986275.
  47. Muthén, L. K., & Muthén, B. O. (1998–2013). Mplus (Version 7.1) [Computer software]. Los Angeles, CA: Muthén & Muthén.Google Scholar
  48. Pendergast, L., von der Embse, N. P., Kilgus, S., & Eklund, K. (2017). Measurement equivalence/invariance in school psychology research: A conceptual overview and example of multi-group confirmatory factor analysis for non-statisticians. Journal of School Psychology, 60, 65–82. doi: 10.1016/j.jsp.2016.11.002.CrossRefPubMedGoogle Scholar
  49. Perou, R., Bitsko, R. H., Blumberg, S. J., Pastor, P., Ghandour, R. M., Gfroerer, J. C., et al. (2013). Mental health surveillance among children—United States, 2005–2011. MMWR Surveillance Summary, 62(2), 1–35.Google Scholar
  50. Petscher, Y., Kim, Y., & Foorman, B. R. (2011). The importance of predictive power in early screening assessments: Implications for placement in the response to intervention framework. Assessment for Effective Intervention, 36, 158–166.CrossRefPubMedPubMedCentralGoogle Scholar
  51. Rice, M. E., & Harris, G. T. (2005). Comparing effect sizes in follow-up studies: ROC area, Cohen’s d, and r. Law and Human Behavior, 29(5), 615–620. doi: 10.1007/s10979-005-6832-7.CrossRefPubMedGoogle Scholar
  52. Serbin, L. A., Temcheff, C. E., Cooperman, J. M., Stack, D. M., Ledingham, J., & Schwartzman, A. E. (2011). Predicting family poverty and other disadvantaged conditions for child rearing from childhood aggression and social withdrawal: A 30-year longitudinal study. International Journal of Behavioral Development, 35, 97–106. doi: 10.1177/0165025410372008.CrossRefGoogle Scholar
  53. Severson, H. H., Walker, H. M., Hope-Doolittle, J., Kratochwill, T. R., & Gresham, F. M. (2007). Proactive, early screening to detect behaviorally at-risk students: Issues, approaches, emerging innovations, and professional practices. Journal of School Psychology, 45(2), 193–223.CrossRefGoogle Scholar
  54. Smith, S. R. (2007). Making sense of multiple informants in child and adolescent psychopathology: A guide for clinicians. Psychoeducational Assessment, 25, 139–149. doi: 10.1177/0734282906296233.CrossRefGoogle Scholar
  55. Streiner, D. L., & Cairney, J. (2007). What’s under the ROC? An introduction to receiver operating characteristic curves. The Canadian Journal of Psychiatry/La Revue Canadienne de Psychiatrie, 52, 121–128.CrossRefGoogle Scholar
  56. Tabachnick, B. G., & Fidell, L. S. (2013). Using multivariate statistics (6th ed.). Boston, MA: Pearson Education.Google Scholar
  57. Taylor, C., Allen, A., Kilgus, S., von der Embse, N., & Garbacz, S. A. (under review). Development and validation of a parent version of the Social, Academic, and Emotional Behavior Risk Screener (SAEBRS). Assessment for Effective Intervention.Google Scholar
  58. von der Embse, N. P. & Kilgus, S. P. (2015). Unpublished technical manual of the Social, Academic, and Emotional Behavior Risk Screener-Student Rating Scale. Minneapolis, MN: Fastbridge Learning.Google Scholar
  59. von der Embse, N.P., Iaccarino, S., Mankin, A., Kilgus, S. P., & Magen, E. (in press). Development and Validation of the Social, Academic, and Emotional Behavior Risk Screener–Student Rating Scale. Assessment for Effective Intervention, 1534508416679410.Google Scholar
  60. von der Embse, N. P., Pendergast, L., Kilgus, S. P., & Eklund, K. R. (2016). Evaluating the applied use of a mental health screener: Structural validity of the Social, Academic, and Emotional Behavior Risk Screener. Psychological Assessment. doi: 10.1037/pas0000253.Google Scholar
  61. Yu, C-Y. (2002). Evaluating cutoff criteria of model fit indices for latent variable models with binary and continuous outcomes. Doctoral Dissertation. https://www.statmodel.com/download/Yudissertation.pdf.

Copyright information

© Springer Science+Business Media New York 2017

Authors and Affiliations

  1. 1.University of South FloridaTampaUSA
  2. 2.University of MissouriColumbiaUSA
  3. 3.College of EducationTemple UniversityPhiladelphiaUSA

Personalised recommendations