Skip to main content
SpringerLink
Log in

Surveillance of Internalizing Behaviors: A Reliability and Validity Generalization Study of Universal Screening Evidence

  • Original Paper
  • Published:
School Mental Health Aims and scope Submit manuscript

Abstract

Due to early onset and rising prevalence rates, there is an established need for early identification of youth demonstrating symptoms of internalizing disorders (e.g., anxiety and depression). Accordingly, calls have been made for the use of universal screening tools to identify internalizing behavior risk. Current research has spoken to the adequacy of individual screeners, but researchers have yet to quantitatively synthesize the broader internalizing screening literature. The purpose of the current study was to systematically review the literature and conduct a reliability and validity generalization meta-analysis of available psychometric evidence. This meta-analysis identified 40 studies in which 18 distinct internalizing behavior screeners were examined. Of interest were both (a) broadband screeners, which included internalizing items along with items specific to other areas (e.g., externalizing problems), and (b) narrowband screeners, which included internalizing items alone. Across examined studies, mean coefficient alpha reliability estimates were equal to .84 for broadband screeners and .78 for narrowband screeners. For broadband screeners, mean validity correlation coefficients were equal to .66 when predicting other internalizing measures and .51 when predicting other criteria (e.g., extant academic and behavioral data). For narrowband screeners, mean validity coefficients were equal to .45 when predicting other internalizing measures and .26 when predicting other criteria. Interestingly, moderator analyses indicated neither reliability nor validity was moderated by any study or measurement characteristics. Taken together, results supported the psychometric defensibility of both broadband and narrowband internalizing screeners. Limitations of the study and implications for research are discussed, including recommendations for additional diagnostic accuracy research.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

References marked with an asterisk indicate studies included in the meta-analysis

  • Achenbach, T. M., & Edelbrock, C. (1978). The classification of child psychopathology: A review and analysis of empirical efforts. Psychological Bulletin, 85, 1275–1301. https://doi.org/10.1037/0033-2909.85.6.1275.

    Article  PubMed  Google Scholar 

  • *Adriaansee, M., van Domburgh, L., Zwirs, B., Doreleijers, T., & Veling, W. (2015). School-based screening for psychiatric disorders in Moroccan-Dutch youth. Child and Adolescent Psychiatry and Mental Health, 9, 13–30. https://doi.org/10.1186/s13034-015-0045-8.

    Article  Google Scholar 

  • *Albayrak-Kaymak, D. (1999). Internalizing or externalizing: Screening for both problem youth. International Journal for the Advancement of Counseling, 21, 125–137. https://doi.org/10.1023/A:1005412219815.

    Article  Google Scholar 

  • American Psychiatric Association. (2000). Diagnostic and statistical manual of mental disorders (4th ed.). Arlington, VA: American Psychiatric Association.

    Google Scholar 

  • American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed.). Arlington, VA: American Psychiatric Association.

    Book  Google Scholar 

  • Barnes, L. L., Harp, D., & Jung, W. S. (2002). Reliability generalization of scores on the Spielberger state-trait anxiety inventory. Educational and Psychological Measurement, 62, 603–618. https://doi.org/10.1177/0013164402062004005.

    Article  Google Scholar 

  • Bayer, J. K., Rapee, R. M., Hiscock, H., Ukoumunne, O. C., Mihalopoulos, C., Clifford, S., et al. (2011). The Cool Little Kids randomized controlled trial: Population-level early prevention for anxiety disorders. BioMed Central Public Health, 11, 2–9. https://doi.org/10.1186/1471-2458-11-11.

    Article  Google Scholar 

  • *Bourdon, K. H., Goodman, R., Rae, D. S., Simpson, G., & Koretz, D. S. (2005). The Strengths and Difficulties Questionnaire: U.S. normative data and psychometric properties. Journal of the American Academy of Child Adolescent Psychiatry, 44, 557–564. https://doi.org/10.1097/01.chj.0000159157.57075c8.

    Article  PubMed  Google Scholar 

  • Bradshaw, C. P., Buckley, J. A., & Ialongo, N. S. (2008). School-based service utilization among urban children with early onset educational and mental health problems: The squeaky wheel phenomenon. School Psychology Quarterly, 23, 169–186. https://doi.org/10.1037/1045-3830.23.2.169.

    Article  Google Scholar 

  • Bruhn, A. L., Woods-Groves, S., & Huddle, S. (2014). A preliminary investigation of emotional and behavioral screening practices in K-12 schools. Education and Treatment of Children, 37, 611–634. https://doi.org/10.1353/etc.2014.0039.

    Article  Google Scholar 

  • Chafouleas, S. M., Kilgus, S. P., & Wallach, N. (2010). Ethical dilemmas in school-based behavioral screening. Assessment for Effective Intervention, 35, 245–252. https://doi.org/10.1177/1534508410379002.

    Article  Google Scholar 

  • Coie, J. D., & Dodge, K. A. (1998). Aggression and antisocial behavior. In W. Damon & N. Eisenberg (Eds.), Handbook of child psychology: Social, emotional, and personality development (Vol. 3, pp. 779–862). New York, NY: Wiley.

    Google Scholar 

  • Compton, S., Burns, B., Egger, H., & Robertson, E. (2002). Review of the evidence base for treatment of childhood psychopathology: Internalizing disorders. Journal of Consulting and Clinical Psychology, 70, 1240–1266. https://doi.org/10.1037/0022-006X.70.6.1240.

    Article  PubMed  Google Scholar 

  • *Cook, C. R., Rasetshwane, K. B., Truelson, E., Grant, S., Dart, E. H., Collins, T. A., et al. (2011). Development and validation of the Student Internalizing Behavior Screener: Examination of reliability, validity, and classification accuracy. Assessment for Effective Intervention, 36, 71–79. https://doi.org/10.1177/1534508410390486.

    Article  Google Scholar 

  • Cortina, J. M. (1993). What is coefficient alpha? An examination of theory and applications. Journal or Applied Psychology, 78, 98–104. https://doi.org/10.1037/0021-9010.78.1.98.

    Article  Google Scholar 

  • Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. New York: Holt, Reinhart, & Winston.

    Google Scholar 

  • Cullinan, D., & Epstein, M. H. (2013). Emotional and behavioral screener (EBS). Austin, TX: PRO-ED.

    Google Scholar 

  • *Davis, S. D. (2012). Teacher nominations and the identification of social, emotional, and behavioral concerns in adolescence (Doctoral dissertation). Retrieved Nov 1, 2017, from https://www.proquest.com/.

  • *Deighton, J., Tymms, P., Vostanis, P., Belsky, J., Fonagy, P., Brown, A., et al. (2013). The development of a school-based measure of child mental health. Journal of Psychoeducational Assessment, 31, 247–257. https://doi.org/10.1177/0734282912465570.

    Article  PubMed  PubMed Central  Google Scholar 

  • Dever, B. V., Raines, T. C., & Barclay, C. M. (2012). Chasing the unicorn: Practical implementation of universal screening for behavioral and emotional risk. School Psychology Forum, 6, 108–118. Retrieved Nov 1, 2017, from http://www.nasponline.org/publications/periodicals/spf/volume-6/volume-6-issue-4-(winter-2012).

  • *DiStefano, C., & Morgan, G. (2011). Examining classification criteria: A comparison of three cut score methods. Psychological Assessment, 23, 354–363. https://doi.org/10.1037/a0021745.

    Article  PubMed  Google Scholar 

  • Doll, B. (2008). The dual-factor model of mental health in youth. School Psychology Review, 37, 69–73. Retrieved Nov 1, 2017, from http://www.nasponline.org/publications/periodicals/spr/volume-37/volume-37-issue-1.

  • *Dowdy, E., Kamphaus, R. W., Abdou, A. S., & Twyford, J. M. (2012). Detection of symptoms of prevalent mental health disorders of childhood with the parent form of the Behavioral and Emotional Screening System. Assessment for Effective Intervention, 38, 192–198. https://doi.org/10.1177/1534508412447009.

    Article  Google Scholar 

  • Eklund, K., Kilgus, S., von der Embse, N., Beardmore, M., & Tanner, N. (2017). Use of universal screening scores to predict distal academic and behavioral outcomes: A multilevel approach. Psychological Assessment, 29, 486–499. https://doi.org/10.1037/pas0000355.

    Article  PubMed  Google Scholar 

  • Fergusson, D. M., Horwood, L. J., Ridder, E. M., Beautrais, A. L. (2005). Subthreshold depression in adolescence and mental health outcomes in adulthood. Archives of General Psychiatry, 62, 66–72. Retrieved Nov 1, 2017, from https://medlineplus.gov/.

  • Glover, T. A., & Albers, C. A. (2007). Considerations for evaluating universal screening assessments. Journal of School Psychology, 45, 117–135. https://doi.org/10.1016/j.jsp.2006.05.005.

    Article  Google Scholar 

  • Goodman, R. (1997). The Strengths and Difficulties Questionnaire: A research note. Journal of Child Psychology and Psychiatry, 38, 581–586. https://doi.org/10.1111/j.1469-7610.1997.tb01545.x.

    Article  PubMed  Google Scholar 

  • *Goodman, R., & Scott, S. (1999). Comparing the Strengths and Difficulties Questionnaire and the Child Behavior Checklist: Is small beautiful? Journal of Abnormal Child Psychology, 27, 17–24. https://doi.org/10.1023/A:1022658222914.

    Article  PubMed  Google Scholar 

  • *Green, J. G., Gruber, M. J., Sampson, N. A., Zaslavsky, A. M., & Kessler, R. C. (2010). Improving the K6 short scale to predict serious emotional disturbance in adolescents in the USA. International Journal of Methods in Psychiatric Research, 19, 23–35. https://doi.org/10.1002/mpr.314.

    Article  PubMed  PubMed Central  Google Scholar 

  • Green, S. B., Lissitz, R. W., & Mulaik, S. A. (1977). Limitations of coefficient alpha as an index of test unidimensionality. Educational and Psychological Measurement, 37, 827–838. https://doi.org/10.1177/001316447703700403.

    Article  Google Scholar 

  • Gresham, F. M., & Kern, L. (2004). Internalizing behavior problems in children and adolescents. In R. B. Rutherford, M. M. Quinn, & S. R. Mathur (Eds.), Handbook of research in emotional and behavioral disorders (pp. 262–281). New York, NY: Guilford.

    Google Scholar 

  • *He, J. P., Burstein, M., Schmitz, A., & Merikangas, K. R. (2013). The Strengths and Difficulties Questionnaire (SDQ): The factor structure and scale validation in U.S. adolescents. Journal of Abnormal Child Psychology, 41, 583–595. https://doi.org/10.1007/s10802-012-9696-6.

    Article  PubMed  Google Scholar 

  • Hedges, L. V., & Cooper, H. (1994). The handbook of research synthesis. New York, NY: Sage.

    Google Scholar 

  • Henson, R. K., Kogan, L. R., & Vacha-Haase, T. (2001). A reliability generalization study on the Teacher Efficacy Scale and related instruments. Educational and Psychological Measurement, 61, 404–420. https://doi.org/10.1177/00131640121971284.

    Article  Google Scholar 

  • Henson, R. K., & Thompson, B. (2002). Characterizing measurement error in scores across studies: Some recommendations for conducting “reliability generalization” studies. Measurement and Evaluation in Counseling and Development, 35, 113–126. https://doi.org/10.1177/00131640121971284.

    Article  Google Scholar 

  • Herman, K. C., Merrell, K. W., Reinke, W. M., & Tucker, C. M. (2004). The role of school psychology in preventing depression. Psychology in the Schools, 41, 763–775. https://doi.org/10.1002/pits.20016.

    Article  Google Scholar 

  • *Hill, C. R., & Hughes, J. N. (2007). An examination of the convergent and discriminant validity of the Strengths and Difficulties Questionnaire. School Psychology Quarterly, 22, 380–406. Retrieved Nov 1, 2017, from https://medlineplus.gov/.

  • Hunter, K. K., Chenier, J. S., & Gresham, F. M. (2014). Evaluation of check in/check out for students with internalizing behavior problems. Journal of Emotional and Behavioral Disorders, 22, 135–148. https://doi.org/10.1177/1063426613476091.

    Article  Google Scholar 

  • *Janssens, A., & Deboutte, D. (2009). Screening for psychopathology in child welfare: The Strengths and Difficulties Questionnaire (SDQ) compared with the Achenbach System of Empirically Based Assessment (ASEBA). European Child and Adolescent Psychiatry, 18, 691–700. https://doi.org/10.1007/s00787-009-0030-y.

    Article  PubMed  Google Scholar 

  • Jenkins, J. R., Hudson, R. F., & Johnson, E. S. (2007). Screening for at-risk readers in a response to intervention framework. School Psychology Review, 36, 582–600. Retrieved Nov 1, 2017, from http://apps.nasponline.org/resources-and-publications/periodicals/spr-index-list.aspx.

  • Jonsson, A., & Svingby, G. (2007). The use of scoring rubrics: Reliability, validity and educational consequences. Educational Research Review, 2, 130–144. https://doi.org/10.1016/j.edurev.2007.05.002.

    Article  Google Scholar 

  • Lane, K. L. Oakes, W. P., Harris, P. J., Menzies, H. M., Cox, M., & Lambert, W. (2012). Initial evidence for the reliability and validity of the Student Risk Screening Scale for Internalizing and Externalizing Behaviors at the elementary level. Behavior Disorders, 37, 99–122. Retrieved Nov 1, 2017, from http://journals.sagepub.com/home/bhd.

  • Kamphaus, R. W. (2012). Screening for behavioral and emotional risk: Constructs and practicalities. School Psychology Forum, 6, 89–97. Retrieved Nov 1, 2017, from http://www.nasponline.org/publications/periodicals/spf/volume-6/volume-6-issue-4-(winter-2012).

  • *Kamphaus, R. W., DiStefano, C., Dowdy, E., Eklund, K., & Rettig Dunn, A. (2010). Determining the presence of a problem: Comparing two approaches for detecting youth behavioral risk. School Psychology Review, 39, 395–407. Retrieved Nov 1, 2017, from http://apps.nasponline.org/resources-and-publications/periodicals/spr-index-list.aspx.

  • Kamphaus, R. W., & Reynolds, C. R. (2007). BASC-2 behavioral and emotional screening system. Minneapolis, MN: Pearson.

    Google Scholar 

  • Kamphaus, R. W., Reynolds, C. R., & Dever, B. V. (2014). Behavioral and mental health screening. In R. J. Kettler, T. A. Glover, C. A. Albers, & K. A. Feeney-Kettler (Eds.), Universal screening in educational settings: Evidence-based decision making for schools (pp. 249–273). Washington, DC: American Psychological Association.

    Chapter  Google Scholar 

  • Kane, M. T. (2013). Validating the interpretations and uses of tests scores. Journal of Educational Measurement, 50, 1–73. https://doi.org/10.1111/jedm.12000.

    Article  Google Scholar 

  • Kessler, R. C., Andrews, G., Colpe, L. J., Hiripi, E., Mroczek, D. K., Normand, S. L. T., et al. (2002). Short screening scales to monitor population prevalences and trends in non-specific psychological distress. Psychological Medicine, 32, 959–976. https://doi.org/10.1017/S0033291702006074.

    Article  PubMed  Google Scholar 

  • *Kilgus, S. P., Eklund, K., von der Embse, N. P., Taylor, C. N., & Sims, W. A. (2016). Psychometric defensibility of the Social, Academic, and Emotional Behavior Risk Screener (SAEBRS) teacher rating scale and multiple gating procedure within elementary and middle school samples. Journal of School Psychology, 58, 21–39. https://doi.org/10.1016/j.jsp.2016.07.001.

    Article  PubMed  Google Scholar 

  • Kilgus, S. P., Methe, S. A., Maggin, D. M., & Tomasula, J. L. (2014). Curriculum-based measurement oral reading (R-CBM): A diagnostic test accuracy meta-analysis of evidence supporting use in universal screening. Journal of School Psychology, 52, 377–405. https://doi.org/10.1016/j.jsp.2014.06.002.

    Article  PubMed  Google Scholar 

  • Kilgus, S. P., Riley-Tillman, T. C., Chafouleas, S. M., Christ, T. J., & Welsh, M. E. (2014). Direct behavior rating as a school-based behavior universal screener: Replication across sites. Journal of School Psychology, 52, 63–82. https://doi.org/10.1016/j.jsp.2013.11.002.

    Article  PubMed  Google Scholar 

  • *Kilgus, S. P., Sims, W. A., von der Embse, N. P., & Taylor, C. N. (2016). Technical adequacy of the social, academic, and emotional behavior risk screener in an elementary sample. Assessment for Effective Intervention, 42, 46–59. https://doi.org/10.1177/1534508415623269.

    Article  Google Scholar 

  • Kilgus, S. P., & von der Embse, N. P. (2014). Unpublished technical manual of the Social, Academic, and Emotional Behavior Risk Screener.

  • *King, K. R., & Reschly, A. L. (2014). A comparison of screening instruments: Predictive validity of the BESS and BSC. Journal of Psychoeducational Assessment, 32, 687–698. https://doi.org/10.1177/0734282914531714.

    Article  Google Scholar 

  • *King, K., Reschly, A. L., & Appleton, J. J. (2012). An examination of the validity of the Behavioral and Emotional Screening System in a rural elementary school: Validity of the BESS. Journal of Psychoeducational Assessment, 30, 527–538. https://doi.org/10.1177/0734282912440673.

    Article  Google Scholar 

  • *Kiperman, S., Black, M. S., McGill, T. M., Harrell-Williams, L. M., & Kamphaus, R. W. (2014). Predicting Behavior Assessment System for Children—Second Edition Self-Report of Personality Child Form results using the Behavioral and Emotional Screening System Student Form: A replication study with an urban predominantly Latino/a sample. Journal of Psychoeducational Assessment, 32, 587–596. https://doi.org/10.1177/0734282914529200.

    Article  Google Scholar 

  • Klein, D. N., Shankman, S. A., Lewinsohn, P. M., & Seeley, J. R. (2009). Subthreshold depressive disorder in adolescents: Predictors of escalation to full-syndrome depressive disorders. Journal of the American Academy of Child and Adolescent Psychiatry, 48, 703–710. https://doi.org/10.1097/CHI.0b013e3181a56606.

    Article  PubMed  PubMed Central  Google Scholar 

  • *Koskelainen, M., Sourander, A., & Kaljonen, A. (2000). The Strengths and Difficulties Questionnaire among Finnish school-aged children and adolescents. European Child & Adolescent Psychiatry, 9, 277–284. Retrieved Nov 1, 2017, from https://medlineplus.gov/.

  • *Lambert, M. C., Epstein, M. H., & Cullinan, D. (2014). The diagnostic quality of the Emotional and Behavioral Screener. Journal of Psychoeducational Assessment, 32, 51–61. https://doi.org/10.1177/0734282913485541.

    Article  Google Scholar 

  • *Lane, K. L., Menzies, H. M., Oakes, W. P., Lambert, W., Cox, M., & Hankins, K. (2012). A validation of the student risk screening scale for internalizing and externalizing behaviors: Patterns in rural and urban elementary schools. Behavior Disorders, 37, 244–270. Retrieved Nov 1, 2017, from http://journals.sagepub.com/home/bhd.

  • *Lane, K. L., Oakes, W. P., Common, E. A., Zorgian, K., Brunsting, N. C., & Schatschneider, C. (2015). A comparison between SRSS-IE and SSiS-PSG scores: Examining convergent validity. Assessment for Effective Intervention, 40, 114–126. https://doi.org/10.1177/1534508414560346.

    Article  Google Scholar 

  • *Lane, K. L., Oakes, W. P., Ennis, R. P., & Royer, D. J. (2015). Additional evidence of convergent validity between SRSS-IE and SSiS-PSG. Behavior Disorders, 40, 213–229. Retrieved Nov 1, 2017, from http://journals.sagepub.com/home/bhd.

  • Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Thousand Oaks, CA: Sage.

    Google Scholar 

  • Masten, A. S., Roisman, G. I., Long, J. D., Burt, K. B., Obradovic, J., Riley, J. R., et al. (2005). Developmental cascades: Linking academic achievement and externalizing and internalizing symptoms over 20 years. Developmental Psychology, 41, 733–746. https://doi.org/10.1037/0012-1649.41.5.733.

    Article  PubMed  Google Scholar 

  • Merikangas, K. R., He, J., Burstein, M., Swanson, S. A., Avenevoli, S., Cui, L., et al. (2010). Lifetime prevalence of mental disorders in U.S. Adolescents: Results from the national comorbidity survey replication—adolescent supplement (NCS-A). Journal of the American Academy of Child and Adolescent Psychiatry, 49, 980–989. https://doi.org/10.1016/j.jaac.2010.05.017.

    Article  PubMed  PubMed Central  Google Scholar 

  • Merikangas, K. R., Zhang, H., Avenevoli, S., Acharyya, S., Neuenschwander, M., & Angst, J. (2003). Longitudinal trajectories of depression and anxiety in a prospective community study: The Zurich Cohort Study. Archives of General Psychiatry, 60, 993–1000. Retrieved Nov 1, 2017, from https://medlineplus.gov/.

  • Merrell, K. W., & Gueldner, B. A. (2010). Social and emotional learning in the classroom: Promoting mental health and academic success. New York, NY: Guilford Press.

    Google Scholar 

  • *Mewton, L., et al. (2016). The psychometric properties of the Kessler Psychological Distress Scale (K6) in a general population sample of adolescents. Psychological Assessment, 28, 1232–1242. https://doi.org/10.1037/pas0000239.

    Article  PubMed  Google Scholar 

  • *Miller, F. G., Cohen, D., Chafouleas, S. M., Riley-Tillman, T. C., Welsh, M. E., & Fabiano, G. A. (2015). A comparison of measures to screen for social, emotional, and behavioral risk. School Psychology Quarterly, 30, 184–196. https://doi.org/10.1037/spq0000085.

    Article  PubMed  Google Scholar 

  • Miller, F. G., Johnson, A. H., Yu, H., Chafouleas, S. M., McCoach, D. B., Riley-Tillman, T. C., et al. (2018). Methods matter: A multi-trait multi-method analysis of student behavior. Journal of School Psychology, 68, 53–72. https://doi.org/10.1016/j.jsp.2018.01.002.

    Article  PubMed  Google Scholar 

  • *Moriwaki, A., & Kamio, Y. (2014). Normative data and psychometric properties of the strengths and Difficulties Questionnaire among Japanese school-aged children. Child and Adolescent Psychiatry and Mental Health, 8, 1–25. https://doi.org/10.1186/1753-2000-8-1.

    Article  PubMed  PubMed Central  Google Scholar 

  • *Naser, S. (2014). Evaluating the utility of the Behavioral and Emotional Screening System (BESS) as a school-based universal screening tool (Doctoral dissertation). Retrieved Nov 1, 2017, from https://www.proquest.com/.

  • National Association of School Psychologists. (2006). School psychology: A blueprint for training and practice, III. Bethesda, MD: National Association of School Psychologists. Retrieved Nov 1, 2017, from http://www.ww.w.naspcenter.org/blueprint/FinalBlueprintInteriors.pdf.

  • National Institute of Mental Health. (2015). NIMH Strategic Plan for Research (NIH Publication No. 02-2650). Retrieved Nov 1, 2017, from http://www.nimh.nih.gov/about/strategic-planning-reports/index.shtml.

  • National Research Council and Institute of Medicine. (2009). In M. E. O’Connell, T. Boat, & K. E. Warner (Eds.), Preventing mental, emotional, and behavioral disorders among young people: Progress and possibilities. Washington, DC: National Academies Press.

  • *Nordness, P. D., Epstein, M. H., Cullinan, D., & Pierce, C. D. (2014). Emotional and Behavioral Screener: Test–retest reliability, inter-rater reliability, and convergent validity. Remedial and Special Education, 35, 211–217. https://doi.org/10.1177/0741932513497596.

    Article  Google Scholar 

  • *Patalay, P., Deighton, J., Fonagy, P., Vostanis, P., & Wolpert, M. (2014). Clinical validity of the Me and My School questionnaire: A self-report mental health measure for children and adolescents. Child and Adolescent Psychiatry and Mental Health, 8, 17–30. Retrieved Nov 1, 2017, from https://medlineplus.gov/.

  • *Pierce, C. D., Lambert, M., & Alamer, H. (2016). Convergent, criterion and social validity of the Emotional and Behavioral Screener. Journal of Child and Family Studies, 25, 77–85. https://doi.org/10.1007/s10826-015-0206-7.

    Article  Google Scholar 

  • *Renshaw, T. L., Eklund, K., Dowdy, R., Jimerson, S. R., Hart, S. R., Earhart, J. J., & Jones, C. N. (2009). Examining the relationship between scores on the Behavioral and Emotional Screening System and student academic, behavioral, and engagement outcomes: An investigation of concurrent validity in elementary school. The California School Psychologist, 14, 81–88. Retrieved Nov 1, 2017, from https://eric.ed.gov/.

  • Reschly, A. L., Busch, T. W., Betts, J., Deno, S. L., & Long, J. D. (2009). Curriculum-based Measurement Oral Reading as an indicator of reading achievement: A meta-analysis of the correlational evidence. Journal of School Psychology, 47, 427–469. https://doi.org/10.1016/j.jsp.2009.07.001.

    Article  PubMed  Google Scholar 

  • *Richardson, M. J., Caldarella, P., Young, B. J., Young, E. L., & Young, K. R. (2009). Further validation of the systematic screening for behavior disorders in middle and junior high school. Psychology in the Schools, 46, 605–615. https://doi.org/10.1002/pits.20401.

    Article  Google Scholar 

  • *Ruchkin, V., Koposov, R., Vermeiren, R., & Schwab-Stone, M. (2012). The Strength and Difficulties Questionnaire: Russian validation of the teacher version and comparison of teacher and student reports. Journal of Adolescence, 35, 87–96. https://doi.org/10.1016/j.adolescence.2011.06.003.

    Article  PubMed  Google Scholar 

  • Schmidt, F. L., & Hunter, J. E. (1977). Development of a general solution to the problem of validity generalization. Journal of Applied Psychology, 62, 529–540. https://doi.org/10.1037/h0078043.

    Article  Google Scholar 

  • Severson, H. H., Walker, H. M., Hope-Doolittle, J., Kratochwill, T. R., & Gresham, F. M. (2007). Proactive, early screening to detect behaviorally at-risk students: Issues, approaches, emerging innovations, and professional practices. Journal of School Psychology, 45, 193–223. https://doi.org/10.1016/j.jsp.2006.11.003.

    Article  Google Scholar 

  • Shinn, M. R. (2010). Building a scientifically based data system for progress monitoring and universal screening across three tiers, including RTI using curriculum-based measurement. In M. R. Shinn & H. M. Walker (Eds.), Interventions for achievement and behavior problems in a three-tier model, including RTI (pp. 259–292). Bethesda, MD: National Association of School Psychologists.

    Google Scholar 

  • Smith, S. R. (2007). Making sense of multiple informants in child and adolescent psychopathology: A guide for clinicians. Journal of Psychoeducational Assessment, 25, 139–149. https://doi.org/10.1177/0734282906296233.

    Article  Google Scholar 

  • Sourander, A., & Helstela, L. (2005). Childhood predictors of externalizing and internalizing problems in adolescence: A prospective follow-up study from age 8 to 16. European Child & Adolescent Psychiatry, 14, 415–423. Retrieved Nov 1, 2017, from https://medlineplus.gov/.

  • U.S. Preventive Services Task Force. (2016). Final Recommendation Statement: Depression in Children and Adolescents: Screening. Retrieved Nov 1, 2017, from http://www.uspreventiveservicestaskforce.org/Page/Document/RecommendationStatementFinal/depression-in-children-and-adolescents-screening1.

  • Vacha-Haase, T., Henson, R. K., & Caruso, J. C. (2002). Reliability generalization: Moving toward improved understanding and use of score reliability. Educational and Psychological Measurement, 62, 562–569. https://doi.org/10.1177/0013164402062004002.

    Article  Google Scholar 

  • Vacha-Haase, T., Kogan, L. R., Tani, C. R., & Woodall, R. A. (2001). Reliability generalization: Exploring variation of reliability coefficients of MMPI clinical scores. Educational and Psychological Measurement, 61, 45–59. https://doi.org/10.1177/107319110100800404.

    Article  Google Scholar 

  • Vacha-Haase, T., Kogan, L. R., & Thompson, B. (2000). Sample compositions and variabilities in published studies versus those in test manuals: Validity of score reliability inductions. Educational and Psychological Measurement, 60, 509–522. https://doi.org/10.1177/00131640021970682.

    Article  Google Scholar 

  • *Van Leeuwen, K., Meerschaert, T., Bosmans, G., De Medts, L., & Braet, C. (2006). The Strengths and Difficulties Questionnaire in a community sample of young children in Flanders. European Journal of Psychological Assessment, 22, 189–197. Retrieved Nov 1, 2017, from https://eric.ed.gov/.

  • *van Widenfelt, B. M., Goedhart, A. W., Treffers, P. D. A., & Goodman, R. (2003). Dutch version of the Strengths and Difficulties Questionnaire (SDQ). European Child and Adolescent Psychiatry, 12, 281–289. https://doi.org/10.1007/s00787-003-0341-3.

    Article  PubMed  Google Scholar 

  • Vasa, R. A., & Pine, D. S. (2006). Anxiety disorders. In C. A. Essau (Ed.), Child and adolescent psychopathology: Theoretical and clinical implications (pp. 78–112). New York, NY: Routledge.

    Google Scholar 

  • Viswesvaran, C., & Ones, D. (2000). Measurement error in “big five factors” personality assessment: Reliability generalization across studies and measures. Educational and Psychological Measurement, 60, 224–235. Retrieved Nov 1, 2017, from http://journals.sagepub.com/home/epm.

  • Walker, H. M., Horner, R. H., Sugai, G., Bullis, M., Sprague, J. R., & Bricker, D. (1996). Integrated approaches to preventing antisocial behavior patterns among school-age children and youth. Journal of Emotional and Behavioral Disorders, 4, 194–209. https://doi.org/10.1177/106342669600400401.

    Article  Google Scholar 

  • Walker, H. M., Nishioka, V. M., Zeller, R., Severson, H. H., & Feil, E. G. (2000). Causal factors and potential solutions for the persistent under-identification of students having emotional or behavioral disorders in the context of schooling. Assessment for Effective Intervention, 26, 29–40. Retrieved Nov 1, 2017, from https://eric.ed.gov/.

  • Walker, H. M., Ramsey, E., & Gresham, F. M. (2004). Antisocial behavior in school: Evidence-based practices (2nd ed.). Boston: Wadsworth Cengage Learning.

    Google Scholar 

  • Walker, H. M., & Severson, H. H. (1990). Systematic screening for behavior disorders. Longmont, CO: Sopris West.

    Google Scholar 

  • *Watkins, M. L. (2008). Promoting the early identification of internalizing problems in preliterate children: Development of the Watkins Early Self-Report of Internalizing Problems. (Doctoral dissertation). Retrieved Nov 1, 2017, from https://www.proquest.com/.

  • *Weiss, B., Harris, V., & Catron, T. (2002). Development and initial validation of the Peer-report Measure of Internalizing and Externalizing Behavior. Journal of Abnormal Child Psychology, 30, 285–294. Retrieved Nov 1, 2017, from https://medlineplus.gov/.

  • *Yao, S., Zhang, C., Zhu, X., Jing, X., McWhinnie, C. M., & Abela, J. R. Z. (2009). Measuring adolescent psychopathology: Psychometric properties of the self-report Strengths and Difficulties Questionnaire in a sample of Chinese Adolescents. Journal of Adolescent Health, 45, 55–62. https://doi.org/10.1080/00223891.2010.482015.

    Article  PubMed  Google Scholar 

  • *Yeo, S. (2010). Predicting performance on state achievement tests using curriculum-based measurement in reading: A multilevel meta-analysis. Remedial and Special Education, 31, 412–422. https://doi.org/10.1177/0741932508327463.

    Article  Google Scholar 

  • Yin, P., & Fan, X. (2000). Assessing the reliability of Beck Depression Inventory scores: Reliability generalization across studies. Educational and Psychological Measurement, 60, 201–223. Retrieved Nov 1, 2017, from https://eric.ed.gov/.

  • Zuckerbrot, R. A., Cheung, A., Jensen, P. S., Stein, R. E., Laraque, D., & GLAD-PC STEERING GROUP. (2018). Guidelines for adolescent depression in primary care (GLAD-PC): Part I. Practice preparation, identification, assessment, and initial management. Pediatrics. https://doi.org/10.1542/peds.2017-4081.

    Article  PubMed  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Educational, School, and Counseling Psychology, University of Missouri, 16 Hill Hall, Columbia, MO, 65211, USA

    Amanda N. Allen, Stephen P. Kilgus, Matthew K. Burns & Caroline Hodgson

Authors
  1. Amanda N. Allen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Stephen P. Kilgus
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Matthew K. Burns
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Caroline Hodgson
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Amanda N. Allen.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical Approval

This article does not contain any studies with human participants performed by any of the authors.

Human and Animal Rights

Authors confirm that the contents of the manuscript and the associated investigation are and were consistent with the APA Ethical Principles of Psychologists and Code of Conduct. This article does not contain any studies with human participants performed by any of the authors.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Allen, A.N., Kilgus, S.P., Burns, M.K. et al. Surveillance of Internalizing Behaviors: A Reliability and Validity Generalization Study of Universal Screening Evidence. School Mental Health 11, 194–209 (2019). https://doi.org/10.1007/s12310-018-9290-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12310-018-9290-3

Keywords

Search

Navigation