Incremental Validity of Teacher and Parent Symptom and Impairment Ratings when Screening for Mental Health Difficulties

Abstract

Although universal screening for mental health difficulties is increasingly recognized as a way to identify children who are at risk and provide early intervention, little research exists to inform decisions about screening, such as the choice of informants and the type of information collected. The present study examined the incremental validity of teacher- and parent-rated (primarily mothers) symptoms and impairment in a non-referred sample of early elementary school children (n = 320, 49 % boys, ages 6 to 9) in terms of predicting impairment as rated by a different teacher 1 year later. Teacher-rated symptoms and impairment and parent-rated impairment were each unique predictors of later impairment; however, parent-rated symptoms did not contribute to the prediction of later impairment above and beyond these other indicators. The results indicate that, when screening for mental health difficulties in the school system, impairment ratings collected across settings add useful information, but it may not be necessary to use parent symptom ratings when teacher symptom ratings are available.

This is a preview of subscription content, access via your institution.

Notes

  1. 1.

    This analysis was run as a linear regression as negative binomial regression is not available for imputed data in Mplus.

References

  1. Achenbach, T. M., McConaughy, S. H., & Howell, C. T. (1987). Child/adolescent behavioral and emotional problems: implications of cross-informant correlations for situational specificity. Psychological Bulletin, 101, 213–232. doi:10.1037/0033-2909.101.2.213.

    Article  PubMed  Google Scholar 

  2. Aitken, M., Martinussen, R., Wolfe, R. G., & Tannock, R. (2015). Factor structure of the Strengths and Difficulties Questionnaire in a Canadian elementary school sample. Assessment for Effective Intervention, 40(3), 155–165. doi:10.1177/1534508414560347.

    Article  Google Scholar 

  3. Albers, C. A., Glover, T. A., & Kratochwill, T. R. (2007). Introduction to the special issue: How can universal screening enhance educational and mental health outcomes? Journal of School Psychology, 45(2), 113–116. doi:10.1016/j.jsp.2006.12.002.

    Article  Google Scholar 

  4. Allison, P. D. (2012). Regression for count data. In Logistic regression using SAS: theory and applications (pp. 265–290). Cary, NC: SAS Institute, Inc..

    Google Scholar 

  5. Andrade, B. F., & Tannock, R. (2013). The direct effects of inattention and hyperactivity/impulsivity on peer problems and mediating roles of prosocial and conduct problem behaviors in a community sample of children. Journal of Attention Disorders, 17(8), 670–680. doi:10.1177/1087054712437580.

    Article  PubMed  Google Scholar 

  6. Angold, A., Costello, E. J., Farmer, E. M. Z., Burns, B. J., & Erkanli, A. (1999). Impaired but undiagnosed. Journal of the American Academy of Child and Adolescent Psychiatry, 38(2), 129–137. doi:10.1097/00004583-199902000-00011.

    Article  PubMed  Google Scholar 

  7. Baker, J. A. (2008). Assessing school risk and protective factors. In B. Doll & J. A. Cummings (Eds.), Transforming school mental health services: population-based approaches to promoting the competency and wellness of children (pp. 43–68). Thousand Oaks, CA: Corwin Press.

    Google Scholar 

  8. Bringewatt, E. H., & Gershoff, E. T. (2010). Falling through the cracks: gaps and barriers in the mental health system for America’s disadvantaged children. Children and Youth Services Review, 32(10), 1291–1299. doi:10.1016/j.childyouth.2010.04.021.

    Article  Google Scholar 

  9. Costello, E. J., Angold, A., & Keeler, G. P. (1999). Adolescent outcomes of childhood disorders: The consequences of severity and impairment. Journal of the American Academy of Child and Adolescent Psychiatry, 38(2), 121–128. doi:10.1097/00004583-199902000-00010.

    Article  PubMed  Google Scholar 

  10. Dadds, M. R., Holland, D. E., Laurens, K. R., Mullins, M., Barrett, P. M., & Spence, S. H. (1999). Early intervention and prevention of anxiety disorders in children: results at 2-year follow-up. Journal of Consulting and Clinical Psychology, 67(1), 145–150.

    Article  PubMed  Google Scholar 

  11. De Los Reyes, A., Henry, D. B., Tolan, P. H., & Wakschlag, L. S. (2009). Linking informant discrepancies to observed variations in young children’s disruptive behavior. Journal of Abnormal Child Psychology, 37(5), 637–642. doi:10.1007/s10802-009-9307-3.

    Article  PubMed  PubMed Central  Google Scholar 

  12. De Los Reyes, A., Thomas, S. A., Goodman, K. L., & Kundey, S. M. (2013). Principles underlying the use of multiple informants’ reports. Annual Review of Clinical Psychology, 9, 123–149. doi:10.1146/annurev-clinpsy-050212-185617.

    Article  PubMed  Google Scholar 

  13. De Los Reyes, A., Augenstein, T. M., Wang, M., Thomas, S. A., Deborah, A. G., Burgers, D. E., & Rabinowitz, J. (2015). The validity of the multi-informant approach to assessing child and adolescent mental health. Psychological Bulletin . doi:10.1037/a0038498.Advance online publication.

    PubMed  PubMed Central  Google Scholar 

  14. Dever, B. V., Dowdy, E., Raines, T. C., & Carnazzo, K. (2015). Stability and change of behavioral and emotional screening scores. Psychology in the Schools, 52(6), 618–629. doi:10.1002/pits.21825.

    Article  Google Scholar 

  15. Dirks, M. A., Boyle, M. H., & Georgiades, K. (2011). Psychological symptoms in youth and later socioeconomic functioning: do associations vary by informant? Journal of Clinical Child and Adolescent Psychology, 40(1), 10–22. doi:10.1080/15374416.2011.533403.

    Article  PubMed  Google Scholar 

  16. Dirks, M. A., De Los Reyes, A., Briggs-Gowan, M., Cella, D., & Wakschlag, L. S. (2012). Annual research review: embracing not erasing contextual variability in children’s behavior - Theory and utility in the selection and use of methods and informants in developmental psychopathology. Journal of Child Psychology and Psychiatry, 53(5), 558–574. doi:10.1111/j.1469-7610.2012.02537.x.

    Article  PubMed  PubMed Central  Google Scholar 

  17. Dodge, K. A., & Pettit, G. S. (2003). A biopsychosocial model of the development of chronic conduct problems in adolescence. Developmental Psychology, 39(2), 349–371. doi:10.1037/0012-1649.39.2.349.

    Article  PubMed  PubMed Central  Google Scholar 

  18. Dodge, K. A., Bierman, K. L., Coie, J. D., Greenberg, M. T., Lochman, J. E., McMahon, R. J., & Pinderhughes, E. E. (2015). Impact of early intervention on psychopathology, crime, and well-being at age 25. American Journal of Psychiatry, 172(1), 59–70. doi:10.1176/appi.ajp.2014.13060786.

    Article  PubMed  Google Scholar 

  19. Dowdy, E., & Kim, E. (2012). Choosing informants when conducting a universal screening for behavioral and emotional risk. School Psychology Forum, 6(4), 1–10.

    Google Scholar 

  20. Dowdy, E., Furlong, M., Eklund, K., Saeki, E., & Ritchey, K. (2010a). Screening for mental health and wellness: Current school-based practices and emerging possibilities. In B. Doll & W. Pfohl (Eds.), Handbook of youth prevention science (pp. 70–95). New York: Routledge.

    Google Scholar 

  21. Dowdy, E., Ritchey, K., & Kamphaus, R. W. (2010b). School-based screening: a population-based approach to inform and monitor children’s mental health needs. School Mental Health, 2, 166–176. doi:10.1007/s12310-010-9036-3.

    Article  PubMed  PubMed Central  Google Scholar 

  22. Drabick, D. A., Gadow, K. D., & Sprafkin, J. (2006). Co-occurrence of conduct disorder and depression in a clinic-based sample of boys with ADHD. Journal of Child Psychology and Psychiatry, 47(8), 766–774. doi:10.1111/j.1469-7610.2006.01625.x.

    Article  PubMed  Google Scholar 

  23. Durlak, J. A., & Wells, A. M. (1998). Evaluation of indicated preventive intervention (secondary prevention) mental health programs for children and adolescents. American Journal of Community Psychology, 26(5), 775–802. doi:10.1023/A:1022162015815.

    Article  PubMed  Google Scholar 

  24. Edelbrock, C., Costello, A. J., Dulcan, M. K., Kalas, R., & Conover, N. C. (1985). Age differences in the reliability of the psychiatric interview of the child. Child Development, 56(1), 265–275.

    Article  PubMed  Google Scholar 

  25. Epkins, C. C. (1993). A preliminary comparison of teacher ratings and child self-report of depression, anxiety, and aggression in inpatient and elementary school samples. Journal of Abnormal Child Psychology, 21(6), 649–661. doi:10.1007/BF00916448.

    Article  PubMed  Google Scholar 

  26. Feeney-Kettler, K. A., Kratochwill, T. R., Kaiser, A. P., Hemmeter, M. L., & Kettler, R. J. (2010). Screening young children’s risk for mental health problems: a review of four measures. Assessment for Effective Intervention, 35(4), 218–230. doi:10.1177/1534508410380557.

    Article  Google Scholar 

  27. Gadow, K. D., Kaat, A. J., & Lecavalier, L. (2013). Relation of symptom-induced impairment with other illness parameters in clinic-referred youth. Journal of Child Psychology and Psychiatry, 54(11), 1198–1207. doi:10.1111/jcpp.12077.

    Article  PubMed  Google Scholar 

  28. Gardner, W., Mulvey, E. P., & Shaw, E. C. (1995). Regression analyses of counts and rates: Poisson, overdispersed Poisson, and negative binomial models. Psychological Bulletin, 118(3), 392–404. doi:10.1037/0033-2909.118.3.392.

    Article  PubMed  Google Scholar 

  29. Girio-Herrera, E., Dvorsky, M. R., & Owens, J. S. (2015). Mental health screening in kindergarten youth: a multistudy examination of the concurrent and diagnostic validity of the Impairment Rating Scale. Psychological Assessment, 27(1), 215–227. doi:10.1037/a0037787.

    Article  PubMed  Google Scholar 

  30. Glover, T. A., & Albers, C. A. (2007). Considerations for evaluating universal screening assessments. Journal of School Psychology, 45, 117–135. doi:10.1016/j.jsp.2006.05.005.

    Article  Google Scholar 

  31. Goodman, R. (1997). The Strengths and Difficulties Questionnaire: a research note. Journal of Child Psychology and Psychiatry, 38(11), 581–586. doi:10.1097/00004583-200111000-00015.

    Article  PubMed  Google Scholar 

  32. Goodman, R. (1999). The extended version of the Strengths and Difficulties Questionnaire as a guide to child psychiatric caseness and consequent burden. Journal of Child Psychology and Psychiatry, 40(5), 791–799. doi:10.1111/1469-7610.00494.

    Article  PubMed  Google Scholar 

  33. Goodman, R. (2001). Psychometric properties of the Strengths and Difficulties Questionnaire. Journal of the American Academy of Child and Adolescent Psychiatry, 40(511), 1337–1345.

    Article  PubMed  Google Scholar 

  34. Goodman, R., Ford, T., Simmons, H., Gatward, R., & Meltzer, H. (2000). Using the Strengths and Difficulties Questionnaire (SDQ) to screen for child psychiatric disorders in a community sample. British Journal of Psychiatry, 177, 534–539. doi:10.1192/bjp.177.6.534.

    Article  PubMed  Google Scholar 

  35. Gordon, M., Antshel, K., Faraone, S., Barkley, R., Lewandowski, L., Hudziak, J. J., & Cunningham, C. (2006). Symptoms versus impairment: the case for respecting DSM-IV’s Criterion D. Journal of Attention Disorders, 9(3), 465–475. doi:10.1521/adhd.2005.13.4.1.

    Article  PubMed  Google Scholar 

  36. Hay, D. F., Pawlby, S., Sharp, D., Schmücker, G., Mills, A., Allen, H., & Kumar, R. (1999). Parents’ judgements about young children's problems: why mothers and fathers might disagree yet still predict later outcomes. Journal of Child Psychology and Psychiatry, 40(8), 1249–1258. doi:10.1111/1469-7610.00541.

    Article  PubMed  Google Scholar 

  37. Herrman, H. (2014). Early intervention as a priority for world psychiatry. Early Intervention in Psychiatry, 8(4), 305–306. doi:10.1111/eip.12198.

    Article  PubMed  Google Scholar 

  38. Hill, L. G., Coie, J. D., Lochman, J. E., & Greenberg, M. T. (2004). Effectiveness of early screening for externalizing problems: issues of screening accuracy and utility. Journal of Consulting and Clinical Psychology, 72(5), 809–820. doi:10.1037/0022-006X.72.5.809.

    Article  PubMed  PubMed Central  Google Scholar 

  39. Huculak, S., & McLennan, J. D. (2014). Using teacher ratings to assess the association between mental health symptoms and impairment in children. School Mental Health, 6, 15–26. doi:10.1007/s12310-013-9107-3.

    Article  Google Scholar 

  40. Hunsley, J. (2003). Introduction to the special section on incremental validity and utility in clinical assessment. Psychological Assessment, 15(4), 443–445. doi:10.1037/1040-3590.15.4.443.

    Article  PubMed  Google Scholar 

  41. Hunsley, J., & Mash, E. J. (2007). Evidence-based assessment. Annual Review of Clinical Psychology, 3, 29–51. doi:10.1146/annurev.clinpsy.3.022806.091419.

    Article  PubMed  Google Scholar 

  42. Jenkins, L. N., Demaray, M. K., Wren, N. S., Secord, S. M., Lyell, K. M., Magers, A. M., & Tennant, J. (2014). A critical review of five commonly used social-emotional and behavioral screeners for elementary or secondary schools. Contemporary School Psychology, 18(4), 241–254. doi:10.1007/s40688-014-0026-6.

    Article  Google Scholar 

  43. Johnson, S., Hollis, C., Marlow, N., Simms, V., & Wolke, D. (2014). Screening for childhood mental health disorders using the Strengths and Difficulties Questionnaire: the validity of multi-informant reports. Developmental Medicine and Child Neurology, 56(5), 453–459. doi:10.1111/dmcn.12360.

    Article  PubMed  Google Scholar 

  44. Johnston, C., & Murray, C. (2003). Incremental validity in the psychological assessment of children and adolescents. Psychological Assessment, 15(4), 496–507. doi:10.1037/1040-3590.15.4.496.

    Article  PubMed  Google Scholar 

  45. Jones, D., Dodge, K. A., Foster, E. M., & Nix, R. (2002). Early identification of children at risk for costly mental health service use. Prevention Science, 3(4), 247–256.

    Article  PubMed  PubMed Central  Google Scholar 

  46. Kremelberg, D. (2011). Logistic, ordered, multinomial, negative binomial, and poisson regression. In Practical statistics: a quick and easy guide to IBM® SPSS® statistics, STATA, and other statistical software (pp. 235–287). Thousand Oaks, CA: SAGE Publications. doi:10.4135/9781483385655.n6.

    Google Scholar 

  47. Lane, K. L., Kalberg, J. R., Menzies, H., Bruhn, A., Eisner, S., & Crnobori, M. (2011). Using systematic screening data to assess risk and identify students for targeted supports: illustrations across the K-12 continuum. Remedial and Special Education, 32(1), 39–54. doi:10.1177/0741932510361263.

    Article  Google Scholar 

  48. Lane, K. L., Menzies, H. M., Oakes, W. P., & Kalberg, J. R. (2012). Systematic screenings of behavior to support instruction. New York: Guilford.

    Google Scholar 

  49. Levitt, J. M., Saka, N., Romanelli, L. H., & Hoagwood, K. (2007). Early identification of mental health problems in schools: the status of instrumentation. Journal of School Psychology, 45(2), 163–191. doi:10.1016/j.jsp.2006.11.005.

    Article  Google Scholar 

  50. Lochman, J. E., & The Conduct Problems Prevention Research Group (1995). Screening of child behavior problems for prevention programs at school entry. Journal of Consulting and Clinical Psychology, 63(4), 549–559. doi:10.1037//0022-006X.63.4.549.

    Article  PubMed  Google Scholar 

  51. MacCallum, R. C., Zhang, S., Preacher, K. J., & Rucker, D. D. (2002). On the practice of dichotomization of quantitative variables. Psychological Methods, 7, 19–40. doi:10.1037//1082-989X.7.1.19.

    Article  PubMed  Google Scholar 

  52. Mental Health Commission of Canada. (2012). Changing directions changing lives: the mental health strategy for Canada. Retrieved from http://www.strategy.mentalhealthcommission.ca.

  53. Normand, S., Flora, D. B., Toplak, M. E., & Tannock, R. (2012). Evidence for a general ADHD factor from a longitudinal general school population study. Journal of Abnormal Child Psychology, 40(4), 555–567. doi:10.1007/s10802-011-9584-5.

    Article  PubMed  Google Scholar 

  54. Offord, D. R., Boyle, M. H., Racine, Y., Szatmari, P., Fleming, J. E., Sanford, M., & Lipman, E. L. (1996). Integrating assessment data from multiple informants. Journal of the American Academy of Child and Adolescent Psychiatry, 35(8), 1078–1085. doi:10.1097/00004583-199608000-00019.

    Article  PubMed  Google Scholar 

  55. Pickles, A., Rowe, R., Simonoff, E., Foley, D., Rutter, M., & Silberg, J. (2001). Child psychiatric symptoms and psychosocial impairment: relationship and prognostic significance. British Journal of Psychiatry, 179, 230–235. doi:10.1192/bjp.179.3.230.

    Article  PubMed  Google Scholar 

  56. Power, T. J., Andrews, T. J., Eiraldi, R. B., Doherty, B. J., Ikeda, M. J., DuPaul, G. J., & Landau, S. (1998). Evaluating attention deficit hyperactivity disorder using multiple informants: the incremental utility of combining teacher with parent reports. Psychological Assessment, 10(3), 250–260. doi:10.1037/1040-3590.10.3.250.

    Article  Google Scholar 

  57. Rapee, R. M., Bőgels, S. M., van der Sluis, C. M., Craske, M. G., & Ollendick, T. (2012). Annual research review: conceptualising functional impairment in children and adolescents. Journal of Child Psychology and Psychiatry, 53(5), 454–468. doi:10.1111/j.1469-7610.2011.02479.x.

    Article  PubMed  Google Scholar 

  58. Rijlaarsdam, J., Stevens, G. W., van der Ende, J., Hofman, A., Jaddoe, V. W., Verhulst, F. C., & Tiemeier, H. (2015). Prevalence of DSM-IV disorders in a population-based sample of 5- to 8-year-old children: the impact of impairment criteria. European Child & Adolescent Psychiatry. doi:10.1007/s00787-015-0684-6.

    Google Scholar 

  59. Sellers, R., Maughan, B., Pickles, A., Thapar, A., & Collishaw, S. (2015). Trends in parent- and teacher-rated emotional, conduct and ADHD problems and their impact in prepubertal children in Great Britain: 1999–2008. Journal of Child Psychology and Psychiatry, 56(1), 49–57. doi:10.1111/jcpp.12273.

    Article  PubMed  Google Scholar 

  60. Sharp, C., Croudace, T. J., & Goodyer, I. M. (2005). The Strengths and Difficulties Questionnaire: predictive validity of parent and teacher ratings for help-seeking behaviour over one year. Educational and Child Psychology, 22(3), 28–44.

    Google Scholar 

  61. Shemmassian, S. K., & Lee, S. S. (2012). Comparing four methods of integrating parent and teacher symptom ratings of attention-deficit/hyperactivity disorder (ADHD). Journal of Psychopathology and Behavioral Assessment, 34(1), 1–10. doi:10.1007/s10862-011-9262-5.

    Article  Google Scholar 

  62. Sowerby, P., & Tripp, G. (2009). Evidence-based assessment of attention-deficit/hyperactivity disorder (ADHD). In J. L. Matson, F. Andrasik, & M. L. Matson (Eds.), Assessing Childhood Psychopathology and Developmental Disabilities (pp. 209–239). New York: Springer. doi:10.1007/978-0-387-09528-8.

    Google Scholar 

  63. Stone, L. L., Otten, R., Engels, R. C., Vermulst, A., & Janssens, J. M. (2010). Psychometric properties of the parent and teacher versions of the strengths and difficulties questionnaire for 4- to 12-Year-olds: a review. Clinical Child and Family Psychology Review, 13, 254–274. doi:10.1007/s10567-010-0071-2.

    Article  PubMed  PubMed Central  Google Scholar 

  64. Stringaris, A., & Goodman, R. (2013). The value of measuring impact alongside symptoms in children and adolescents: a longitudinal assessment in a community sample. Journal of Abnormal Child Psychology, 41(7), 1109–1120. doi:10.1007/s10802-013-9744-x.

    Article  PubMed  PubMed Central  Google Scholar 

  65. van Dulmen, M. H., & Egeland, B. (2011). Analyzing multiple informant data on child and adolescent behavior problems: predictive validity and comparison of aggregation procedures. International Journal of Behavioral Development, 35, 84–92. doi:10.1177/0165025410392112.

    Article  Google Scholar 

  66. Vandeventer, M. C. (2008). Universal emotional and behavioral screening for children and adolescents: are multiple gates and informants worth it? University of Georgia.

  67. Verhulst, F. C., Koot, H. M., & van der Ende, J. (1994). Differential predictive value of parents’ and teachers' reports of children's problem behaviors: a longitudinal study. Journal of Abnormal Child Psychology, 22(5), 531–546.

    Article  PubMed  Google Scholar 

  68. White, J., Connelly, G., Thompson, L., & Wilson, P. (2013). Assessing wellbeing at school entry using the strengths and difficulties questionnaire: professional perspectives. Educational Research, 55(1), 87–98. doi:10.1080/00131881.2013.767027.

    Article  Google Scholar 

  69. Wille, N., Bettge, S., Wittchen, H.-U., & Ravens-Sieberer, U. (2008). How impaired are children and adolescents by mental health problems? Results of the BELLA study. European Child & Adolescent Psychiatry, 17 Suppl 1(S1), 42–51. doi:10.1007/s00787-008-1005-0

  70. Woerner, W., Fleitlich-Bilyk, B., Martinussen, R., Fletcher, J., Cucchiaro, G., Dalgalarrondo, P., & Tannock, R. (2004). The strengths and difficulties questionnaire overseas: evaluations and applications of the SDQ beyond Europe. European Child & Adolescent Psychiatry . doi:10.1007/s00787-004-2008-0.Supplement, 13(2004)

    Google Scholar 

  71. World Health Organization. (2013). Mental health action plan 2013–2020. Retrieved from http://apps.who.int/iris/bitstream/10665/89966/1/9789241506021_eng.pdf.

Download references

Acknowledgments

This research was financially supported by a grant from the Social Sciences and Humanities Research Council of Canada (SSHRC #410-2008-1052) and the Canada Research Chair Program (RT), and by a Social Sciences and Humanities Research Council of Canada Joseph-Armand Bombardier Canada Graduate Doctoral Scholarship and an Ontario Graduate Scholarship (MA).

Author information

Affiliations

Authors

Corresponding author

Correspondence to Madison Aitken.

Ethics declarations

Conflict of Interest

Dr. Tannock was a member of the DSM-5 Work Group for Attention-Deficit/Hyperactivity Disorder (ADHD), developed the Teacher Telephone Interview for ADHD, and has received honoraria and travel costs from Shire, Janssen, Opopharma Vertiebs VG for unrestricted scientific talks. Ms. Aitken and Dr. Martinussen have no potential conflicts of interest.

Informed Consent

Informed consent was obtained from all individual participants included in the study.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Aitken, M., Martinussen, R. & Tannock, R. Incremental Validity of Teacher and Parent Symptom and Impairment Ratings when Screening for Mental Health Difficulties. J Abnorm Child Psychol 45, 827–837 (2017). https://doi.org/10.1007/s10802-016-0188-y

Download citation

Keywords

  • Incremental validity
  • Multiple informants
  • Symptoms
  • Impairment
  • Screening