Skip to main content
SpringerLink
Log in

Systematic Integration of Multi-Informant Externalizing Ratings in Clinical Settings

  • Published:
Research on Child and Adolescent Psychopathology Aims and scope Submit manuscript

Abstract

Best practice clinical assessment of externalizing problems often necessitates collection of information from parents, youth themselves, and teachers. The present study tested the predictive validity of a psychometrically-driven scoring procedure to integrate multi-informant, dimensional ratings of externalizing problems. Participants were 2264 clinic-referred youth ages 6–18. Parents, teachers, and youth completed questionnaire ratings of externalizing problems (hyperactivity-inattention, conduct problems, and oppositionality-defiance) prior to an initial clinical appointment. The predictive validity of simple (highest informant rating; and all informant ratings separately) and more complex (latent S-1 bifactor model with specific informant factors; and moderated nonlinear factor analysis accounting for child age and sex) methods of informant integration was tested in predicting impairment, comorbidity, and number of clinical encounters. A simple model, in which all informant ratings were included, showed the best predictive validity across outcomes, performing as well or better than the use of the highest informant ratings or more complex latent variable models. The addition of child age and sex as moderators in the factor model did not improve predictive validity. Each informant (parent, teacher, and youth) contributes important information to the prediction of clinically-relevant outcomes. There is insufficient evidence at present to suggest that complex latent variable models should be favored over simpler models that preserve each informant’s ratings.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  • Achenbach, T. M., & Rescorla, L. A. (2001). Manual for the ASEBA School-Age Forms & Profiles. University of Vermont, Research Center for Children, Youth, & Families.

  • Bajeux, E., Klemanski, D. H., Husky, M., Leray, E., Chan Chee, C., Shojaei, T., Fermanian, C., & Kovess-Masfety, V. (2018). Factors associated with parent-child discrepancies in reports of mental health disorders in young children. Child Psychiatry & Human Development, 49(6), 1003–1010. https://doi.org/10.1007/s10578-018-0815-7.

    Article  Google Scholar 

  • Bauer, D. J., Howard, A. L., Baldasaro, R. E., Curran, P. J., Hussong, A. M., Chassin, L., & Zucker, R. A. (2013). A trifactor model for integrating ratings across multiple informants. Psychological Methods, 18(4), 475–493. https://doi.org/10.1037/a0032475.

    Article  PubMed  PubMed Central  Google Scholar 

  • Beidas, R. S., Stewart, R. E., Walsh, L., Lucas, S., Downey, M. M., Jackson, K., Fernandez, T., & Mandell, D. S. (2015). Free, brief, and validated: Standardized instruments for low-resource mental health settings. Cognitive and Behavioral Practice, 22(1), 5–19. https://doi.org/10.1016/j.cbpra.2014.02.002.

    Article  PubMed  PubMed Central  Google Scholar 

  • Bird, H. R., Shaffer, D., Fisher, P., & Gould, M. S. (1993). The Columbia Impairment Scale (CIS): Pilot findings on a measure of global impairment for children and adolescents. International Journal of Methods in Psychiatric Research, 3(3), 167–176.

    Google Scholar 

  • Bird, H. R., Andrews, H., Schwab-Stone, M., Goodman, S., Dulcan, M., Richters, J., Rubio-Stipec, M., Moore, R. E., Chiang, P., Hoven, C., Canino, G., Fisher, P. W., & Gould, M. S. (1996). Global measures of impairment for epidemiological and clinical use with children and adolescents. International Journal of Methods in Psychiatric Research, 6(4), 295–307. https://doi.org/10.1002/(SICI)1234-988X(199612)6:4<295::AID-MPR173>3.3.CO;2-5.

    Article  Google Scholar 

  • Burns, G. L., Geiser, C., Servera, M., Becker, S. P., & Beauchaine, T. P. (2020). Application of the bifactor S-1 model to multisource ratings of ADHD/ODD symptoms: An appropriate bifactor model for symptom ratings. Journal of Abnormal Child Psychology, 48(7), 881–894. https://doi.org/10.1007/s10802-019-00608-4.

    Article  PubMed  PubMed Central  Google Scholar 

  • Charamut, N. R., Racz, S. J., Wang, M., & De Los Reyes, A. (2022). Integrating multi-informant reports of youth mental health: A construct validation test of Kraemer and colleagues’ (2003) Satellite Model. Frontiers in Psychology, 13, 911629. https://doi.org/10.3389/fpsyg.2022.911629.

    Article  PubMed  PubMed Central  Google Scholar 

  • Curran, P. J., McGinley, J. S., Bauer, D. J., Hussong, A. M., Burns, A., Chassin, L., Sher, K., & Zucker, R. (2014). A moderated nonlinear factor model for the development of commensurate measures in integrative data analysis. Multivariate Behavioral Research, 49(3), 214–231. https://doi.org/10.1080/00273171.2014.889594.

    Article  PubMed  PubMed Central  Google Scholar 

  • Curran, P. J., Cole, V. T., Bauer, D. J., Rothenberg, W. A., & Hussong, A. M. (2018). Recovering predictor–criterion relations using covariate-informed factor score estimates. Structural Equation Modeling: A Multidisciplinary Journal, 25(6), 860–875. https://doi.org/10.1080/10705511.2018.1473773.

    Article  PubMed  Google Scholar 

  • De Los Reyes, A., & Epkins, C. C. (2023). Introduction to the special issue: A dozen years of demonstrating that informant discrepancies are more than measurement error—toward guidelines for integrating data from multi-informant assessments of youth mental health. Journal of Clinical Child & Adolescent Psychology, 52(1), 1–18. https://doi.org/10.1080/15374416.2022.2158843.

    Article  Google Scholar 

  • De Los Reyes, A., & Kazdin, A. E. (2005). Informant discrepancies in the assessment of childhood psychopathology: A critical review, theoretical framework, and recommendations for further study. Psychological Bulletin, 131(4), 483–509. https://doi.org/10.1037/0033-2909.131.4.483.

    Article  Google Scholar 

  • De Los Reyes, A., Henry, D. B., Tolan, P. H., & Wakschlag, L. S. (2009). Linking informant discrepancies to observed variations in young children’s disruptive behavior. Journal of Abnormal Child Psychology, 37(5), 637–642. https://doi.org/10.1007/s10802-009-9307-3.

    Article  PubMed Central  Google Scholar 

  • De Los Reyes, A., Thomas, S. A., Goodman, K. L., & Kundey, S. M. (2013). Principles underlying the use of multiple informants’ reports. Annual Review of Clinical Psychology, 9, 123–149. https://doi.org/10.1146/annurev-clinpsy-050212-185617.

    Article  Google Scholar 

  • De Los Reyes, A., Augenstein, T. M., Wang, M., Thomas, S. A., Deborah, A. G., Burgers, D. E., Rabinowitz, J., Reyes, A. D. L., Augenstein, T. M., Wang, M., Thomas, S. A., & Rabinowitz, J. (2015). The validity of the multi-informant approach to assessing child and adolescent mental health. Psychological Bulletin, 141(4), 858–900. https://doi.org/10.1037/a0038498

    Article  PubMed Central  Google Scholar 

  • De Los Reyes, A., Talbott, E., Power, T. J., Michel, J. J., Cook, C. R., Racz, S. J., & Fitzpatrick, O. (2022). The needs-to-goals gap: How informant discrepancies in youth mental health assessments impact service delivery. Clinical Psychology Review, 92, 102114. https://doi.org/10.1016/j.cpr.2021.102114.

    Article  Google Scholar 

  • De Los Reyes, A., Epkins, C. C., Asmundson, G. J. G., Augenstein, T. M., Becker, K. D., Becker, S. P., Bonadio, F. T., Borelli, J. L., Boyd, R. C., Bradshaw, C. P., Burns, G. L., Casale, G., Causadias, J. M., Cha, C. B., Chorpita, B. F., Cohen, J. R., Comer, J. S., Crowell, S. E., Dirks, M. A., & Youngstrom, E. A. (2023). Editorial statement about JCCAP’s 2023 special issue on informant discrepancies in youth mental health assessments: Observations, guidelines, and future directions grounded in 60 years of research. Journal of Clinical Child & Adolescent Psychology, 52(1), 147–158. https://doi.org/10.1080/15374416.2022.2158842.

    Article  Google Scholar 

  • Dirks, M. A., Boyle, M. H., & Georgiades, K. (2011). Psychological symptoms in youth and later socioeconomic functioning: Do associations vary by informant? Journal of Clinical Child and Adolescent Psychology, 40(1), 10–22. https://doi.org/10.1080/15374416.2011.533403.

    Article  PubMed  Google Scholar 

  • DuPaul, G. J., Fu, Q., Anastopoulos, A. D., Reid, R., & Power, T. J. (2020). ADHD parent and teacher symptom ratings: Differential Item functioning across gender, age, race, and ethnicity. Journal of Abnormal Child Psychology, 48(5), 679–691. https://doi.org/10.1007/s10802-020-00618-7.

    Article  PubMed  Google Scholar 

  • Ferdinand, R. F., Van Der Ende, J., & Verhulst, F. C. (2007). Parent-teacher disagreement regarding psychopathology in children: A risk factor for adverse outcome? Acta Psychiatrica Scandinavica, 115(1), 48–55. https://doi.org/10.1111/j.1600-0447.2006.00843.x.

    Article  PubMed  Google Scholar 

  • Gizer, I. R., Waldman, I. D., Abramowitz, A., Barr, C. L., Feng, Y., Wigg, K. G., Misener, V. L., & Rowe, D. C. (2008). Relations between multi-informant assessments of ADHD symptoms, DAT1, and DRD4. Journal of Abnormal Psychology, 117(4), 869–880. https://doi.org/10.1037/a0013297.

    Article  PubMed  Google Scholar 

  • Hallquist, M. N., & Wiley, J. F. (2018). MplusAutomation: An R Package for Facilitating large-scale latent variable analyses in Mplus. Structural Equation Modeling: A Multidisciplinary Journal, 25(4), 621–638. https://doi.org/10.1080/10705511.2017.1402334.

    Article  PubMed  Google Scholar 

  • Hawley, K. M., & Weisz, J. R. (2003). Child, parent and therapist (dis)agreement on target problems in outpatient therapy: The therapist’s dilemma and its implications. Journal of Consulting and Clinical Psychology, 71(1), 62–70. https://doi.org/10.1037/0022-006X.71.1.62.

    Article  PubMed  Google Scholar 

  • Holmbeck, G. N. (2002). Collecting and managing multisource and multimethod fata in studies of pediatric populations. Journal of Pediatric Psychology, 27(1), 5–18. https://doi.org/10.1093/jpepsy/27.1.5.

    Article  PubMed  Google Scholar 

  • Johnston, O. G., & Burke, J. D. (2020). Parental problem recognition and help-seeking for disruptive behavior disorders. The Journal of Behavioral Health Services & Research, 47(1), 146–163. https://doi.org/10.1007/s11414-018-09648-y.

    Article  Google Scholar 

  • Kotov, R., Krueger, R. F., Watson, D., Achenbach, T. M., Althoff, R. R., Bagby, R. M., Brown, T. A., Carpenter, W. T., Caspi, A., Clark, L. A., Eaton, N. R., Forbes, M. K., Forbush, K. T., Goldberg, D., Hasin, D., Hyman, S. E., Ivanova, M. Y., Lynam, D. R., Markon, K., & Zimmerman, M. (2017). The hierarchical taxonomy of psychopathology (HiTOP): A dimensional alternative to traditional nosologies. Journal of Abnormal Psychology, 126(4), 454–477. https://doi.org/10.1037/abn0000258.

    Article  PubMed  Google Scholar 

  • Kraemer, H. C., Measelle, J., Ablow, J. C., Essex, M. J., Boyce, W. T., & Kupfer, D. (2003). A new approach to integrating data from multiple sources in psychiatric assessments and research: Mixing and matching contexts and perspectives. American Journal of Psychiatry, 160(September), 1566–1577. https://doi.org/10.1176/appi.ajp.160.9.1566.

    Article  PubMed  Google Scholar 

  • Krueger, R. F., Kotov, R., Watson, D., Forbes, M. K., Eaton, N. R., Ruggero, C. J., Simms, L. J., Widiger, T. A., Achenbach, T. M., Bach, B., Bagby, R. M., Bornovalova, M. A., Carpenter, W. T., Morey, L. C., Mullins-sweatt, S. N., Ormel, J., Patalay, P., Patrick, C. J., & Pincus, A. L. (2018). Progress in achieving quantitative classification of psychopathology. World Psychiatry, 17(3).

  • Kush, J. M., Masyn, K. E., Amin-Esmaeili, M., Susukida, R., Wilcox, H. C., & Musci, R. J. (2023). Utilizing moderated non-linear factor analysis models for integrative data analysis: A tutorial. Structural Equation Modeling: A Multidisciplinary Journal, 30(1), 149–164. https://doi.org/10.1080/10705511.2022.2070753.

    Article  PubMed  Google Scholar 

  • Lahey, B. B., & Schwab-Stone, M. (2000). Age and gender differences in oppositional behavior and conduct problems: A cross-sectional household study of middle childhood and adolescence. Journal of Abnormal Psychology, 109(3), 488–503.

    Article  PubMed  Google Scholar 

  • Lahey, B. B., Moore, T. M., Kaczkurkin, A. N., & Zald, D. H. (2021). Hierarchical models of psychopathology: Empirical support, implications, and remaining issues. World Psychiatry, 20(1), 57–63. https://doi.org/10.1002/wps.20824.

    Article  PubMed  PubMed Central  Google Scholar 

  • Lau, A. S., Garland, A. F., Yeh, M., Mccabe, K. M., Wood, P. A., & Hough, R. L. (2004). Race/ethnicity and inter-informant agreement in assessing adolescent psychopathology. Journal of Emotional and Behavioral Disorders, 12(3), 145–156. https://doi.org/10.1177/10634266040120030201.

    Article  Google Scholar 

  • Loeber, R., Green, S. M. S., & Lahey, B. B. B. (1990). Mental health professionals’ perception of the utility of children, mothers, and teachers as informants on childhood psychopathology. Journal of Clinical Child Psychology, 19(2), 136–143. https://doi.org/10.1207/s15374424jccp1902.

    Article  Google Scholar 

  • Marsh, J. K., Zeveney, A. S., & De Los Reyes, A. (2020). Informant discrepancies in judgments about change during mental health treatments. Clinical Psychological Science, 8(2), 318–332. https://doi.org/10.1177/2167702619894905.

    Article  Google Scholar 

  • Martel, M. M., Markon, K., & Smith, G. T. (2017a). Research Review: Multi-informant integration in child and adolescent psychopathology diagnosis. Journal of Child Psychology and Psychiatry and Allied Disciplines, 58(2), 116–128. https://doi.org/10.1111/jcpp.12611.

    Article  PubMed  Google Scholar 

  • Martel, M. M., Pan, P. M., Hoffmann, M. S., Gadelha, A., do Rosário, M. C., Mari, J. J., Manfro, G. G., Miguel, E. C., Paus, T., Bressan, R. A., Rohde, L. A., & Salum, G. A. (2017b). A general psychopathology factor (P factor) in children: Structural model analysis and external validation through familial risk and child global executive function. Journal of Abnormal Psychology, 126(1), 137–148. https://doi.org/10.1037/abn0000205.

    Article  PubMed  Google Scholar 

  • Mash, E. J., & Barkley, R. A. (2010). Assessment of childhood disorders (4th ed.). Guilford.

  • Muthén, L. K., & Muthén, B. O. (2017). Mplus User’s Guide.

  • NICE (2017). Antisocial behaviour and conduct disorders in children and young people: Recognition and management. https://www.nice.org.uk/guidance/CG158.

  • Offord, D. R., Boyle, M. H., Racine, Y., Szatmari, P., Fleming, J. E., Sanford, M., & Lipman, E. L. (1996). Integrating assessment data from multiple informants. Journal of the American Academy of Child & Adolescent Psychiatry, 35(8), 1078–1085. https://doi.org/10.1097/00004583-199608000-00019.

    Article  Google Scholar 

  • Rettew, D. C., van Oort, F. V. A., Verhulst, F. C., Buitelaar, J. K., Ormel, J., Hartman, C. A., Veenstra, R., Althoff, R. R., & Hudziak, J. J. (2011). When parent and teacher ratings don’t agree: The Tracking Adolescents’ individual lives Survey (TRAILS). Journal of Child and Adolescent Psychopharmacology, 21(5), 389–397. https://doi.org/10.1089/cap.2010.0153.

    Article  PubMed  Google Scholar 

  • van der Meer, M., Dixon, A., & Rose, D. (2008). Parent and child agreement on reports of problem behaviour obtained from a screening questionnaire, the SDQ. European Child & Adolescent Psychiatry, 17(8), 491–497. https://doi.org/10.1007/s00787-008-0691-y.

    Article  Google Scholar 

  • van Dulmen, M. H. M., & Egeland, B. (2011). Analyzing multiple informant data on child and adolescent behavior problems: Predictive validity and comparison of aggregation procedures. International Journal of Behavioral Development, 35(1), 84–92. https://doi.org/10.1177/0165025410392112.

    Article  Google Scholar 

  • Watts, A. L., Poore, H. E., & Waldman, I. D. (2019). Riskier tests of the validity of the bifactor model of psychopathology. Clinical Psychological Science, 7(6), 1285–1303. https://doi.org/10.1177/2167702619855035.

    Article  Google Scholar 

  • Watts, A. L., Makol, B. A., Palumbo, I. M., De Los Reyes, A., Olino, T. M., Latzman, R. D., DeYoung, C. G., Wood, P. K., & Sher, K. J. (2022). How robust is the p factor? Using multitrait-multimethod modeling to inform the meaning of general factors of youth psychopathology. Clinical Psychological Science, 10(4), 640–661. https://doi.org/10.1177/21677026211055170.

    Article  PubMed  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Centre for Addiction and Mental Health, Toronto, Canada

    Madison Aitken & Brendan F. Andrade

  2. Department of Psychiatry, University of Toronto, Toronto, Canada

    Madison Aitken & Brendan F. Andrade

  3. Department of Psychology, York University, Toronto, Canada

    Madison Aitken

  4. Département des Fondements et Pratiques en Éducation, Université Laval, Québec, Canada

    André Plamondon

  5. Department of Health Sciences, Brock University, Saint Catherine’s, Canada

    John Krzeczkowski

  6. Department of Psychology, Simon Fraser University, Burnaby, Canada

    Hali Kil

Authors
  1. Madison Aitken
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. André Plamondon
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. John Krzeczkowski
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hali Kil
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Brendan F. Andrade
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Madison Aitken.

Ethics declarations

Conflict of Interest

The authors declare that they have no conflict of interest.

Ethical Approval

All procedures performed in studies involving human participants were in accordance with the ethicalstandards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its lateramendments or comparable ethical standards.

Informed Consent

The use of de-identified clinical data in the present study, without written participant consent, wasapproved by the Research Ethics Board at the Centre for Addiction and Mental Health.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic Supplementary Material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Aitken, M., Plamondon, A., Krzeczkowski, J. et al. Systematic Integration of Multi-Informant Externalizing Ratings in Clinical Settings. Res Child Adolesc Psychopathol 52, 635–644 (2024). https://doi.org/10.1007/s10802-023-01119-z

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10802-023-01119-z

Keywords

Search

Navigation