Skip to main content
SpringerLink
Log in

Only complementary voices tell the truth: a reevaluation of validity in multi-informant approaches of child and adolescent clinical assessments

  • Psychiatry and Preclinical Psychiatric Studies - Review Article
  • Published:
Journal of Neural Transmission Aims and scope Submit manuscript

Abstract

Multi-informant approaches are thought to be key to clinical assessment. Classical theories of psychological measurements assume that only convergence among different informants’ reports allows for an estimate of the true nature and causes of clinical presentations. However, the integration of multiple accounts is fraught with problems because findings in child and adolescent psychiatry do not conform to the fundamental expectation of convergence. Indeed, reports provided by different sources (self, parents, teachers, peers) share little variance. Moreover, in some cases informant divergence may be meaningful and not error variance. In this review, we give an overview of conceptual and theoretical foundations of valid multi-informant assessment and discuss why our common concepts of validity need revaluation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  • Achenbach TM, McConaughy SH, Howell CT (1987) Child/adolescent behavioural and emotional problems: implications of cross-informant correlations for situational specificity. Psychol Bull 101(2):213

    Article  CAS  PubMed  Google Scholar 

  • Ahadi S, Diener E (1989) Multiple determinants and effect size. J Pers Soc Psychol 56:398–340

    Article  Google Scholar 

  • Angold A, Costello EJ (2000) The child and adolescent psychiatric assessment (CAPA). J Am Acad Child Adolesc Psychiatry 39(1):39–48

    Article  CAS  PubMed  Google Scholar 

  • Asendorpf JB, Banse R, Mücke D (2002) Double dissociation between implicit and explicit personality self-concept: the case of shy behaviour. J Pers Soc Psychol 83(2):380

    Article  PubMed  Google Scholar 

  • Back MD, Schmukle SC, Egloff B (2009) Predicting actual behaviour from the explicit and implicit self-concept of personality. J Pers Soc Psychol 97(3):533

    Article  PubMed  Google Scholar 

  • Block J (1995) A contrarian view of the five-factor approach to personality description. Psychol Bull 117(2):187

    Article  CAS  PubMed  Google Scholar 

  • Bögels SM, Alden L, Beidel DC, Clark LA, Pine DS, Stein MB, Voncken M (2010) Social anxiety disorder: questions and answers for the DSM-V. Depress Anxiety 27(2):168–189

    Article  PubMed  Google Scholar 

  • Borsboom D (2005) Measuring the mind: conceptual issues in contemporary psychometrics. Cambridge University Press, Cambridge

    Book  Google Scholar 

  • Brown W (1910) Some experimental results in the correlation of mental abilities 1. Br J Psychol, 1904–1920 3(3):296–322

  • Burns GL, Haynes SN (2006) Clinical psychology: construct validation with multiple sources of information and multiple settings. In Eid M, Diener E (eds) Handbook of multimethod measurement in psychology. American Psychological Association, Washington, DC, pp 401–418

  • Burns GL, Walsh JA, Gomez R (2003) Convergent and discriminant validity of trait and source effects in ADHD-inattention and hyperactivity/impulsivity measures across a 3-month interval. J Abnorm Child Psychol 31(5):529–541

    Article  PubMed  Google Scholar 

  • Cairns R, Green J (1979) How to assess personality and social patterns: observations or ratings. Methods, issues, and illustrations, The analysis of social interactions, pp 209–226

    Google Scholar 

  • Campbell DT, Fiske DW (1959) Convergent and discriminant validation by the multitrait-multimethod matrix. Psychol Bull 56(2):81

    Article  CAS  PubMed  Google Scholar 

  • Cardinet J, Tourneur Y, Allal L (1976) The symmetry of generalizability theory: applications to educational measurement. J Educ Meas 13(2):119–135

    Article  Google Scholar 

  • Corvin A, Buchanan RW, Carpenter WT, Kennedy JL, Keshavan MS, MacDonald AW, Sass L, Wessa M (2013) Which aspects of heterogeneity are useful to translational success? In: Silverstein SM, Moghaddam B, Wykes T (eds) Schizophrenia – Evolution and Synthesis. The MIT Press, Cambridge, pp 77–92

    Chapter  Google Scholar 

  • Cronbach LJ, Meehl PE (1955) Construct validity in psychological tests. Psychol Bull 52(4):281

    Article  CAS  PubMed  Google Scholar 

  • Cronbach LJ, Gleser G, Nanda H, Rajaratnam N (1972) Theory of generalizability for scores and profiles. The dependability of behavioural measurements. Wiley, New York

  • De Los Reyes A (2011) Introduction to the special section: more than measurement error: discovering meaning behind informant discrepancies in clinical assessments of children and adolescents. J Clin Child Adolesc Psychol 40(1):1–9

    Article  PubMed  Google Scholar 

  • De Los Reyes A (2013) Strategic objectives for improving understanding of informant discrepancies in developmental psychopathology research. Dev Psychopathol 25(03):669–682

    Article  PubMed  Google Scholar 

  • De Los Reyes A, Alfano CA, Beidel DC (2010) The relations among measurements of informant discrepancies within a multisite trial of treatments for childhood social phobia. J Abnorm Child Psychol 38(3):395–404

    Article  PubMed  Google Scholar 

  • De Los Reyes A, Thomas SA, Goodman KL, Kundey SM (2013) Principles underlying the use of multiple informants’ reports. Annu Rev Clin Psychol 9:123–149

    Article  PubMed  Google Scholar 

  • DiBartolo PM, Albano AM, Barlow DH, Heimberg RG (1998) Cross-informant agreement in the assessment of social phobia in youth. J Abnorm Child Psychol 26(3):213–220

    Article  CAS  PubMed  Google Scholar 

  • Dirks MA, Boyle MH, Georgiades K (2011) Psychological symptoms in youth and later socioeconomic functioning: do associations vary by informant? J Clin Child Adolesc Psychol 40(1):10–22

    Article  PubMed  Google Scholar 

  • Dirks MA, De Los Reyes A, Briggs-Gowan M, Cella D, Wakschlag LS (2012) Annual research review: embracing not erasing contextual variability in children’s behaviour—theory and utility in the selection and use of methods and informants in developmental psychopathology. J Child Psychol Psychiatry 53(5):558–574

    Article  PubMed  PubMed Central  Google Scholar 

  • Edgeworth FY (1888) The statistics of examinations. J Roy Stat Soc 51(3):599–635

    Google Scholar 

  • Egloff B, Schmukle SC (2002) Predictive validity of an implicit association test for assessing anxiety. J Pers Soc Psychol 83(6):1441

    Article  PubMed  Google Scholar 

  • Eid M, Lischetzke T, Nussbeck FW, Trierweiler LI (2003) Separating trait effects from trait-specific method effects in multitrait-multimethod models: a multiple-indicator CT-C (M-1) model. Psychol Methods 8(1):38

    Article  PubMed  Google Scholar 

  • Funder DC (1995) On the accuracy of personality judgment: a realistic approach. Psychol Rev 102(4):652

    Article  CAS  PubMed  Google Scholar 

  • Funder DC (2012) Accurate personality judgment. Curr Dir Psychol Sci 21(3):177–182

    Article  Google Scholar 

  • Garb HN (2005) Clinical judgment and decision making. Annu Rev Clin Psychol 2005(1):67–89

    Article  Google Scholar 

  • Gomez R, Burns GL, Walsh JA, De Moura MA (2003) Multitrait-multisource confirmatory factor analytic approach to the construct validity of ADHD rating scales. Psychol Assess 15(1):3

    Article  PubMed  Google Scholar 

  • Greenbaum PE, Decrick RF, Prange ME, Friedman RM (1994) Parent, teacher, and child ratings of problem behaviours of youngsters with serious emotional disturbances. Psychol Assess 6(2):141

    Article  Google Scholar 

  • Grills AE, Ollendick TH (2003) Multiple informant agreement and the anxiety disorders interview schedule for parents and children. J Am Acad Child Adolesc Psychiatry 42(1):30–40

    Article  PubMed  Google Scholar 

  • Hirschmüller S, Egloff B, Nestler S, Back MD (2013) The dual lens model: a comprehensive framework for understanding self–other agreement of personality judgments at zero acquaintance. J Pers Soc Psychol 104(2):335

    Article  PubMed  Google Scholar 

  • Jöreskog KG (1969) Efficient estimation in image factor analysis. Psychometrika 34(1):51–75

    Article  Google Scholar 

  • Judd CM, Smith ER, Kidder LH (1991) Research methods in social relations, 6th edn. Holt, Rinehart, and Winston, New York

  • Kahneman D (2003) A perspective on judgment and choice: mapping bounded rationality. Am Psychol 58(9):697

    Article  PubMed  Google Scholar 

  • Klonsky ED, Oltmanns TF (2002) Informant-reports of personality disorder: relation to self-reports and future research directions. Clin Psychol Sci Pract 9(3):300–311

    Article  Google Scholar 

  • Kolko DJ, Kazdin AE (1993) Emotional/behavioural problems in clinic and nonclinic children: correspondence among child, parent and teacher reports. J Child Psychol Psychiatry 34(6):991–1006

    Article  CAS  PubMed  Google Scholar 

  • Kraemer HC, Measelle JR, Ablow JC, Essex MJ, Boyce WT, Kupfer DJ (2003) A new approach to integrating data from multiple informants in psychiatric assessment and research: mixing and matching contexts and perspectives. Am J Psychiatry 160(9):1566–1577

    Article  PubMed  Google Scholar 

  • Kwon K, Kim EM, Sheridan SM (2012) A contextual approach to social skills assessment in the peer group: who is the best judge? School Psychol Q 27(3):121

    Article  Google Scholar 

  • Lord FM, Novick MR (1968) Statistical theories of mental test scores. Addison-Wesley, Reading, MA

    Google Scholar 

  • Luby JL (2012) Dispelling the “they’ll grow out of it” myth: implications for intervention. Am J Psychiatry 169(11):1127–1129

    Article  PubMed  Google Scholar 

  • Luby JL, Belden A, Sullivan J, Spitznagel E (2007) Preschoolers’ contribution to their diagnosis of depression and anxiety: uses and limitations of young child self-report of symptoms. Child Psychiatry Hum Dev 38(4):321–338

    Article  PubMed  Google Scholar 

  • Mason J, Scior K (2004) ‘Diagnostic overshadowing’ amongst clinicians working with people with intellectual disabilities in the UK. J Appl Res Intellect Disabil 17(2):85–90

    Article  Google Scholar 

  • McConaughy SH, Stanger C, Achenbach TM (1992) Three-year course of behavioural/emotional problems in a national sample of 4-to 16-year-olds: I. Agreement among informants. J Am Acad Child Adolesc Psychiatry 31(5):932–940

    Article  CAS  PubMed  Google Scholar 

  • Müller JM, Achtergarde S, Furniss T (2011) The influence of maternal psychopathology on ratings of child psychiatric symptoms: an SEM analysis on cross-informant agreement. Eur Child Adolesc Psychiatry 20(5):241–252

    Article  PubMed  Google Scholar 

  • Offord DR, Boyle MH, Racine Y, Szatmari P, Fleming JE, Sanford M, Lipman EL (1996) Integrating assessment data from multiple informants. J Am Acad Child Adolesc Psychiatry 35(8):1078–1085

    Article  CAS  PubMed  Google Scholar 

  • Owens JS, Hoza B (2003) The role of inattention and hyperactivity/impulsivity in the positive illusory bias. J Consult Clin Psychol 71(4):680

    Article  PubMed  Google Scholar 

  • Pickles A, Angold A (2003) Natural categories or fundamental dimensions: on carving nature at the joints and the rearticulation of psychopathology. Dev Psychopathol 15(03):529–551

    Article  PubMed  Google Scholar 

  • Pine DS, Helfinstein SM, Bar-Haim Y, Nelson E, Fox NA (2009) Challenges in developing novel treatments for childhood disorders: lessons from research on anxiety. Neuropsychopharmacology 34(1):213–228

    Article  CAS  PubMed  Google Scholar 

  • Polanczyk GV, Willcutt EG, Salum GA, Kieling C, Rohde LA (2014) ADHD prevalence estimates across three decades: an updated systematic review and meta-regression analysis. Int J Epidemiol 43(2):434–442

    Article  PubMed  PubMed Central  Google Scholar 

  • Roberts BW, Caspi A (2001) Personality Development and the person-situation debate: it’s Déjà Vu all over again. Psychol Inq 12(2):104–109

    Article  Google Scholar 

  • Schmidt FL, Hunter JE (1999) Theory testing and measurement error. Intelligence 27(3):183–198

    Article  Google Scholar 

  • Schmitt M (2006) Conceptual, theoretical, and historical foundations of multimethod assessment. In: Eid M, Diener E (eds) Handbook of multimethod measurement in psychology. American Psychological Association, Washington, DC, pp 9–25

    Chapter  Google Scholar 

  • Shadish WR, Cook TD, Campbell DT (2002) Experimental and quasi-experimental designs for generalized causal inference. Wadsworth Cengage learning, Boston

  • Silverman WK, Ollendick TH (2005) Evidence-based assessment of anxiety and its disorders in children and adolescents. J Clin Child Adolesc Psychol 34(3):380–411

    Article  PubMed  Google Scholar 

  • Smith BH, Pelham WE Jr, Gnagy E, Molina B, Evans S (2000) The reliability, validity, and unique contributions of self-report by adolescents receiving treatment for attention-deficit/hyperactivity disorder. J Consult Clin Psychol 68(3):489

    Article  CAS  PubMed  Google Scholar 

  • Spearman C (1910) Correlation calculated from faulty data. Br J Psychol, 1904–1920 3(3):271–295

  • Strack F, Deutsch R (2004) Reflective and impulsive determinants of social behaviour. Pers Soc Psychol Rev 8(3):220–247

    Article  PubMed  Google Scholar 

  • Stringaris A (2015) Emotion regulation and emotional disorders: conceptual issues for clinicians and neuroscientists. In: Thapar A, Pine DS, Leckman JF, Scott S, Snowling MJ, Taylor EA (eds) Rutter’s child and adolescent psychiatry, 6th edn. Wiley-Blackwell, New York

  • Sutcliffe JP (1965) A probability model for errors of classification I. General considerations. Psychometrika 30(1):73–96

    Article  CAS  PubMed  Google Scholar 

  • Vazire S (2010) Who knows what about a person? The self–other knowledge asymmetry (SOKA) model. J Pers Soc Psychol 98(2):281

    Article  PubMed  Google Scholar 

  • Weisbrot DM, Gadow KD, DeVincent CJ, Pomeroy J (2005) The presentation of anxiety in children with pervasive developmental disorders. J Child Adolesc Psychopharmacol 15(3):477–496

    Article  PubMed  Google Scholar 

  • Youngstrom EA, Findling RL, Calabrese JR, Gracious BL, Demeter C, Bedoya DD, Price M (2004) Comparing the diagnostic accuracy of six potential screening instruments for bipolar disorder in youths aged 5 to 17 years. J Am Acad Child Adolesc Psychiatry 43(7):847–858

    Article  PubMed  Google Scholar 

Download references

Acknowledgments

AK wants to thank Stefan Berti, Henning Müller and Jan Matti Dollbaum for their helpful comments on a version of this paper.

Author information

Authors and Affiliations

  1. Department of Clinical Psychology and Neuropsychology, Institute of Psychology, Johannes Gutenberg-University Mainz, Mainz, Germany

    Aleksandra Kaurin & Michèle Wessa

  2. Department of Personality Psychology and Psychological Assessment, Institute of Psychology, Johannes Gutenberg-University Mainz, Mainz, Germany

    Boris Egloff

  3. Department of Child and Adolescent Psychiatry, Institute of Psychiatry, Psychology and Neuroscience, King’s College London, London, UK

    Argyris Stringaris

Authors
  1. Aleksandra Kaurin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Boris Egloff
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Argyris Stringaris
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Michèle Wessa
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Michèle Wessa.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kaurin, A., Egloff, B., Stringaris, A. et al. Only complementary voices tell the truth: a reevaluation of validity in multi-informant approaches of child and adolescent clinical assessments. J Neural Transm 123, 981–990 (2016). https://doi.org/10.1007/s00702-016-1543-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00702-016-1543-4

Keywords

Search

Navigation