Advertisement

Journal of Neural Transmission

, Volume 123, Issue 8, pp 981–990 | Cite as

Only complementary voices tell the truth: a reevaluation of validity in multi-informant approaches of child and adolescent clinical assessments

  • Aleksandra Kaurin
  • Boris Egloff
  • Argyris Stringaris
  • Michèle WessaEmail author
Psychiatry and Preclinical Psychiatric Studies - Review Article

Abstract

Multi-informant approaches are thought to be key to clinical assessment. Classical theories of psychological measurements assume that only convergence among different informants’ reports allows for an estimate of the true nature and causes of clinical presentations. However, the integration of multiple accounts is fraught with problems because findings in child and adolescent psychiatry do not conform to the fundamental expectation of convergence. Indeed, reports provided by different sources (self, parents, teachers, peers) share little variance. Moreover, in some cases informant divergence may be meaningful and not error variance. In this review, we give an overview of conceptual and theoretical foundations of valid multi-informant assessment and discuss why our common concepts of validity need revaluation.

Keywords

Multimethod assessment Cross-informant agreement Construct validity Incremental validity Meaningful divergence 

Notes

Acknowledgments

AK wants to thank Stefan Berti, Henning Müller and Jan Matti Dollbaum for their helpful comments on a version of this paper.

References

  1. Achenbach TM, McConaughy SH, Howell CT (1987) Child/adolescent behavioural and emotional problems: implications of cross-informant correlations for situational specificity. Psychol Bull 101(2):213CrossRefPubMedGoogle Scholar
  2. Ahadi S, Diener E (1989) Multiple determinants and effect size. J Pers Soc Psychol 56:398–340CrossRefGoogle Scholar
  3. Angold A, Costello EJ (2000) The child and adolescent psychiatric assessment (CAPA). J Am Acad Child Adolesc Psychiatry 39(1):39–48CrossRefPubMedGoogle Scholar
  4. Asendorpf JB, Banse R, Mücke D (2002) Double dissociation between implicit and explicit personality self-concept: the case of shy behaviour. J Pers Soc Psychol 83(2):380CrossRefPubMedGoogle Scholar
  5. Back MD, Schmukle SC, Egloff B (2009) Predicting actual behaviour from the explicit and implicit self-concept of personality. J Pers Soc Psychol 97(3):533CrossRefPubMedGoogle Scholar
  6. Block J (1995) A contrarian view of the five-factor approach to personality description. Psychol Bull 117(2):187CrossRefPubMedGoogle Scholar
  7. Bögels SM, Alden L, Beidel DC, Clark LA, Pine DS, Stein MB, Voncken M (2010) Social anxiety disorder: questions and answers for the DSM-V. Depress Anxiety 27(2):168–189CrossRefPubMedGoogle Scholar
  8. Borsboom D (2005) Measuring the mind: conceptual issues in contemporary psychometrics. Cambridge University Press, CambridgeCrossRefGoogle Scholar
  9. Brown W (1910) Some experimental results in the correlation of mental abilities 1. Br J Psychol, 1904–1920 3(3):296–322Google Scholar
  10. Burns GL, Haynes SN (2006) Clinical psychology: construct validation with multiple sources of information and multiple settings. In Eid M, Diener E (eds) Handbook of multimethod measurement in psychology. American Psychological Association, Washington, DC, pp 401–418Google Scholar
  11. Burns GL, Walsh JA, Gomez R (2003) Convergent and discriminant validity of trait and source effects in ADHD-inattention and hyperactivity/impulsivity measures across a 3-month interval. J Abnorm Child Psychol 31(5):529–541CrossRefPubMedGoogle Scholar
  12. Cairns R, Green J (1979) How to assess personality and social patterns: observations or ratings. Methods, issues, and illustrations, The analysis of social interactions, pp 209–226Google Scholar
  13. Campbell DT, Fiske DW (1959) Convergent and discriminant validation by the multitrait-multimethod matrix. Psychol Bull 56(2):81CrossRefPubMedGoogle Scholar
  14. Cardinet J, Tourneur Y, Allal L (1976) The symmetry of generalizability theory: applications to educational measurement. J Educ Meas 13(2):119–135CrossRefGoogle Scholar
  15. Corvin A, Buchanan RW, Carpenter WT, Kennedy JL, Keshavan MS, MacDonald AW, Sass L, Wessa M (2013) Which aspects of heterogeneity are useful to translational success? In: Silverstein SM, Moghaddam B, Wykes T (eds) Schizophrenia – Evolution and Synthesis. The MIT Press, Cambridge, pp 77–92CrossRefGoogle Scholar
  16. Cronbach LJ, Meehl PE (1955) Construct validity in psychological tests. Psychol Bull 52(4):281CrossRefPubMedGoogle Scholar
  17. Cronbach LJ, Gleser G, Nanda H, Rajaratnam N (1972) Theory of generalizability for scores and profiles. The dependability of behavioural measurements. Wiley, New YorkGoogle Scholar
  18. De Los Reyes A (2011) Introduction to the special section: more than measurement error: discovering meaning behind informant discrepancies in clinical assessments of children and adolescents. J Clin Child Adolesc Psychol 40(1):1–9CrossRefPubMedGoogle Scholar
  19. De Los Reyes A (2013) Strategic objectives for improving understanding of informant discrepancies in developmental psychopathology research. Dev Psychopathol 25(03):669–682CrossRefPubMedGoogle Scholar
  20. De Los Reyes A, Alfano CA, Beidel DC (2010) The relations among measurements of informant discrepancies within a multisite trial of treatments for childhood social phobia. J Abnorm Child Psychol 38(3):395–404CrossRefPubMedGoogle Scholar
  21. De Los Reyes A, Thomas SA, Goodman KL, Kundey SM (2013) Principles underlying the use of multiple informants’ reports. Annu Rev Clin Psychol 9:123–149CrossRefPubMedGoogle Scholar
  22. DiBartolo PM, Albano AM, Barlow DH, Heimberg RG (1998) Cross-informant agreement in the assessment of social phobia in youth. J Abnorm Child Psychol 26(3):213–220CrossRefPubMedGoogle Scholar
  23. Dirks MA, Boyle MH, Georgiades K (2011) Psychological symptoms in youth and later socioeconomic functioning: do associations vary by informant? J Clin Child Adolesc Psychol 40(1):10–22CrossRefPubMedGoogle Scholar
  24. Dirks MA, De Los Reyes A, Briggs-Gowan M, Cella D, Wakschlag LS (2012) Annual research review: embracing not erasing contextual variability in children’s behaviour—theory and utility in the selection and use of methods and informants in developmental psychopathology. J Child Psychol Psychiatry 53(5):558–574CrossRefPubMedPubMedCentralGoogle Scholar
  25. Edgeworth FY (1888) The statistics of examinations. J Roy Stat Soc 51(3):599–635Google Scholar
  26. Egloff B, Schmukle SC (2002) Predictive validity of an implicit association test for assessing anxiety. J Pers Soc Psychol 83(6):1441CrossRefPubMedGoogle Scholar
  27. Eid M, Lischetzke T, Nussbeck FW, Trierweiler LI (2003) Separating trait effects from trait-specific method effects in multitrait-multimethod models: a multiple-indicator CT-C (M-1) model. Psychol Methods 8(1):38CrossRefPubMedGoogle Scholar
  28. Funder DC (1995) On the accuracy of personality judgment: a realistic approach. Psychol Rev 102(4):652CrossRefPubMedGoogle Scholar
  29. Funder DC (2012) Accurate personality judgment. Curr Dir Psychol Sci 21(3):177–182CrossRefGoogle Scholar
  30. Garb HN (2005) Clinical judgment and decision making. Annu Rev Clin Psychol 2005(1):67–89CrossRefGoogle Scholar
  31. Gomez R, Burns GL, Walsh JA, De Moura MA (2003) Multitrait-multisource confirmatory factor analytic approach to the construct validity of ADHD rating scales. Psychol Assess 15(1):3CrossRefPubMedGoogle Scholar
  32. Greenbaum PE, Decrick RF, Prange ME, Friedman RM (1994) Parent, teacher, and child ratings of problem behaviours of youngsters with serious emotional disturbances. Psychol Assess 6(2):141CrossRefGoogle Scholar
  33. Grills AE, Ollendick TH (2003) Multiple informant agreement and the anxiety disorders interview schedule for parents and children. J Am Acad Child Adolesc Psychiatry 42(1):30–40CrossRefPubMedGoogle Scholar
  34. Hirschmüller S, Egloff B, Nestler S, Back MD (2013) The dual lens model: a comprehensive framework for understanding self–other agreement of personality judgments at zero acquaintance. J Pers Soc Psychol 104(2):335CrossRefPubMedGoogle Scholar
  35. Jöreskog KG (1969) Efficient estimation in image factor analysis. Psychometrika 34(1):51–75CrossRefGoogle Scholar
  36. Judd CM, Smith ER, Kidder LH (1991) Research methods in social relations, 6th edn. Holt, Rinehart, and Winston, New YorkGoogle Scholar
  37. Kahneman D (2003) A perspective on judgment and choice: mapping bounded rationality. Am Psychol 58(9):697CrossRefPubMedGoogle Scholar
  38. Klonsky ED, Oltmanns TF (2002) Informant-reports of personality disorder: relation to self-reports and future research directions. Clin Psychol Sci Pract 9(3):300–311CrossRefGoogle Scholar
  39. Kolko DJ, Kazdin AE (1993) Emotional/behavioural problems in clinic and nonclinic children: correspondence among child, parent and teacher reports. J Child Psychol Psychiatry 34(6):991–1006CrossRefPubMedGoogle Scholar
  40. Kraemer HC, Measelle JR, Ablow JC, Essex MJ, Boyce WT, Kupfer DJ (2003) A new approach to integrating data from multiple informants in psychiatric assessment and research: mixing and matching contexts and perspectives. Am J Psychiatry 160(9):1566–1577CrossRefPubMedGoogle Scholar
  41. Kwon K, Kim EM, Sheridan SM (2012) A contextual approach to social skills assessment in the peer group: who is the best judge? School Psychol Q 27(3):121CrossRefGoogle Scholar
  42. Lord FM, Novick MR (1968) Statistical theories of mental test scores. Addison-Wesley, Reading, MAGoogle Scholar
  43. Luby JL (2012) Dispelling the “they’ll grow out of it” myth: implications for intervention. Am J Psychiatry 169(11):1127–1129CrossRefPubMedGoogle Scholar
  44. Luby JL, Belden A, Sullivan J, Spitznagel E (2007) Preschoolers’ contribution to their diagnosis of depression and anxiety: uses and limitations of young child self-report of symptoms. Child Psychiatry Hum Dev 38(4):321–338CrossRefPubMedGoogle Scholar
  45. Mason J, Scior K (2004) ‘Diagnostic overshadowing’ amongst clinicians working with people with intellectual disabilities in the UK. J Appl Res Intellect Disabil 17(2):85–90CrossRefGoogle Scholar
  46. McConaughy SH, Stanger C, Achenbach TM (1992) Three-year course of behavioural/emotional problems in a national sample of 4-to 16-year-olds: I. Agreement among informants. J Am Acad Child Adolesc Psychiatry 31(5):932–940CrossRefPubMedGoogle Scholar
  47. Müller JM, Achtergarde S, Furniss T (2011) The influence of maternal psychopathology on ratings of child psychiatric symptoms: an SEM analysis on cross-informant agreement. Eur Child Adolesc Psychiatry 20(5):241–252CrossRefPubMedGoogle Scholar
  48. Offord DR, Boyle MH, Racine Y, Szatmari P, Fleming JE, Sanford M, Lipman EL (1996) Integrating assessment data from multiple informants. J Am Acad Child Adolesc Psychiatry 35(8):1078–1085CrossRefPubMedGoogle Scholar
  49. Owens JS, Hoza B (2003) The role of inattention and hyperactivity/impulsivity in the positive illusory bias. J Consult Clin Psychol 71(4):680CrossRefPubMedGoogle Scholar
  50. Pickles A, Angold A (2003) Natural categories or fundamental dimensions: on carving nature at the joints and the rearticulation of psychopathology. Dev Psychopathol 15(03):529–551CrossRefPubMedGoogle Scholar
  51. Pine DS, Helfinstein SM, Bar-Haim Y, Nelson E, Fox NA (2009) Challenges in developing novel treatments for childhood disorders: lessons from research on anxiety. Neuropsychopharmacology 34(1):213–228CrossRefPubMedGoogle Scholar
  52. Polanczyk GV, Willcutt EG, Salum GA, Kieling C, Rohde LA (2014) ADHD prevalence estimates across three decades: an updated systematic review and meta-regression analysis. Int J Epidemiol 43(2):434–442CrossRefPubMedPubMedCentralGoogle Scholar
  53. Roberts BW, Caspi A (2001) Personality Development and the person-situation debate: it’s Déjà Vu all over again. Psychol Inq 12(2):104–109CrossRefGoogle Scholar
  54. Schmidt FL, Hunter JE (1999) Theory testing and measurement error. Intelligence 27(3):183–198CrossRefGoogle Scholar
  55. Schmitt M (2006) Conceptual, theoretical, and historical foundations of multimethod assessment. In: Eid M, Diener E (eds) Handbook of multimethod measurement in psychology. American Psychological Association, Washington, DC, pp 9–25CrossRefGoogle Scholar
  56. Shadish WR, Cook TD, Campbell DT (2002) Experimental and quasi-experimental designs for generalized causal inference. Wadsworth Cengage learning, BostonGoogle Scholar
  57. Silverman WK, Ollendick TH (2005) Evidence-based assessment of anxiety and its disorders in children and adolescents. J Clin Child Adolesc Psychol 34(3):380–411CrossRefPubMedGoogle Scholar
  58. Smith BH, Pelham WE Jr, Gnagy E, Molina B, Evans S (2000) The reliability, validity, and unique contributions of self-report by adolescents receiving treatment for attention-deficit/hyperactivity disorder. J Consult Clin Psychol 68(3):489CrossRefPubMedGoogle Scholar
  59. Spearman C (1910) Correlation calculated from faulty data. Br J Psychol, 1904–1920 3(3):271–295Google Scholar
  60. Strack F, Deutsch R (2004) Reflective and impulsive determinants of social behaviour. Pers Soc Psychol Rev 8(3):220–247CrossRefPubMedGoogle Scholar
  61. Stringaris A (2015) Emotion regulation and emotional disorders: conceptual issues for clinicians and neuroscientists. In: Thapar A, Pine DS, Leckman JF, Scott S, Snowling MJ, Taylor EA (eds) Rutter’s child and adolescent psychiatry, 6th edn. Wiley-Blackwell, New YorkGoogle Scholar
  62. Sutcliffe JP (1965) A probability model for errors of classification I. General considerations. Psychometrika 30(1):73–96CrossRefPubMedGoogle Scholar
  63. Vazire S (2010) Who knows what about a person? The self–other knowledge asymmetry (SOKA) model. J Pers Soc Psychol 98(2):281CrossRefPubMedGoogle Scholar
  64. Weisbrot DM, Gadow KD, DeVincent CJ, Pomeroy J (2005) The presentation of anxiety in children with pervasive developmental disorders. J Child Adolesc Psychopharmacol 15(3):477–496CrossRefPubMedGoogle Scholar
  65. Youngstrom EA, Findling RL, Calabrese JR, Gracious BL, Demeter C, Bedoya DD, Price M (2004) Comparing the diagnostic accuracy of six potential screening instruments for bipolar disorder in youths aged 5 to 17 years. J Am Acad Child Adolesc Psychiatry 43(7):847–858CrossRefPubMedGoogle Scholar

Copyright information

© Springer-Verlag Wien 2016

Authors and Affiliations

  • Aleksandra Kaurin
    • 1
  • Boris Egloff
    • 2
  • Argyris Stringaris
    • 3
  • Michèle Wessa
    • 1
    Email author
  1. 1.Department of Clinical Psychology and Neuropsychology, Institute of PsychologyJohannes Gutenberg-University MainzMainzGermany
  2. 2.Department of Personality Psychology and Psychological Assessment, Institute of PsychologyJohannes Gutenberg-University MainzMainzGermany
  3. 3.Department of Child and Adolescent Psychiatry, Institute of Psychiatry, Psychology and NeuroscienceKing’s College LondonLondonUK

Personalised recommendations