Journal of Abnormal Child Psychology

, Volume 19, Issue 2, pp 149–178 | Cite as

The ecological validity of laboratory and analogue assessment methods of ADHD symptoms

  • Russell A. Barkley
Article

Abstract

An important question in the attempt to generalize laboratory findings on attention deficit-hyperactivity disorder (ADHD) children to clinical issues concerning their assessment, diagnosis, outcome, and treatment response is the ecological validity of the commonly used tasks and analogue behavioral observation procedures. This paper examines the concept of ecological validity and issues inherent in its evaluation. The evidence from a variety of sources is then reviewed on the relationship between laboratory methods of assessing inattention, impulsivity, and overactivity and measures of these same constructs in natural settings. Additional findings pertaining to this issue from a recent study of 140 ADHD and normal children and 159 ADHD and normal adolescents using a multimethod battery of tests are also reported. In general, the ecological validity of most methods is of a low to moderate degree, with some traditional laboratory tasks proving unsatisfactory. A few tasks demonstrated acceptable degrees of ecological validity but even these require improvement. It is concluded that future advances in ecological validity are likely to come from: (a) a greater reliance on assessments of the target behaviors in natural settings and (b) combining several of the more promising tasks and analogue methods into a battery that is taken over longer time intervals than has been customary and averaged across repeated administrations.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Achenbach, T. M., & Edelbrock, C. (1983).Manual for the Child Behavior Checklist and Revised Child Behavior Profile. Burlington, VT: Thomas Achenbach.Google Scholar
  2. Achenbach, T. M., McConaughy, S. H., & Howell, C. T. (1987). Child/adolescent behavioral and emotional problems: Implications of cross-informant correlations for situational specificity.Psychological Bulletin, 101, 213–232.Google Scholar
  3. Aman, M. G., & Turbott, S. H. (1986). Incidental learning, distraction, and sustained attention in hyperactive and control subjects.Journal of Abnormal Child Psychology, 14, 441–455.Google Scholar
  4. American Psychiatric Association. (1980).Diagnostic and statistical manual of mental disorders (3rd ed.). Washington, DC: Author.Google Scholar
  5. American Psychiatric Association (1987).Diagnostic and statistical manual of mental disorders (3rd ed. revised). Washington, DC: Author.Google Scholar
  6. Anastasi, A. (1976).Psychological testing (4th ed.). New York: Macmillan.Google Scholar
  7. Atkins, M. S., Pelham, W. E., & Licht, M. H. (1985). A comparison of objective classroom measures and teacher ratings of attention deficit disorder.Journal of Abnormal Child Psychology, 13, 155–167.Google Scholar
  8. Barkley, R. A. (1977a). A review of stimulant drug research with hyperactive children.Journal of Child Psychology and Psychiatry, 18, 137–165.Google Scholar
  9. Barkley, R. A. (1977b). The effects of methylphenidate on various measures of activity level and attention in hyperkinetic children.Journal of Abnormal Child Psychology, 5, 351–369.Google Scholar
  10. Barkley, R. A. (1981).Hyperactive children: A handbook for diagnosis and treatment. New York: Guilford.Google Scholar
  11. Barkley, R. A. (1988a). Attention deficit-hyperactivity disorder. In E. Mash & L. Terdal (Eds.),Behavioral assessment of childhood disorders (2nd ed., pp. 69–104). New York: Guilford Press.Google Scholar
  12. Barkley, R. A. (1988b). Attention. In M. Tramontana & S. Hooper (Eds.),Issues in child clinical neuropsychology (pp. 145–176). New York: Plenum Press.Google Scholar
  13. Barkley, R. A. (1988c). Child behavior rating scales and checklists. In M. Rutter, H. Tuma, & I. Lann (Eds.),Assessment and diagnosis in child psychopathology (pp. 113–155). New York: Guilford Press.Google Scholar
  14. Barkley, R. A. (1990a).Attention deficit hyperactivity disorder: A handbook for diagnosis and treatment. New York: Guilford Press.Google Scholar
  15. Barkley, R. A. (1990b). The problem of stimulus control and rule-governed behavior in children with attention deficit disorder with hyperactivity. In J. Swanson & L. Bloomingdale (Eds.),Attention deficit disorders (pp. 203–228). New York: Pergamon Press.Google Scholar
  16. Barkley, R. A., & Cunningham, C. E. (1979). Stimulant drugs and activity level in hyperactive children.American Journal of Orthopsychiatry, 49, 491–499.Google Scholar
  17. Barkley, R. A., DuPaul, G. J., & McMurray, M. B. (1990). A comprehensive evaluation of attention deficit disorder with and without hyperactivity as defined by research criteria.Journal of Consulting and Clinical Psychology, 58, 775–789.Google Scholar
  18. Barkley, R A, DuPaul, G. J., & McMurray, M. B. (1991). Attention deficit disorder with and without hyperactivity: Clinical response to three dose levels of methylphenidate.Pediatrics, 87.Google Scholar
  19. Barkley, R. A., Fischer, M., Newby, R., & Breen, M. (1988). Development of a multi-method clinical protocol for assessing stimulant drug responses in ADHD children.Journal of Clinical Child Psychology, 17, 14–24.Google Scholar
  20. Barkley, R. A., McMurray, M. B., Edelbrock, C. S., & Robbins, K. (1989). The response of aggressive and non-aggressive ADHD children to two doses of methylphenidate.Journal of the American Academy of Child and Adolescent Psychiatry, 28, 873–881.Google Scholar
  21. Barkley, R. A., & Ullman, D. G. (1975). A comparison of objective measures of activity and distractibility in hyperactive and nonhyperactive children.Journal of Abnormal Child Psychology, 3, 231–244.Google Scholar
  22. Breen, M. J. (1989). ADHD girls and boys: An analysis of attentional, emotional, cognitive, and family variables.Journal of Child Psychology and Psychiatry, 30, 711–716.Google Scholar
  23. Brown, R. T. (1982). A developmental analysis of visual and auditory sustained attention and reflectionimpulsivity in hyperactive and normal children.Journal of Learning Disabilities, 15, 353–357.Google Scholar
  24. Brown, R. T., & Quay, L. C. (1977). Reflection-impulsivity of normal and behavior-disordered children.Journal of Abnormal Child Psychology, 5, 457–462.Google Scholar
  25. Brown, R. T., & Wynne, M. E. (1982a). Correlates of teacher ratings, sustained attention, and impulsivity in hyperactive and normal boys.Journal of Clinical Child Psychology, 11, 262–267.Google Scholar
  26. Brown, R. T., & Wynne, M. E. (1982b). An analysis of attentional components in hyperactive and normal boys.Journal of Learning Disabilities, 15, 358–363.Google Scholar
  27. Buss, D. M., Block, J. H., & Block, J. (1980). Preschool activity level: Personality correlates and developmental implications.Child Development, 51, 401–408.Google Scholar
  28. Cairns, E., & Cammock, T. (1978). Development of a more reliable version of the Matching Familiar Figures Test.Developmental Psychology, 11, 244–248.Google Scholar
  29. Campbell, D. T., & Fiske, D. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix.Psychological Bulletin, 56, 81–105.Google Scholar
  30. Campbell, S. B. (1974). Cognitive styles and behavior problems of clinic boys: A comparison of epileptic, hyperactive, learning-disabled, and normal groups.Journal of Abnormal Child Psychology, 2, 307–312.Google Scholar
  31. Campbell, S. B., Douglas, V. I., & Morgenstern, G. (1971). Cognitive styles in hyperactive children and the effect of methylphenidate.Journal of Child Psychology and Psychiatry, 12, 55–67.Google Scholar
  32. Campbell, S. B., Szumowski, E. K., Ewing, L. J., Gluck, D. S., & Breaux, A. M. (1982). A multidimensional assessment of parent-identified behavior problem toddlers.Journal of Abnormal Child Psychology, 10, 569–592.Google Scholar
  33. Charles, L., Schain, R. J., Zelniker, T., & Guthrie, D. (1979). Effects of methylphenidate on hyperactive children's ability to sustain attention.Pediatrics, 64, 412–418.Google Scholar
  34. Copeland, A. P., & Weissbrod, C. S. (1978). Behavioral correlates of the hyperactivity factor of the Conners Teacher Questionnaire.Journal of Abnormal Child Psychology, 6, 339–343.Google Scholar
  35. de Haas, P. A. (1986). Attention styles and peer relationships of hyperactive and normal boys and girls.Journal of Abnormal Child Psychology, 14, 457–467.Google Scholar
  36. de Haas, P. A., & Young, R. D. (1984). Attention styles of hyperactive and normal girls.Journal of Abnormal Child Psychology, 12, 531–546.Google Scholar
  37. Douglas, V. I. (1972). Stop, look, and listen: The problem of sustained attention and impulse control in hyperactive and normal children.Canadian Journal of Behaviouml Science, 4, 259–282.Google Scholar
  38. Douglas, V. I. (1983). Attention and cognitive problems. In M. Rutter (Ed.),Developmental neuropsychiatry (pp. 280–329). New York: Guilford Press.Google Scholar
  39. Douglas, V. I., Barr, R. G., Amin, K, O'Neill, M. E., & Britton, B. G. (1988). Dosage effects and individual responsivity to methylphenidate in attention deficit disorder.Journal of Child Psychology and Psychiatry, 29, 453–476.Google Scholar
  40. Douglas, V. I., & Peters, K. G. (1979). Toward a clearer definition of the attentional deficit of hyperactive children. In G. A. Hale & M. Lewis (Eds.).Attention cognitive development (pp. 173–248). New York: Plenum Press.Google Scholar
  41. Draeger, S., Prior, M., & Sanson, A. (1986). Visual and auditory attention performance in hyperactive children: Competence or compliance?Journal of Abnormal Child Psychology, 14, 411–424.Google Scholar
  42. DuPaul, G. J. (1990).Parent and teacher ratings of ADHD symptoms: Psychometric properties in a community-based sample. Manuscript submitted for publication.Google Scholar
  43. Firestone, P., & Martin, J. E. (1979). An analysis of the hyperactive syndrome: A comparison of hyperactive, behavior problem, asthmatic, and normal children.Journal of Abnormal Child Psychology, 7, 261–273.Google Scholar
  44. Fischer, M., Barkley, R. A., Edelbrock, C. S., & Smallish, L. (1990). The adolescent outcome of hyperactive children diagnosed by research criteria, II: Academic, attentional, and neuropsychological status.Journal of Consulting and Clinical Psychology, 58, 580–588.Google Scholar
  45. Fuhrman, M. J., & Kendall, P. C. (1986). Cognitive tempo and behavioral adjustment in children.Cognitive Therapy and Research, 10, 45–50.Google Scholar
  46. Golden, M., Montare, A., & Bridger, W. (1977). Verbal control of delay behavior in two year old boys as a function of social class.Child Development, 48, 1107–1111.Google Scholar
  47. Goldstein, S., & Goldstein, M. (1990).Managing attention disorders in children: A guide for practitioners. New York: Wiley.Google Scholar
  48. Gordon, M. (1979). The assessment of impulsivity and mediating behaviors in hyperactive and non-hyperactive children.Journal of Abnormal Child Psychology, 7, 317–326.Google Scholar
  49. Gordon, M. (1983).The Gordon Diagnostic System. DeWitt, NY: Gordon Systems.Google Scholar
  50. Gordon, M. (1985, August).Current GDS research: The vicissitudes of validation. Paper presented at the 93rd Annual Meeting of the American Psychological Association, Los Angeles.Google Scholar
  51. Gordon, M., & McClure, F. D. (1983, August).The objective assessment of attention deficit disorders. Paper presented at the 91st Annual Convention of the American Psychological Association, Anaheim, CA.Google Scholar
  52. Gordon, M., & Mettelman, B. B. (1988). The assessment of attention: I. Standardization and reliability of a behavior based measure.Journal of Clinical Psychology, 44, 682–690.Google Scholar
  53. Goyette, C. H., Conners, C. K., & Ulrich, R. F. (1978). Normative data for Revised Conners Parent and Teacher Rating Scales.Journal of Abnormal Child Psychology, 6, 221–236.Google Scholar
  54. Haenlein, M., & Caul, W. F. (1987). Attention deficit disorder with hyperactivity: A specific hypothesis of reward dysfunction.Journal of the American Academy of Child and Adolescent Psychiatry, 26, 356–362.Google Scholar
  55. Halperin, J. M., Wolf, L. E., Pascaulvaca, D. M., Newcorn, J. H., Healey, J. M., O'Brien, J. D., Morganstein, A., & Young, G. (1988). Differential assessment of attention and impulsivity in children.Journal of the American Academy of Child and Adolescent Psychiatry, 27, 326–329.Google Scholar
  56. Hochberg, Y., & Tamhane, A. C. (1987).Multiple comparisons procedures. New York: Wiley.Google Scholar
  57. Homatidis, S., & Konstantareas, M. M. (1981). Assessment of hyperactivity: Isolating measures of high discriminant ability.Journal of Consulting and Clinical Psychology, 49, 533–541.Google Scholar
  58. Horn, W. F., Wagner, A. E., & Ialongo, N. (1989). Sex differences in school-aged children with pervasive attention deficit hyperactivity disorder.Journal of Abnormal Child Psychology, 17, 109–125.Google Scholar
  59. Hutt, C, Hutt, S. J., & Ounsted, C. (1963). A method for the study of children's behaviour.Developmental Medicine and Child Neurology, 5, 233.Google Scholar
  60. Kagan, J. (1966). Reflection-impulsivity: The generality and dynamics of conceptual tempo.Journal of Abnormal Psychology, 71, 17–24.Google Scholar
  61. Kazdin, A. E., Esveldt-Dawson, K., & Loar, L. L. (1983). Correspondence of teacher ratings and direct observations of classroom behavior of psychiatric inpatient children.Journal of Abnormal Child Psychology, 11, 549–564.Google Scholar
  62. Kendall, P. C., & Brophy, C. (1981). Activity and attentional correlates of teacher ratings of hyperactivity.Journal of Pediatric Psychology, 6, 451–458.Google Scholar
  63. Keogh, B. K., & Margolis, J. S. (1976). A component analysis of attentional problems of educationally handicapped boys.Journal of Abnormal Child Psychology, 4, 349–359.Google Scholar
  64. Klee, S. H., & Garfinkel, B. D. (1983). The computerized continuous performance task: A new measure of inattention.Journal of Abnormal Child Psychology, 11, 487–496.Google Scholar
  65. Koriath, U., Gualtieri, T., Van Bourgondien, M. E., Quade, D., & Werry, J. S. (1985). Construct validity of clinical diagnosis in pediatric psychiatry: Relationship among measures.Journal of the American Academy of Child Psychiatry, 242, 429–436.Google Scholar
  66. Kuehne, C., Kehle, T. J., & McMahon, W. (1987). Differences between children with attention deficit disorder, children with specific learning disabilities, and normal children.Journal of School Psychology, 25, 161–166.Google Scholar
  67. Luk, S. (1985). Direct observations studies of hyperactive behaviors.Journal of the American Academy of Child Psychiatry, 24, 338–344.Google Scholar
  68. Margolis, J. S. (1972).Academic correlates of sustained attention. Unpublished doctoral dissertation, University of California, Los Angeles.Google Scholar
  69. McClure, F. D., & Gordon, M. (1984). Performance of disturbed hyperactive and nonhyperactive children on an objective measure of hyperactivity.Journal of Abnormal Child Psychology, 12, 561–572.Google Scholar
  70. Milich, R. (1984). Cross-sectional and longitudinal observations of activity level and sustained attention in a normative sample.Journal of Abnormal Child Psychology, 12, 261–276.Google Scholar
  71. Milich, R., & Kramer, J. (1984). Reflections on impulsivity: An empirical investigation of impulsivity as a construct. In K. Gadow & I. Bialer (Eds.),Advances in learning and behavioral disabilities (Vol. 3, pp. 117–150). Greenwich, CT: JAI Press.Google Scholar
  72. Milich, R., Loney, J., & Landau, S. (1982). The independent dimensions of hyperactivity and aggression: A validation with playroom observation data.Journal of Abnormal Psychology, 91, 183–198.Google Scholar
  73. Milich, R., Loney, J., & Roberts, M. A. (1986). Playroom observations of activity level and sustained attention: Two-year stability.Journal of Consulting and Clinical Psychology, 54, 272–274.Google Scholar
  74. O'Dougherty, M., Nuechterlein, K H., & Drew, B. (1984). Hyperactive and hypoxic children: signal detection, sustained attention, and behavior.Journal of Abnormal Psychology, 93, 178–191.Google Scholar
  75. Pascaulvaca, D. M., Wolf, L. E, Healey, J. M., Tweedy, J. R., & Halperin, J. M. (1988, January).Sex differences in attention and behavior in school-age children. Paper presented at the 16th Annual Meeting of the International Neuropsychological Society, New Orleans.Google Scholar
  76. Pelham, W. E., Bender, M. E., Caddell, J., Booth, S., & Moorer, S. H. (1985). Methylphenidate and children with attention deficit disorder.Archives of General Psychiatry, 42, 948–952.Google Scholar
  77. Pope, L. (1970). Motor activity in brain-injured children.American Jojrnal of Orthopsychiatry, 40, 783–794.Google Scholar
  78. Prinz, R. J., Tarnowski, K. J., & Nay, S. M. (1984). Assessment of sustained attention and distraction in children using a classroom analogue task.Journal of Clinical Child Psychology, 13, 250–256.Google Scholar
  79. Quay, H. C. (1988). The behavioral reward and inhibition system in childhood behavior disorder. In L. Bloomingdale (Ed.),Attention deficit disorder, Vol. 3: New research in attention, treatment, and psychopharmacology (pp. 176–186). New York: Pergamon Press.Google Scholar
  80. Rapport, M. D., DuPaul, G. J., Stoner, G., Birmingham, B. K., & Masse, G. (1985). Attention deficit disorder with hyperactivity: Differential effects of methylphenidate on impulsivity.Pediatrics, 76, 938–943.Google Scholar
  81. Rapport, M. D., DuPaul, G. J., Stoner, G., & Jones, J. T. (1986). Comparing classroom and clinic measures of attention deficit disorder: Differential, idiosyncratic, and dose-response effects of methylphenidate.Journal of Consulting and Clinical Psychology, 54, 334–341.Google Scholar
  82. Rapport, M. D., Jones, J. T., DuPaul, G. J., Kelly, K. L., Gardner, M. J., Tucker, S. B., & Shea, M. S. (1987). Attention deficit disorder and methylphenidate: Group and singlesubject analyses of dose effects on attention in clinic and classroom settings.Journal of Clinical Child Psychology, 16, 329–338.Google Scholar
  83. Rapport, M. D., Tucker, S. B., DuPaul, G. J., Merlo, M., & Stoner, G. (1986). Hyperactivity and frustration: The influence of control over and size of rewards in delaying gratification.Journal of Abnormal Child Psychology, 14, 191–204.Google Scholar
  84. Ribbler, A. (1988, January).The relationship between attention deficits and childhood behavior problems. Paper presented at the 16th Annual Meeting of the International Neuropsychological Society, New Orleans.Google Scholar
  85. Roberts, M. A. (1987). How is playroom behavior observation used in the diagnosis of attention deficit disorder? In J. Loney (Ed.),The young hyperactive child: Answers to questions about diagnosis, prognosis, and treatment (pp. 65–74). Binghamton, NY: Haworth Press.Google Scholar
  86. Roberts, M. A. (1990). A behavioral observation method for differentiating hyperactive and aggressive boys.Journal of Abnormal Child Psychology, 18, 131–142.Google Scholar
  87. Ross, D. M., & Ross, S. A. (1982).Hyperactivity: Current issues, research, and theory (2nd ed.). New York: Wiley.Google Scholar
  88. Routh, D. K., & Schroeder, C. S. (1976). Standardized playroom measures as indices of hyperactivity.Journal of Abnormal Child Psychology, 4, 199–207.Google Scholar
  89. Routh, D. K., Schroeder, C. S., & O'Tuama, L. (1974). Development of activity level in children.Developmental Psychology, 10, 163–168.Google Scholar
  90. Sandberg, S. T., Rutter, M., & Taylor, E. (1978). Hyperkinetic disorder in psychiatric clinic attenders.Developmental Medicine and Child Neurology, 20, 279–299.Google Scholar
  91. Schachar, R., Sandberg, S., & Rutter, M. (1986). Agreement between teachers' ratings and observations of hyperactivity, inattentiveness, and defiance.Journal of Abnormal Child Psychology, 14, 331–345.Google Scholar
  92. Schulman, J. L., & Reisman, J. M. (1959). An objective measure of hyperactivity.American Journal of Mental Deficiency, 64, 455–456.Google Scholar
  93. Seidel, W. T., & Joschko, M. (1990). Evidence of difficulties in sustained attention in children with ADDH.Journal of Abnormal Child Psychology, 18, 217–229.Google Scholar
  94. Shaffer, D., McNamara, N., & Pincus, J. H. (1974). Controlled observations on patterns of activity, attention, and impulsivity in brain-damaged and psychiatrically disturbed boys.Psychological Medicine, 4, 4–18.Google Scholar
  95. Sleator, E. K., & Pelham, W. E. (1986).Attention deficit disorder. Norwalk, CT: AppletonCentury-Crofts.Google Scholar
  96. Sprague, R. L., & Sleator, E. K. (1977). Methylphenidate in hyperkinetic children: Difference in dose effects on learning and social behavior.Science, 198, 1274–1276.Google Scholar
  97. Stevens, T. M., Kupst, M. J., Suran, B. G., & Schulman, J. L. (1978). Activity level: A comparison between actometer scores and observer ratings.Journal of Abnormal Child Psychology, 6, 163–173.Google Scholar
  98. Swanson, J., & Kinsbourne, M. (1979). The cognitive effects of stimulant drugs on hyperactive children. In G. Hale & M. Lewis (Eds.),Attention and cognitive development (pp. 249–274). New York: Plenum Press.Google Scholar
  99. Touwen, B. C. L., & Kalverboer, A. F. (1973). Neurologic and behavioral assessment of children with minimal brain dysfunction.Seminars in Psychiatry, 5, 79–94.Google Scholar
  100. Tryon, W. W. (1984). Principles and methods of mechanically measuring motor activity.Behavioral Assessment, 6, 129–140.Google Scholar
  101. Ullman, D. G., Barkley, R. A., & Brown, H. W. (1978). The behavioral symptoms of hyperkinetic children who successfully responded to stimulant drug treatment.American Journal of Orthopsychiatry, 48, 425–437.Google Scholar
  102. van der Meere, J., & Sergeant, J. (1988a). Focused attention in pervasively hyperactive children.Journal of Abnormal Child Psychology, 16, 627–640.Google Scholar
  103. van der Meere, J., & Sergeant, J. (1988b). Controlled processing and vigilance in hyperactivity: Time will tell.Journal of Abnormal Child Psychology, 16, 641–656.Google Scholar
  104. Weissberg, R., Ruff, H. A., & Lawson, K. R. (1990). The usefulness of reaction time tasks in studying attention and organization of behavior in young children.Journal of Developmental and Behavioral Pediatrics, 11, 59–64.Google Scholar
  105. Werry, J. S., Elkind, G. S., & Reeves, J. C. (1987). Attention deficit, conduct, oppositional, and anxiety disorders in children: III. Laboratory differences.Journal of Abnormal Child Psychology, 15, 409–428.Google Scholar

Copyright information

© Plenum Publishing Corporation 1991

Authors and Affiliations

  • Russell A. Barkley
    • 1
  1. 1.Department of PsychiatryUniversity of Massachusetts Medical CenterWorcester

Personalised recommendations