Abstract
An important question in the attempt to generalize laboratory findings on attention deficit-hyperactivity disorder (ADHD) children to clinical issues concerning their assessment, diagnosis, outcome, and treatment response is the ecological validity of the commonly used tasks and analogue behavioral observation procedures. This paper examines the concept of ecological validity and issues inherent in its evaluation. The evidence from a variety of sources is then reviewed on the relationship between laboratory methods of assessing inattention, impulsivity, and overactivity and measures of these same constructs in natural settings. Additional findings pertaining to this issue from a recent study of 140 ADHD and normal children and 159 ADHD and normal adolescents using a multimethod battery of tests are also reported. In general, the ecological validity of most methods is of a low to moderate degree, with some traditional laboratory tasks proving unsatisfactory. A few tasks demonstrated acceptable degrees of ecological validity but even these require improvement. It is concluded that future advances in ecological validity are likely to come from: (a) a greater reliance on assessments of the target behaviors in natural settings and (b) combining several of the more promising tasks and analogue methods into a battery that is taken over longer time intervals than has been customary and averaged across repeated administrations.
Similar content being viewed by others
References
Achenbach, T. M., & Edelbrock, C. (1983).Manual for the Child Behavior Checklist and Revised Child Behavior Profile. Burlington, VT: Thomas Achenbach.
Achenbach, T. M., McConaughy, S. H., & Howell, C. T. (1987). Child/adolescent behavioral and emotional problems: Implications of cross-informant correlations for situational specificity.Psychological Bulletin, 101, 213–232.
Aman, M. G., & Turbott, S. H. (1986). Incidental learning, distraction, and sustained attention in hyperactive and control subjects.Journal of Abnormal Child Psychology, 14, 441–455.
American Psychiatric Association. (1980).Diagnostic and statistical manual of mental disorders (3rd ed.). Washington, DC: Author.
American Psychiatric Association (1987).Diagnostic and statistical manual of mental disorders (3rd ed. revised). Washington, DC: Author.
Anastasi, A. (1976).Psychological testing (4th ed.). New York: Macmillan.
Atkins, M. S., Pelham, W. E., & Licht, M. H. (1985). A comparison of objective classroom measures and teacher ratings of attention deficit disorder.Journal of Abnormal Child Psychology, 13, 155–167.
Barkley, R. A. (1977a). A review of stimulant drug research with hyperactive children.Journal of Child Psychology and Psychiatry, 18, 137–165.
Barkley, R. A. (1977b). The effects of methylphenidate on various measures of activity level and attention in hyperkinetic children.Journal of Abnormal Child Psychology, 5, 351–369.
Barkley, R. A. (1981).Hyperactive children: A handbook for diagnosis and treatment. New York: Guilford.
Barkley, R. A. (1988a). Attention deficit-hyperactivity disorder. In E. Mash & L. Terdal (Eds.),Behavioral assessment of childhood disorders (2nd ed., pp. 69–104). New York: Guilford Press.
Barkley, R. A. (1988b). Attention. In M. Tramontana & S. Hooper (Eds.),Issues in child clinical neuropsychology (pp. 145–176). New York: Plenum Press.
Barkley, R. A. (1988c). Child behavior rating scales and checklists. In M. Rutter, H. Tuma, & I. Lann (Eds.),Assessment and diagnosis in child psychopathology (pp. 113–155). New York: Guilford Press.
Barkley, R. A. (1990a).Attention deficit hyperactivity disorder: A handbook for diagnosis and treatment. New York: Guilford Press.
Barkley, R. A. (1990b). The problem of stimulus control and rule-governed behavior in children with attention deficit disorder with hyperactivity. In J. Swanson & L. Bloomingdale (Eds.),Attention deficit disorders (pp. 203–228). New York: Pergamon Press.
Barkley, R. A., & Cunningham, C. E. (1979). Stimulant drugs and activity level in hyperactive children.American Journal of Orthopsychiatry, 49, 491–499.
Barkley, R. A., DuPaul, G. J., & McMurray, M. B. (1990). A comprehensive evaluation of attention deficit disorder with and without hyperactivity as defined by research criteria.Journal of Consulting and Clinical Psychology, 58, 775–789.
Barkley, R A, DuPaul, G. J., & McMurray, M. B. (1991). Attention deficit disorder with and without hyperactivity: Clinical response to three dose levels of methylphenidate.Pediatrics, 87.
Barkley, R. A., Fischer, M., Newby, R., & Breen, M. (1988). Development of a multi-method clinical protocol for assessing stimulant drug responses in ADHD children.Journal of Clinical Child Psychology, 17, 14–24.
Barkley, R. A., McMurray, M. B., Edelbrock, C. S., & Robbins, K. (1989). The response of aggressive and non-aggressive ADHD children to two doses of methylphenidate.Journal of the American Academy of Child and Adolescent Psychiatry, 28, 873–881.
Barkley, R. A., & Ullman, D. G. (1975). A comparison of objective measures of activity and distractibility in hyperactive and nonhyperactive children.Journal of Abnormal Child Psychology, 3, 231–244.
Breen, M. J. (1989). ADHD girls and boys: An analysis of attentional, emotional, cognitive, and family variables.Journal of Child Psychology and Psychiatry, 30, 711–716.
Brown, R. T. (1982). A developmental analysis of visual and auditory sustained attention and reflectionimpulsivity in hyperactive and normal children.Journal of Learning Disabilities, 15, 353–357.
Brown, R. T., & Quay, L. C. (1977). Reflection-impulsivity of normal and behavior-disordered children.Journal of Abnormal Child Psychology, 5, 457–462.
Brown, R. T., & Wynne, M. E. (1982a). Correlates of teacher ratings, sustained attention, and impulsivity in hyperactive and normal boys.Journal of Clinical Child Psychology, 11, 262–267.
Brown, R. T., & Wynne, M. E. (1982b). An analysis of attentional components in hyperactive and normal boys.Journal of Learning Disabilities, 15, 358–363.
Buss, D. M., Block, J. H., & Block, J. (1980). Preschool activity level: Personality correlates and developmental implications.Child Development, 51, 401–408.
Cairns, E., & Cammock, T. (1978). Development of a more reliable version of the Matching Familiar Figures Test.Developmental Psychology, 11, 244–248.
Campbell, D. T., & Fiske, D. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix.Psychological Bulletin, 56, 81–105.
Campbell, S. B. (1974). Cognitive styles and behavior problems of clinic boys: A comparison of epileptic, hyperactive, learning-disabled, and normal groups.Journal of Abnormal Child Psychology, 2, 307–312.
Campbell, S. B., Douglas, V. I., & Morgenstern, G. (1971). Cognitive styles in hyperactive children and the effect of methylphenidate.Journal of Child Psychology and Psychiatry, 12, 55–67.
Campbell, S. B., Szumowski, E. K., Ewing, L. J., Gluck, D. S., & Breaux, A. M. (1982). A multidimensional assessment of parent-identified behavior problem toddlers.Journal of Abnormal Child Psychology, 10, 569–592.
Charles, L., Schain, R. J., Zelniker, T., & Guthrie, D. (1979). Effects of methylphenidate on hyperactive children's ability to sustain attention.Pediatrics, 64, 412–418.
Copeland, A. P., & Weissbrod, C. S. (1978). Behavioral correlates of the hyperactivity factor of the Conners Teacher Questionnaire.Journal of Abnormal Child Psychology, 6, 339–343.
de Haas, P. A. (1986). Attention styles and peer relationships of hyperactive and normal boys and girls.Journal of Abnormal Child Psychology, 14, 457–467.
de Haas, P. A., & Young, R. D. (1984). Attention styles of hyperactive and normal girls.Journal of Abnormal Child Psychology, 12, 531–546.
Douglas, V. I. (1972). Stop, look, and listen: The problem of sustained attention and impulse control in hyperactive and normal children.Canadian Journal of Behaviouml Science, 4, 259–282.
Douglas, V. I. (1983). Attention and cognitive problems. In M. Rutter (Ed.),Developmental neuropsychiatry (pp. 280–329). New York: Guilford Press.
Douglas, V. I., Barr, R. G., Amin, K, O'Neill, M. E., & Britton, B. G. (1988). Dosage effects and individual responsivity to methylphenidate in attention deficit disorder.Journal of Child Psychology and Psychiatry, 29, 453–476.
Douglas, V. I., & Peters, K. G. (1979). Toward a clearer definition of the attentional deficit of hyperactive children. In G. A. Hale & M. Lewis (Eds.).Attention cognitive development (pp. 173–248). New York: Plenum Press.
Draeger, S., Prior, M., & Sanson, A. (1986). Visual and auditory attention performance in hyperactive children: Competence or compliance?Journal of Abnormal Child Psychology, 14, 411–424.
DuPaul, G. J. (1990).Parent and teacher ratings of ADHD symptoms: Psychometric properties in a community-based sample. Manuscript submitted for publication.
Firestone, P., & Martin, J. E. (1979). An analysis of the hyperactive syndrome: A comparison of hyperactive, behavior problem, asthmatic, and normal children.Journal of Abnormal Child Psychology, 7, 261–273.
Fischer, M., Barkley, R. A., Edelbrock, C. S., & Smallish, L. (1990). The adolescent outcome of hyperactive children diagnosed by research criteria, II: Academic, attentional, and neuropsychological status.Journal of Consulting and Clinical Psychology, 58, 580–588.
Fuhrman, M. J., & Kendall, P. C. (1986). Cognitive tempo and behavioral adjustment in children.Cognitive Therapy and Research, 10, 45–50.
Golden, M., Montare, A., & Bridger, W. (1977). Verbal control of delay behavior in two year old boys as a function of social class.Child Development, 48, 1107–1111.
Goldstein, S., & Goldstein, M. (1990).Managing attention disorders in children: A guide for practitioners. New York: Wiley.
Gordon, M. (1979). The assessment of impulsivity and mediating behaviors in hyperactive and non-hyperactive children.Journal of Abnormal Child Psychology, 7, 317–326.
Gordon, M. (1983).The Gordon Diagnostic System. DeWitt, NY: Gordon Systems.
Gordon, M. (1985, August).Current GDS research: The vicissitudes of validation. Paper presented at the 93rd Annual Meeting of the American Psychological Association, Los Angeles.
Gordon, M., & McClure, F. D. (1983, August).The objective assessment of attention deficit disorders. Paper presented at the 91st Annual Convention of the American Psychological Association, Anaheim, CA.
Gordon, M., & Mettelman, B. B. (1988). The assessment of attention: I. Standardization and reliability of a behavior based measure.Journal of Clinical Psychology, 44, 682–690.
Goyette, C. H., Conners, C. K., & Ulrich, R. F. (1978). Normative data for Revised Conners Parent and Teacher Rating Scales.Journal of Abnormal Child Psychology, 6, 221–236.
Haenlein, M., & Caul, W. F. (1987). Attention deficit disorder with hyperactivity: A specific hypothesis of reward dysfunction.Journal of the American Academy of Child and Adolescent Psychiatry, 26, 356–362.
Halperin, J. M., Wolf, L. E., Pascaulvaca, D. M., Newcorn, J. H., Healey, J. M., O'Brien, J. D., Morganstein, A., & Young, G. (1988). Differential assessment of attention and impulsivity in children.Journal of the American Academy of Child and Adolescent Psychiatry, 27, 326–329.
Hochberg, Y., & Tamhane, A. C. (1987).Multiple comparisons procedures. New York: Wiley.
Homatidis, S., & Konstantareas, M. M. (1981). Assessment of hyperactivity: Isolating measures of high discriminant ability.Journal of Consulting and Clinical Psychology, 49, 533–541.
Horn, W. F., Wagner, A. E., & Ialongo, N. (1989). Sex differences in school-aged children with pervasive attention deficit hyperactivity disorder.Journal of Abnormal Child Psychology, 17, 109–125.
Hutt, C, Hutt, S. J., & Ounsted, C. (1963). A method for the study of children's behaviour.Developmental Medicine and Child Neurology, 5, 233.
Kagan, J. (1966). Reflection-impulsivity: The generality and dynamics of conceptual tempo.Journal of Abnormal Psychology, 71, 17–24.
Kazdin, A. E., Esveldt-Dawson, K., & Loar, L. L. (1983). Correspondence of teacher ratings and direct observations of classroom behavior of psychiatric inpatient children.Journal of Abnormal Child Psychology, 11, 549–564.
Kendall, P. C., & Brophy, C. (1981). Activity and attentional correlates of teacher ratings of hyperactivity.Journal of Pediatric Psychology, 6, 451–458.
Keogh, B. K., & Margolis, J. S. (1976). A component analysis of attentional problems of educationally handicapped boys.Journal of Abnormal Child Psychology, 4, 349–359.
Klee, S. H., & Garfinkel, B. D. (1983). The computerized continuous performance task: A new measure of inattention.Journal of Abnormal Child Psychology, 11, 487–496.
Koriath, U., Gualtieri, T., Van Bourgondien, M. E., Quade, D., & Werry, J. S. (1985). Construct validity of clinical diagnosis in pediatric psychiatry: Relationship among measures.Journal of the American Academy of Child Psychiatry, 242, 429–436.
Kuehne, C., Kehle, T. J., & McMahon, W. (1987). Differences between children with attention deficit disorder, children with specific learning disabilities, and normal children.Journal of School Psychology, 25, 161–166.
Luk, S. (1985). Direct observations studies of hyperactive behaviors.Journal of the American Academy of Child Psychiatry, 24, 338–344.
Margolis, J. S. (1972).Academic correlates of sustained attention. Unpublished doctoral dissertation, University of California, Los Angeles.
McClure, F. D., & Gordon, M. (1984). Performance of disturbed hyperactive and nonhyperactive children on an objective measure of hyperactivity.Journal of Abnormal Child Psychology, 12, 561–572.
Milich, R. (1984). Cross-sectional and longitudinal observations of activity level and sustained attention in a normative sample.Journal of Abnormal Child Psychology, 12, 261–276.
Milich, R., & Kramer, J. (1984). Reflections on impulsivity: An empirical investigation of impulsivity as a construct. In K. Gadow & I. Bialer (Eds.),Advances in learning and behavioral disabilities (Vol. 3, pp. 117–150). Greenwich, CT: JAI Press.
Milich, R., Loney, J., & Landau, S. (1982). The independent dimensions of hyperactivity and aggression: A validation with playroom observation data.Journal of Abnormal Psychology, 91, 183–198.
Milich, R., Loney, J., & Roberts, M. A. (1986). Playroom observations of activity level and sustained attention: Two-year stability.Journal of Consulting and Clinical Psychology, 54, 272–274.
O'Dougherty, M., Nuechterlein, K H., & Drew, B. (1984). Hyperactive and hypoxic children: signal detection, sustained attention, and behavior.Journal of Abnormal Psychology, 93, 178–191.
Pascaulvaca, D. M., Wolf, L. E, Healey, J. M., Tweedy, J. R., & Halperin, J. M. (1988, January).Sex differences in attention and behavior in school-age children. Paper presented at the 16th Annual Meeting of the International Neuropsychological Society, New Orleans.
Pelham, W. E., Bender, M. E., Caddell, J., Booth, S., & Moorer, S. H. (1985). Methylphenidate and children with attention deficit disorder.Archives of General Psychiatry, 42, 948–952.
Pope, L. (1970). Motor activity in brain-injured children.American Jojrnal of Orthopsychiatry, 40, 783–794.
Prinz, R. J., Tarnowski, K. J., & Nay, S. M. (1984). Assessment of sustained attention and distraction in children using a classroom analogue task.Journal of Clinical Child Psychology, 13, 250–256.
Quay, H. C. (1988). The behavioral reward and inhibition system in childhood behavior disorder. In L. Bloomingdale (Ed.),Attention deficit disorder, Vol. 3: New research in attention, treatment, and psychopharmacology (pp. 176–186). New York: Pergamon Press.
Rapport, M. D., DuPaul, G. J., Stoner, G., Birmingham, B. K., & Masse, G. (1985). Attention deficit disorder with hyperactivity: Differential effects of methylphenidate on impulsivity.Pediatrics, 76, 938–943.
Rapport, M. D., DuPaul, G. J., Stoner, G., & Jones, J. T. (1986). Comparing classroom and clinic measures of attention deficit disorder: Differential, idiosyncratic, and dose-response effects of methylphenidate.Journal of Consulting and Clinical Psychology, 54, 334–341.
Rapport, M. D., Jones, J. T., DuPaul, G. J., Kelly, K. L., Gardner, M. J., Tucker, S. B., & Shea, M. S. (1987). Attention deficit disorder and methylphenidate: Group and singlesubject analyses of dose effects on attention in clinic and classroom settings.Journal of Clinical Child Psychology, 16, 329–338.
Rapport, M. D., Tucker, S. B., DuPaul, G. J., Merlo, M., & Stoner, G. (1986). Hyperactivity and frustration: The influence of control over and size of rewards in delaying gratification.Journal of Abnormal Child Psychology, 14, 191–204.
Ribbler, A. (1988, January).The relationship between attention deficits and childhood behavior problems. Paper presented at the 16th Annual Meeting of the International Neuropsychological Society, New Orleans.
Roberts, M. A. (1987). How is playroom behavior observation used in the diagnosis of attention deficit disorder? In J. Loney (Ed.),The young hyperactive child: Answers to questions about diagnosis, prognosis, and treatment (pp. 65–74). Binghamton, NY: Haworth Press.
Roberts, M. A. (1990). A behavioral observation method for differentiating hyperactive and aggressive boys.Journal of Abnormal Child Psychology, 18, 131–142.
Ross, D. M., & Ross, S. A. (1982).Hyperactivity: Current issues, research, and theory (2nd ed.). New York: Wiley.
Routh, D. K., & Schroeder, C. S. (1976). Standardized playroom measures as indices of hyperactivity.Journal of Abnormal Child Psychology, 4, 199–207.
Routh, D. K., Schroeder, C. S., & O'Tuama, L. (1974). Development of activity level in children.Developmental Psychology, 10, 163–168.
Sandberg, S. T., Rutter, M., & Taylor, E. (1978). Hyperkinetic disorder in psychiatric clinic attenders.Developmental Medicine and Child Neurology, 20, 279–299.
Schachar, R., Sandberg, S., & Rutter, M. (1986). Agreement between teachers' ratings and observations of hyperactivity, inattentiveness, and defiance.Journal of Abnormal Child Psychology, 14, 331–345.
Schulman, J. L., & Reisman, J. M. (1959). An objective measure of hyperactivity.American Journal of Mental Deficiency, 64, 455–456.
Seidel, W. T., & Joschko, M. (1990). Evidence of difficulties in sustained attention in children with ADDH.Journal of Abnormal Child Psychology, 18, 217–229.
Shaffer, D., McNamara, N., & Pincus, J. H. (1974). Controlled observations on patterns of activity, attention, and impulsivity in brain-damaged and psychiatrically disturbed boys.Psychological Medicine, 4, 4–18.
Sleator, E. K., & Pelham, W. E. (1986).Attention deficit disorder. Norwalk, CT: AppletonCentury-Crofts.
Sprague, R. L., & Sleator, E. K. (1977). Methylphenidate in hyperkinetic children: Difference in dose effects on learning and social behavior.Science, 198, 1274–1276.
Stevens, T. M., Kupst, M. J., Suran, B. G., & Schulman, J. L. (1978). Activity level: A comparison between actometer scores and observer ratings.Journal of Abnormal Child Psychology, 6, 163–173.
Swanson, J., & Kinsbourne, M. (1979). The cognitive effects of stimulant drugs on hyperactive children. In G. Hale & M. Lewis (Eds.),Attention and cognitive development (pp. 249–274). New York: Plenum Press.
Touwen, B. C. L., & Kalverboer, A. F. (1973). Neurologic and behavioral assessment of children with minimal brain dysfunction.Seminars in Psychiatry, 5, 79–94.
Tryon, W. W. (1984). Principles and methods of mechanically measuring motor activity.Behavioral Assessment, 6, 129–140.
Ullman, D. G., Barkley, R. A., & Brown, H. W. (1978). The behavioral symptoms of hyperkinetic children who successfully responded to stimulant drug treatment.American Journal of Orthopsychiatry, 48, 425–437.
van der Meere, J., & Sergeant, J. (1988a). Focused attention in pervasively hyperactive children.Journal of Abnormal Child Psychology, 16, 627–640.
van der Meere, J., & Sergeant, J. (1988b). Controlled processing and vigilance in hyperactivity: Time will tell.Journal of Abnormal Child Psychology, 16, 641–656.
Weissberg, R., Ruff, H. A., & Lawson, K. R. (1990). The usefulness of reaction time tasks in studying attention and organization of behavior in young children.Journal of Developmental and Behavioral Pediatrics, 11, 59–64.
Werry, J. S., Elkind, G. S., & Reeves, J. C. (1987). Attention deficit, conduct, oppositional, and anxiety disorders in children: III. Laboratory differences.Journal of Abnormal Child Psychology, 15, 409–428.
Author information
Authors and Affiliations
Additional information
An earlier version of this paper was presented at the International Symposium on Attention Deficit Hyperactivity Disorder, Amsterdam, The Netherlands, April 6 and 7, 1989. The author wishes to thank Charles E. Cunningham, Ph.D., Susan Campbell, Ph.D., Mark Rapport, Ph.D., Eric Mash, Ph.D., and Herbert Quay, Ph.D., for their comments on an earlier version of this paper, and to Marcel Kinsbourne, M.D., for his comments on the presentation itself. The author was supported by funds from NIMH grants 41464, 41583, and 42181 during the preparation of this review.
Rights and permissions
About this article
Cite this article
Barkley, R.A. The ecological validity of laboratory and analogue assessment methods of ADHD symptoms. J Abnorm Child Psychol 19, 149–178 (1991). https://doi.org/10.1007/BF00909976
Revised:
Issue Date:
DOI: https://doi.org/10.1007/BF00909976