Abstract
The identification of reliable and valid means for evaluating the effectiveness of school-based treatments and completing diagnostic evaluations of middle school aged students are needed. The present study examined the inter-rater agreement of teacher ratings and the relationship between ratings and observational data in a middle school setting. The data are interpreted in the context of differences between a secondary and elementary school setting. Teacher ratings and observational data were collected regularly over the course of two academic years for middle school students diagnosed with ADHD. The results indicate low rates of inter-rater agreement as well as low rates of agreement between teachers and observational data, and between observational data collected in different classrooms. Inter-rater agreement was lowest in late fall and gradually increased over the second half of the year. Implications for conducting treatment outcome evaluations of school-based treatment programs and diagnostic evaluations are discussed.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Abikoff, H. B., Jensen, P. S., Arnold, L. L. E., Hoza, B., Hechtman, L., Pollack, S., Martin, D., Alvir, J., March, J. S., Hinshaw, S., Vitielo, B., Newcorn, J., Greiner, A., Cantwell, D P., Conners, C. K., Elliott, G., Greenhill, L. L., Kraemer, H., Pelham, W. E., Severe, J. B., Swanson, J. M., Wells, K., & Wigal, T. (2002). Observed classroom behavior of children with ADHD: Relationship to gender and comorbidity. Journal of Abnormal Child Psychology, 30, 349–359.
Abikoff, H. B., Courtney, M., Pelham, W. E., & Koplewicz, H. S. (1993). Teachers' ratings of disruptive behaviors: The influence of halo effects. Journal of Abnormal Child Psychology, 21, 519–534.
Achenbach, T. M. (1991). Integrative guide for the 1991 CBCL/4-18, YSR, and TRF profiles. Burlington: University of Vermont, Department of Psychiatry.
Achenbach, T. M., McConaughy, S. H., & Howell, C. T. (1987). Child/adolescent behavioral and emotional problems: Implications of cross-informant correlations for situational specificity. Psychological Bulletin, 101, 213–232.
American Academy of Child and Adolescent Psychiatry (1997). Practice parameters for the assessment and treatment of children, adolescents, and adults with Attention-Deficit/Hyperactivity Disorder. Journal of the American Academy of Child and Adolescent Psychiatry, 36, 85S–121S.
American Academy of Pediatrics (2000). Clinical practice guidelines: Diagnosis and evaluation of the child with Attention-Deficit/Hyperactivity Disorder. Pediatrics, 105, 1158–1170.
American Psychiatric Association (1994). Diagnostic and statistical manual of mental disorders (4th ed.). Washington, DC: Author.
Barkley, R. A., Edwards, G., Laneri, M., Fletcher, K., & Metevia, L. (2001). The efficacy of problem-solving communication training alone, behavior management training alone, and their combination for parent-adolescent conflict in teenagers with ADHD and ODD. Journal of Consulting and Clinical Psychology, 69, 926–941.
Bartko, J. J., & Carpenter, W. T. (1976). On the methods and theory of reliability. Journal of Nervous and Mental Disease, 163, 307–317.
Biederman, J., Faraone, S. V., Mick, E., Williamson, S., Wilens, T. E., Spencer, T. J., Weber, W., Fetton, F., Kraus, I., Pert, J., & Zallen, B. (1999). Clinical correlates of ADHD in females: Findings from a large group of girls ascertained from pediatric and psychiatric referral sources. Journal of the American Academy of Child and Adolescent Psychiatry, 38, 966–975.
Chang, D. F., & Sue, S. (2003). The effects of race and problem type on teachers' assessments of student behavior. Journal of Consulting and Clinical Psychology, 7, 235–242.
DuPaul, G. J., Power, T. J., Anastopoulos, A. D., & Reid, R. (1998). ADHD Rating Scale—IV. New York: The Guilford Press.
Evans, S. W., Axelrod, J. L., & Langberg, J. M., (2004). Efficacy of a school-based treatment program for middle school youth with ADHD: Pilot data. Behavior Modification, 28, 528–547.
Evans, S. W., Langberg, J., Raggi, V., Allen, J., & Buvinger, E. (in press). Development of a school-based treatment program for middle school youth with ADHD. Journal of Attention Disorders.
Evans, S. W., Raggi, V., Thompson, J., & Garland, B. (2004). Clinical utility of the parent version of the Impairment Rating Scale for children. Manuscript submitted for publication.
Fabiano, G. A., & Pelham, W. E. (2002). Measuring impairment in children with attention-deficit hyperactivity disorder. The ADHD Report, 10, 6–10.
Fabiano, G. A., Pelham, W. E., Gnagy, E. M., Waschbusch, D. A., Lahey, B. B., Chronis, A. M., Onyango, A. N., Kipp, H., & Williams, A. (2004). The reliability and validity of the Children's Impairment Rating Scale: A practical measure of impairment in children with ADHD. Manuscript submitted for publication.
Gomez, R., Burns, G. L., Walsh, J. A., & De Moura, M. A. (2003). A multitrait-multisource confirmatory factor analytic approach to the construct validity of ADHD rating scales. Psychological Assessment, 15, 3–16.
Greenbaum, P. E., Dedrick, R. F., Prange, M. E., & Friedman, R. M. (1994). Parent, teacher, and child ratings of problem behaviors of youngsters with serious emotional disturbances. Psychological Assessment, 6, 141–148.
Jensen, P. S., Rubio-Stipec, M., Canino, G., Bird, H. R., Dulcan, M. K., Schwab-Stone, M. E., & Lahey, B. B., (1999). Parent and child contributions to diagnosis of mental disorder: Are both informants always necessary? Journal of the American Academy of Child and Adolescent Psychiatry, 38, 1569–1579.
Kraemer, H. C., Measelle, J. R., Ablow, J. C., Exxes, M. J., Boyce, W. T., & Kupfer, D. J. (2003). A new approach to integrating data from multiple informants in psychiatric assessments and research: Mixing and matching contexts and perspectives. American Journal of Psychiatry, 160, 1566–1577.
Lahey, B. B., Applegate, B., McBurnett, K., Biederman, J., Greenhill, L., Hynd, G. W., Barkley, R. A., Newcorn, J., Jensen, P., Richters, J., Garfinkel, B., Kerdyk, L., Frick, P. J., Ollendick, T., Perez, D., Hart, E. L., Waldman, I., & Shaffer, D. (1994). DSM-IV field trials for attention deficit hyperactivity disorder in children and adolescents. The American Journal of Psychiatry, 151, 1673–1685.
Lee, S. W., Elliott, J., & Barbour, J. D. (1994). A comparison of cross-informant behavior ratings in school-based diagnosis. Behavioral Disorders, 19, 87–97.
Merrel, K. W. (2000). Informant reports: Theory and research in using child behavior rating scales in school settings. In E. S. Shapiro & T. R. Kratochwill (Eds.), Behavioral assessment in schools: Theory, research, and clinical foundations (2nd ed., pp. 233–256). New York: The Guilford Press.
Mitsis, E. M., McKay, K. E., Schulz, K. P., Newcorn, J. H., & Halperin, J. M. (2000). Parent-teacher concordance for DSM-IV attention-deficit/hyperactivity disorder in a clinic-referred sample. Journal of the American Academy of Child and Adolescent Psychiatry, 39, 308–313.
Molina, B. S. G., Pelham, W. E., Blumenthal, J., & Galiszewski, E. (1998). Agreement among teachers' behavior ratings of adolescents with a childhood history of attention deficit hyperactivity disorder. Journal of Clinical Child Psychology, 27, 330–339.
Pelham, W. E., Gnagy, E., Burrows-MacLean, L., Williams, A., Fabiano, G. A., Morrisey, S. M., Chronis, A. M., Forehand, G. L., Nguyen, C. A., Hoffman, M. T., Lock, T. M., Fielbelkorn, K., Coles, E. K., Panahon, C. J., Steiner, R. L., Meichenbaum, D. L., Onyango, A. N., & Morse, G. D. (2001). Once-a-day Concerta methylphenidate versus three-times-daily methylphenidate in laboratory and natural settings. Pediatrics, 107, 1–15.
Phares, V., Compas, B. E., & Howell, D. C. (1989). Perspectives on child behavior problems: Comparisons of children's self-reports with parent and teacher reports. Psychological Assessment, 1, 68–71.
Pliszka, S. R., Lopez, M., Crismon, L., Toprac, M. G., Hughes, C. W., Emslie, G. J., & Boemer, C. (2003). A feasibility study of the Children's Medication Algorithm Project (CMAP) algorithm for the treatment of ADHD. Journal of the American Academy of Child and Adolescent Psychiatry, 42, 279–287.
Reid, J. B., Eddy, J. M., Fetrow, R. A., & Stoolmiller, M. (1999). Description and immediate impacts of a preventive intervention for conduct problems. American Journal of Community Psychology, 27, 483–517.
Reynolds, C., & Kamphaus, R. (1992). Behavior Assessment System for Children: Manual. Circle Pines, MN: American Guidance.
Robin, A. L. (1990). Training families with ADHD adolescents. In R. A. Barkley (Ed.), Attention Deficit Hyperactivity Disorder. A Handbook for Diagnosis and Treatment (pp. 462–497). New York: Guilford Press.
Shaffer, D., Fisher, P., Lucas, C. P., Dulcan, M. K., & Schwab-Stone, M. E. (2000). NIMH Diagnostic Interview Schedule for Children version IV (NIMH DISC-IV): Description, differences from previous versions, and reliability of some common diagnoses. Journal of the American Academy of Child and Adolescent Psychiatry, 39, 28–38.
Simpson, R. G. (1991). Agreement among teachers of secondary students in using the Revised Behavior Problem Checklist to identify deviant behavior. Behavioral Disorders, 17, 66–71.
Wechsler, D. (1991). Wechsler Intelligence Scale for Children-Third Edition. New York: Psychological Corporation.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Evans, S.W., Allen, J., Moore, S. et al. Measuring Symptoms and Functioning of Youth with ADHD in Middle Schools. J Abnorm Child Psychol 33, 695–706 (2005). https://doi.org/10.1007/s10802-005-7648-0
Received:
Revised:
Accepted:
Issue Date:
DOI: https://doi.org/10.1007/s10802-005-7648-0