Abstract
Observational measures can add objective data to both research and clinical evaluations of children’s behavior in the classroom. However, they pose challenges for training and attaining high levels of interrater reliability between observers. The Behavioral Observation of Students in Schools (BOSS) is a commonly used school-based observation instrument that is well adapted to measure symptoms of attention deficit/hyperactivity disorder (ADHD) in the classroom setting. Reliable use of the BOSS for clinical or research purposes requires training to reach reliable standards (kappa ≥ 0.80). The current study conducted training observations in one suburban and one urban elementary school in the Greater Boston area. To enhance interrater reliability and reduce training time, supplemental guidelines, including 30 additional rules to follow, were developed over two consecutive school years. The complete protocol was then used for training in the third school year. To reach sufficient interrater reliability (kappa ≥ 0.80) during training, 45 training observations were required in the first year while, in the third year, only 17 observations were required. High interrater reliability was sustained after training across all three school years, accumulating a total of 1,001 post-training observations. It is estimated that clinicians or researchers following this proposed protocol, who are naive to the BOSS, will require approximately 30 training observations to reach proficient reliability. We believe this protocol will make the BOSS more accessible for clinical and research usage, and the procedures used to obtain high interrater reliability using the BOSS are broadly applicable to a variety of observational measures.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
American Psychiatric Association. (2000). Diagnostic and Statistical Manual of Mental Disorders-Text revision (DSM-IV-TR) (4th ed.). Washington, DC: American Psychiatric Association.
Atkins, M. S., & Pelham, W. E. (1991). School-based assessment of attention deficit-hyperactivity disorder. Journal of Learning Disabilities, 24, 197–205. doi:10.1177/002221949102400403.
DuPaul, G. J., & Stoner, G. (1994). ADHD in the schools: Assessment and intervention strategies. New York: Guilford Press.
DuPaul, G. J., Anastopoulos, A. D., Shelton, T. L., Guevremont, D. C., & Metevia, L. (1992). Multimethod assessment of attention deficit hyperactivity disorder: The diagnostic utility of clinic-based tests. Journal of Clinical Child Psychology (now called Journal of Clinical Child and Adolescent Psychology), 21(4), 394–402. doi:10.1207/s15374424jccp2104_10.
DuPaul, G. J., Volpe, R. J., Jitendra, A. K., Lutz, J. G., Lorah, K. S., & Gruber, R. (2004). Elementary school students with AD/HD: Predictors for academia achievement. Journal of School Psychology, 42, 285–301.
Gelfand, D. M., & Hartmann, D. P. (1975). Child behavior analysis and therapy. New York: Pergamon Press.
Hartman, C. A., Rhee, S. H., Wilcutt, E. G., & Pennington, B. F. (2007). Modeling rater disagreement for ADHD: Are parents or teachers biased? Journal of Abnormal Child Psychology, 35, 536–542. doi:10.1007/s10802-007-9110-y.
Hintze, J. M. (2005). Psychometrics of direct observation. School Psychology Review, 34, 507–519.
Hintze, J. M., & Matthews, W. J. (2004). The generalizability of systematic direct observations across time and setting: A preliminary investigation of the psychometrics of behavioral assessment. School Psychology Review, 33, 258–270.
Hintze, J. M., Volpe, R. J., & Shapiro, E. S. (2008). Best practices in systematic direct observation of student behavior. In A. Thomas & J. Grimes (Eds.), Best practices in school psychology V (Vol. 2, pp. 319–335). Bethesda, MD: National Association of School Psychologists.
Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159–174. http://www.jstor.org/stable/2529310.
Lord, C., Risi, S., Lambrecht, L., Cook, E. W., Leventhal, B. L., DiLavore, P., et al. (2000). The autism diagnostic observation schedule-generic: A standard measure of social and communication deficits associated with spectrum of autism. Journal of Autism and Developmental Disorders, 30(3), 205–223.
Mautone, J. A., DuPaul, A. K., & Jitendra, A. K. (2005). The effects of computer-assisted instruction on the mathematics performance and classroom behavior of children with ADHD. Journal of Attention Disorder, 9, 301–312. doi:10.1177/1087054705278832.
Merikangas, K. R., He, J., Burstein, M., Swanson, S. A., Avenevoli, S., Cui, L., et al. (2010). Lifetime prevalence of mental disorders in U.S. adolescents: Results from the National Comorbidity Study-Adolescent Supplement (NCS-A). Journal of the American Academy of Child and Adolescent Psychiatry, 49(10), 980–989. doi:10.1016/j.jaac.2010.05.017.
Merrell, K. W. (1999). Behavioral, social, and emotional assessment of children and adolescents. Mawah, NJ: Lawrence Erlbaum Associates.
Miles, M. B., & Huberman, M. (1994). Qualitative data analysis: A sourcebook of new methods (2nd ed.). Beverly Hills, CA: Sage Publications.
Montague, M., McKinney, J. D., & Hocutt, A. (1994). Assessing students for attention deficit disorder. Intervention in School and Clinic, 29, 212–218.
Riley-Tilman, T. C., Kalberer, S. M., & Chafouleas, S.M. (2005). Selecting the right tool for the job: A review of behavior monitoring tools used to assess student response-to-intervention. The California School Psychologist, 10, 81–91. http://www.caspwebcasts.org/pdfs/JRNLv_081.pdf.
Shapiro, E. S. (2011). Behavior observations of students in schools. In E. S. Shapiro (Ed.), Academic skills problems fourth edition workbook (pp. 35–56). New York: Guilford Press.
Shapiro, E. S., & Heick, P. (2004). School psychologist assessment practices in the evaluation of students referred for social/behavioral/emotional problems. Psychology in the Schools, 41, 551–561. doi:10.1002/pits.10176.
Vile-Junod, R. E., DuPaul, G. J., Jitendra, A. K., Volpe, R. J., & Cleary, K. S. (2006). Classroom observations of students with and without ADHD: Differences across types of engagement. Journal of School Psychology, 44, 87–104.
Volpe, R. J., DiPerna, J. C., Hintze, J. M., & Shapiro, E. S. (2005). Observing students in a classroom setting: A review of seven coding schemes. School Psychology Review, 34(4), 454–474.
Wilson, M. S., & Reschly, D. J. (1996). Assessment in school psychology training and practice. School Psychology Review, 25, 9–23.
Acknowledgments
This research was financially supported by the Institute of Education Sciences (Project: Computer Attention Training in Schools for Children with Attention Deficit/Hyperactivity Disorder (ADHD), Project No: R305A090100). We would like to thank the Newton and Boston Public Schools and acknowledge R Chris Sheldrick for his support in the development of the Interrater Agreement Calculator and the following graduate students for their hard work and support of this research study: Jessica Bennett and Jessica Chen.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Steiner, N.J., Sidhu, T., Rene, K. et al. Development and testing of a direct observation code training protocol for elementary aged students with attention deficit/hyperactivity disorder. Educ Asse Eval Acc 25, 281–302 (2013). https://doi.org/10.1007/s11092-013-9166-x
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11092-013-9166-x