The objectivity of the Autism Diagnostic Observation Schedule (ADOS) in naturalistic clinical settings
- 1.2k Downloads
The Autism Diagnostic Observation Schedule (ADOS) is a first-choice diagnostic tool in autism spectrum disorder (ASD). Excellent interpersonal objectivity (interrater reliability) has been demonstrated for the ADOS under optimal conditions, i.e., within groups of highly trained “research reliable” examiners in research setting. We investigated the spontaneous interrater reliability among clinically trained ADOS users across multiple sites in clinical routine. Forty videotaped administrations of the ADOS modules 1–4 were rated by five different raters each from a pool of in total 15 raters affiliated to 13 different clinical sites. G(q,k) coefficients (analogous to intraclass correlations), kappas (ĸ) and percent agreement (PA) were calculated. The median interrater reliability for items across the four modules was G(q,k) = .74–.83, with the single ADOS items ranging from .23 to .94. G(q,k) for total scores was .85–.92. For diagnostic classification (ASD/non-spectrum), PA was 64–82 % and Fleiss’ ĸ .19–.55. Objectivity was lower for pervasive developmental disorder not otherwise specified and non-spectrum diagnoses as compared to autism. Interrater reliabilities of the ADOS items and domain totals among clinical users across multiple sites were in the same range as previously reported for research reliable users, while the one for diagnostic classification was lower. Differences in sample characteristics, rater skills and statistics compared with previous studies are discussed. Findings endorse the objectivity of the ADOS in naturalistic clinical settings, but also pinpoint its limitations and the need and value of adequate and continuous rater training.
KeywordsAutism spectrum disorder Interrater reliability Diagnostic instrument
We thank the Child and Adolescent Psychiatry of Stockholm County Council, Region Västra Götaland and Region Skåne, PRIMA Child and Adult Psychiatry, Tiohundra Norrtälje and Astrid Lindgren Children’s Hospital and Sachs’ Children and Youth Hospital, Stockholm County Council, for the generous cooperation in realizing this study. We also thank our funders Frimurare Barnhuset, Swedish Research Council, Stockholm County Council and finally Karolinska Institutet. Finally, we are especially grateful toward all the children and parents that participated in the project.
Compliance with ethical standards
Conflicts of interest
On behalf of all authors, the corresponding author states that there is no conflict of interest.
- 2.Volkmar F, Siegel M, Woodbury-Smith M, King B, McCracken J, State M, American Academy of C, Adolescent Psychiatry Committee on Quality I (2014) Practice parameter for the assessment and treatment of children and adolescents with autism spectrum disorder. J Am Acad Child Adolesc Psychiatry 53(2):237–257CrossRefPubMedGoogle Scholar
- 3.Autism: Recognition, Referral, Diagnosis and Management of Adults on the Autism Spectrum (2012). The British Psychological Society & The Royal College of Psychiatrists. Leicester, UKGoogle Scholar
- 4.Swedish Council on Health Technology Assessment (SBU) (2013) Autismspektrumtillstånd. Diagnostik och insatser, vårdens organisation och patientens delaktighet—En systematisk litteraturöversikt, vol 215. Swedish Council on Health Technology AssessmentGoogle Scholar
- 5.American Psychiatric Association (APA) (2013) Diagnostic and statistical manual of mental disorders DSM-5, 5th edn. American Psychiatric Association, ArlingtonGoogle Scholar
- 6.Lord C, Rutter M, DiLavore P, Risi S (1999) Autism Diagnostic Observation Schedule (ADOS). Western Psychological Publishing, Los AngelesGoogle Scholar
- 7.Lord C, Rutter M, DiLavore P, Risi S, Gotham K, Bishop S (2012) Autism Diagnostic Observation Schedule, Second Edition (ADOS-2) Manual (Part I). Western Psychological Services, TorranceGoogle Scholar
- 8.Rutter M, Le Couteur A, Lord C (2003) Autism Diagnostic Interview Revised (ADI-R). Western Psychological Services, Los AngelesGoogle Scholar
- 12.Wolff JJ, Gu H, Gerig G, Elison JT, Styner M, Gouttard S, Botteron KN, Dager SR, Dawson G, Estes AM, Evans AC, Hazlett HC, Kostopoulos P, McKinstry RC, Paterson SJ, Schultz RT, Zwaigenbaum L, Piven J (2012) Differences in white matter fiber tract development present from 6 to 24 months in infants with autism. Am J Psychiatry 169(6):589–600CrossRefPubMedPubMedCentralGoogle Scholar
- 18.Gotham K, Risi S, Dawson G, Tager-Flusberg H, Joseph R, Carter A, Hepburn S, McMahon W, Rodier P, Hyman SL, Sigman M, Rogers S, Landa R, Spence MA, Osann K, Flodman P, Volkmar F, Hollander E, Buxbaum J, Pickles A, Lord C (2008) A replication of the Autism Diagnostic Observation Schedule (ADOS) revised algorithms. J Am Acad Child Adolesc Psychiatry 47(6):642–651CrossRefPubMedPubMedCentralGoogle Scholar
- 20.Bastiaansen JA, Meffert H, Hein S, Huizinga P, Ketelaars C, Pijnenborg M, Bartels A, Minderaa R, Keysers C, de Bildt A (2011) Diagnosing autism spectrum disorders in adults: the use of Autism Diagnostic Observation Schedule (ADOS) module 4. J Autism Dev Disord 41(9):1256–1266CrossRefPubMedGoogle Scholar
- 26.Lord C, Risi S, Lambrecht L, Cook EH Jr, Leventhal BL, DiLavore PC, Pickles A, Rutter M (2000) The autism diagnostic observation schedule-generic: a standard measure of social and communication deficits associated with the spectrum of autism. J Autism Dev Disord 30(3):205–223CrossRefPubMedGoogle Scholar
- 27.Oosterling I, Roos S, de Bildt A, Rommelse N, de Jonge M, Visser J, Lappenschaar M, Swinkels S, van der Gaag RJ, Buitelaar J (2010) Improved diagnostic validity of the ADOS revised algorithms: a replication study in an independent sample. J Autism Dev Disord 40(6):689–703CrossRefPubMedPubMedCentralGoogle Scholar
- 30.de Bildt A, Sytema S, Meffert H, Bastiaansen JCJ (2015) The Autism Diagnostic observation schedule, module 4: application of the revised algorithms in an independent, well-defined, Dutch sample (n = 93). J Autism Dev Disord 30 August 2015. doi: 10.1007/s10803-015-2532-4
- 34.Schendel DE, Diguiseppi C, Croen LA, Fallin MD, Reed PL, Schieve LA, Wiggins LD, Daniels J, Grether J, Levy SE, Miller L, Newschaffer C, Pinto-Martin J, Robinson C, Windham GC, Alexander A, Aylsworth AS, Bernal P, Bonner JD, Blaskey L, Bradley C, Collins J, Ferretti CJ, Farzadegan H, Giarelli E, Harvey M, Hepburn S, Herr M, Kaparich K, Landa R, Lee LC, Levenseller B, Meyerer S, Rahbar MH, Ratchford A, Reynolds A, Rosenberg S, Rusyniak J, Shapira SK, Smith K, Souders M, Thompson PA, Young L, Yeargin-Allsopp M (2012) The Study to Explore Early Development (SEED): a multisite epidemiologic study of autism by the Centers for Autism and Developmental Disabilities Research and Epidemiology (CADDRE) network. J Autism Dev Disord 42(10):2121–2140CrossRefPubMedPubMedCentralGoogle Scholar
- 39.Shavelson RJ, Webb NM (2006) Generalizability theory. Handbook of complementary methods in education research. Lawrence Erlbaum Associates Publishers, Mahwah, pp 309–322Google Scholar