Advertisement

European Child & Adolescent Psychiatry

, Volume 25, Issue 7, pp 769–780 | Cite as

The objectivity of the Autism Diagnostic Observation Schedule (ADOS) in naturalistic clinical settings

  • Eric Zander
  • Charlotte Willfors
  • Steve Berggren
  • Nora Choque-Olsson
  • Christina Coco
  • Anna Elmund
  • Åsa Hedfors Moretti
  • Anette Holm
  • Ida Jifält
  • Renata Kosieradzki
  • Jenny Linder
  • Viviann Nordin
  • Karin Olafsdottir
  • Lina Poltrago
  • Sven Bölte
Original Contribution

Abstract

The Autism Diagnostic Observation Schedule (ADOS) is a first-choice diagnostic tool in autism spectrum disorder (ASD). Excellent interpersonal objectivity (interrater reliability) has been demonstrated for the ADOS under optimal conditions, i.e., within groups of highly trained “research reliable” examiners in research setting. We investigated the spontaneous interrater reliability among clinically trained ADOS users across multiple sites in clinical routine. Forty videotaped administrations of the ADOS modules 1–4 were rated by five different raters each from a pool of in total 15 raters affiliated to 13 different clinical sites. G(q,k) coefficients (analogous to intraclass correlations), kappas (ĸ) and percent agreement (PA) were calculated. The median interrater reliability for items across the four modules was G(q,k) = .74–.83, with the single ADOS items ranging from .23 to .94. G(q,k) for total scores was .85–.92. For diagnostic classification (ASD/non-spectrum), PA was 64–82 % and Fleiss’ ĸ .19–.55. Objectivity was lower for pervasive developmental disorder not otherwise specified and non-spectrum diagnoses as compared to autism. Interrater reliabilities of the ADOS items and domain totals among clinical users across multiple sites were in the same range as previously reported for research reliable users, while the one for diagnostic classification was lower. Differences in sample characteristics, rater skills and statistics compared with previous studies are discussed. Findings endorse the objectivity of the ADOS in naturalistic clinical settings, but also pinpoint its limitations and the need and value of adequate and continuous rater training.

Keywords

Autism spectrum disorder Interrater reliability Diagnostic instrument 

Notes

Acknowledgments

We thank the Child and Adolescent Psychiatry of Stockholm County Council, Region Västra Götaland and Region Skåne, PRIMA Child and Adult Psychiatry, Tiohundra Norrtälje and Astrid Lindgren Children’s Hospital and Sachs’ Children and Youth Hospital, Stockholm County Council, for the generous cooperation in realizing this study. We also thank our funders Frimurare Barnhuset, Swedish Research Council, Stockholm County Council and finally Karolinska Institutet. Finally, we are especially grateful toward all the children and parents that participated in the project.

Compliance with ethical standards

Conflicts of interest

On behalf of all authors, the corresponding author states that there is no conflict of interest.

Supplementary material

787_2015_793_MOESM1_ESM.pdf (132 kb)
Supplementary material 1 (PDF 133 kb)

References

  1. 1.
    Klin A, Lang J, Cicchetti DV, Volkmar FR (2000) Brief report: interrater reliability of clinical diagnosis and DSM-IV criteria for autistic disorder: results of the DSM-IV autism field trial. J Autism Dev Disord 30(2):163–167CrossRefPubMedGoogle Scholar
  2. 2.
    Volkmar F, Siegel M, Woodbury-Smith M, King B, McCracken J, State M, American Academy of C, Adolescent Psychiatry Committee on Quality I (2014) Practice parameter for the assessment and treatment of children and adolescents with autism spectrum disorder. J Am Acad Child Adolesc Psychiatry 53(2):237–257CrossRefPubMedGoogle Scholar
  3. 3.
    Autism: Recognition, Referral, Diagnosis and Management of Adults on the Autism Spectrum (2012). The British Psychological Society & The Royal College of Psychiatrists. Leicester, UKGoogle Scholar
  4. 4.
    Swedish Council on Health Technology Assessment (SBU) (2013) Autismspektrumtillstånd. Diagnostik och insatser, vårdens organisation och patientens delaktighet—En systematisk litteraturöversikt, vol 215. Swedish Council on Health Technology AssessmentGoogle Scholar
  5. 5.
    American Psychiatric Association (APA) (2013) Diagnostic and statistical manual of mental disorders DSM-5, 5th edn. American Psychiatric Association, ArlingtonGoogle Scholar
  6. 6.
    Lord C, Rutter M, DiLavore P, Risi S (1999) Autism Diagnostic Observation Schedule (ADOS). Western Psychological Publishing, Los AngelesGoogle Scholar
  7. 7.
    Lord C, Rutter M, DiLavore P, Risi S, Gotham K, Bishop S (2012) Autism Diagnostic Observation Schedule, Second Edition (ADOS-2) Manual (Part I). Western Psychological Services, TorranceGoogle Scholar
  8. 8.
    Rutter M, Le Couteur A, Lord C (2003) Autism Diagnostic Interview Revised (ADI-R). Western Psychological Services, Los AngelesGoogle Scholar
  9. 9.
    Molloy CA, Murray DS, Akers R, Mitchell T, Manning-Courtney P (2011) Use of the Autism Diagnostic Observation Schedule (ADOS) in a clinical setting. Autism 15(2):143–162CrossRefPubMedGoogle Scholar
  10. 10.
    Akshoomoff N, Corsello C, Schmidt H (2006) The role of the autism diagnostic observation schedule in the assessment of autism spectrum disorders in school and community settings. Calif School Psychol 11:7–19CrossRefPubMedPubMedCentralGoogle Scholar
  11. 11.
    Ashwood KL, Buitelaar J, Murphy D, Spooren W, Charman T (2015) European clinical network: autism spectrum disorder assessments and patient characterisation. Eur Child Adolesc Psychiatry 24(8):985–995CrossRefPubMedGoogle Scholar
  12. 12.
    Wolff JJ, Gu H, Gerig G, Elison JT, Styner M, Gouttard S, Botteron KN, Dager SR, Dawson G, Estes AM, Evans AC, Hazlett HC, Kostopoulos P, McKinstry RC, Paterson SJ, Schultz RT, Zwaigenbaum L, Piven J (2012) Differences in white matter fiber tract development present from 6 to 24 months in infants with autism. Am J Psychiatry 169(6):589–600CrossRefPubMedPubMedCentralGoogle Scholar
  13. 13.
    Bryson SE, Zwaigenbaum L, Brian J, Roberts W, Szatmari P, Rombough V, McDermott C (2007) A prospective case series of high-risk infants who developed autism. J Autism Dev Disord 37(1):12–24CrossRefPubMedGoogle Scholar
  14. 14.
    Ozonoff S, Young GS, Belding A, Hill M, Hill A, Hutman T, Johnson S, Miller M, Rogers SJ, Schwichtenberg AJ, Steinfeld M, Iosif AM (2014) The broader autism phenotype in infancy: when does it emerge? J Am Acad Child Adolesc Psychiatry 53(4):398–407CrossRefPubMedPubMedCentralGoogle Scholar
  15. 15.
    Bölte S, Poustka F (2004) Diagnostic Observation Scale for Autistic Disorders: initial results of reliability and validity. Z Kinder Jugendpsychiatr Psychother 32(1):45–50CrossRefPubMedGoogle Scholar
  16. 16.
    Zander E, Sturm H, Bölte S (2015) The added value of the combined use of the Autism Diagnostic Interview-Revised and the Autism Diagnostic Observation Schedule: Diagnostic validity in a clinical Swedish sample of toddlers and young preschoolers. Autism 19(2):187–199CrossRefPubMedGoogle Scholar
  17. 17.
    Gotham K, Risi S, Pickles A, Lord C (2007) The Autism Diagnostic Observation Schedule: revised algorithms for improved diagnostic validity. J Autism Dev Disord 37(4):613–627CrossRefPubMedGoogle Scholar
  18. 18.
    Gotham K, Risi S, Dawson G, Tager-Flusberg H, Joseph R, Carter A, Hepburn S, McMahon W, Rodier P, Hyman SL, Sigman M, Rogers S, Landa R, Spence MA, Osann K, Flodman P, Volkmar F, Hollander E, Buxbaum J, Pickles A, Lord C (2008) A replication of the Autism Diagnostic Observation Schedule (ADOS) revised algorithms. J Am Acad Child Adolesc Psychiatry 47(6):642–651CrossRefPubMedPubMedCentralGoogle Scholar
  19. 19.
    de Bildt A, Sytema S, van Lang ND, Minderaa RB, van Engeland H, de Jonge MV (2009) Evaluation of the ADOS revised algorithm: the applicability in 558 Dutch children and adolescents. J Autism Dev Disord 39(9):1350–1358CrossRefPubMedPubMedCentralGoogle Scholar
  20. 20.
    Bastiaansen JA, Meffert H, Hein S, Huizinga P, Ketelaars C, Pijnenborg M, Bartels A, Minderaa R, Keysers C, de Bildt A (2011) Diagnosing autism spectrum disorders in adults: the use of Autism Diagnostic Observation Schedule (ADOS) module 4. J Autism Dev Disord 41(9):1256–1266CrossRefPubMedGoogle Scholar
  21. 21.
    Gray KM, Tonge BJ, Sweeney DJ (2008) Using the Autism Diagnostic Interview-Revised and the Autism Diagnostic Observation Schedule with young children with developmental delay: evaluating diagnostic validity. J Autism Dev Disord 38(4):657–667CrossRefPubMedGoogle Scholar
  22. 22.
    Hus V, Lord C (2014) The autism diagnostic observation schedule, module 4: revised algorithm and standardized severity scores. J Autism Dev Disord 44(8):1996–2012CrossRefPubMedPubMedCentralGoogle Scholar
  23. 23.
    Kim SH, Lord C (2012) Combining information from multiple sources for the diagnosis of autism spectrum disorders for toddlers and young preschoolers from 12 to 47 months of age. J Child Psychol Psychiatry 53(2):143–151CrossRefPubMedGoogle Scholar
  24. 24.
    Klein-Tasman BP, Risi S, Lord CE (2007) Effect of language and task demands on the diagnostic effectiveness of the autism diagnostic observation schedule: the impact of module choice. J Autism Dev Disord 37(7):1224–1234CrossRefPubMedGoogle Scholar
  25. 25.
    Le Couteur A, Haden G, Hammal D, McConachie H (2008) Diagnosing autism spectrum disorders in pre-school children using two standardised assessment instruments: the ADI-R and the ADOS. J Autism Dev Disord 38(2):362–372CrossRefPubMedGoogle Scholar
  26. 26.
    Lord C, Risi S, Lambrecht L, Cook EH Jr, Leventhal BL, DiLavore PC, Pickles A, Rutter M (2000) The autism diagnostic observation schedule-generic: a standard measure of social and communication deficits associated with the spectrum of autism. J Autism Dev Disord 30(3):205–223CrossRefPubMedGoogle Scholar
  27. 27.
    Oosterling I, Roos S, de Bildt A, Rommelse N, de Jonge M, Visser J, Lappenschaar M, Swinkels S, van der Gaag RJ, Buitelaar J (2010) Improved diagnostic validity of the ADOS revised algorithms: a replication study in an independent sample. J Autism Dev Disord 40(6):689–703CrossRefPubMedPubMedCentralGoogle Scholar
  28. 28.
    Lord C, Rutter M, Goode S, Heemsbergen J, Jordan H, Mawhood L, Schopler E (1989) Autism diagnostic observation schedule: a standardized observation of communicative and social behavior. J Autism Dev Disord 19(2):185–212CrossRefPubMedGoogle Scholar
  29. 29.
    DiLavore PC, Lord C, Rutter M (1995) The pre-linguistic autism diagnostic observation schedule. J Autism Dev Disord 25(4):355–379CrossRefPubMedGoogle Scholar
  30. 30.
    de Bildt A, Sytema S, Meffert H, Bastiaansen JCJ (2015) The Autism Diagnostic observation schedule, module 4: application of the revised algorithms in an independent, well-defined, Dutch sample (n = 93). J Autism Dev Disord 30 August 2015. doi: 10.1007/s10803-015-2532-4
  31. 31.
    McClure I, Mackay T, Mamdani H, McCaughey R (2010) A comparison of a specialist autism spectrum disorder assessment team with local assessment teams. Autism 14(6):589–603CrossRefPubMedGoogle Scholar
  32. 32.
    Kim YS, Leventhal BL, Koh YJ, Fombonne E, Laska E, Lim EC, Cheon KA, Kim SJ, Kim YK, Lee H, Song DH, Grinker RR (2011) Prevalence of autism spectrum disorders in a total population sample. Am J Psychiatry 168(9):904–912CrossRefPubMedGoogle Scholar
  33. 33.
    Fombonne E, Marcin C, Bruno R, Tinoco CM, Marquez CD (2012) Screening for autism in Mexico. Autism Res 5(3):180–189CrossRefPubMedGoogle Scholar
  34. 34.
    Schendel DE, Diguiseppi C, Croen LA, Fallin MD, Reed PL, Schieve LA, Wiggins LD, Daniels J, Grether J, Levy SE, Miller L, Newschaffer C, Pinto-Martin J, Robinson C, Windham GC, Alexander A, Aylsworth AS, Bernal P, Bonner JD, Blaskey L, Bradley C, Collins J, Ferretti CJ, Farzadegan H, Giarelli E, Harvey M, Hepburn S, Herr M, Kaparich K, Landa R, Lee LC, Levenseller B, Meyerer S, Rahbar MH, Ratchford A, Reynolds A, Rosenberg S, Rusyniak J, Shapira SK, Smith K, Souders M, Thompson PA, Young L, Yeargin-Allsopp M (2012) The Study to Explore Early Development (SEED): a multisite epidemiologic study of autism by the Centers for Autism and Developmental Disabilities Research and Epidemiology (CADDRE) network. J Autism Dev Disord 42(10):2121–2140CrossRefPubMedPubMedCentralGoogle Scholar
  35. 35.
    Putka DJ, Le H, McCloy RA, Diaz T (2008) Ill-structured measurement designs in organizational research: implications for estimating interrater reliability. J Appl Psychol 93(5):959–981CrossRefPubMedGoogle Scholar
  36. 36.
    Hallgren KA (2012) Computing inter-rater reliability for observational data: an overview and tutorial. Tutor Quant Methods Psychol 8(1):23–34PubMedPubMedCentralGoogle Scholar
  37. 37.
    Conger AJ (1980) Integration and generalization of kappas for multiple raters. Psychol Bull 88(2):322–328CrossRefGoogle Scholar
  38. 38.
    Cicchetti DV, Sparrow SA (1981) Developing criteria for establishing interrater reliability of specific items: applications to assessment of adaptive behavior. Am J Ment Defic 86(2):127–137PubMedGoogle Scholar
  39. 39.
    Shavelson RJ, Webb NM (2006) Generalizability theory. Handbook of complementary methods in education research. Lawrence Erlbaum Associates Publishers, Mahwah, pp 309–322Google Scholar
  40. 40.
    Fleiss JL (1971) Measuring nominal scale agreement among many raters. Psychol Bull 76(5):378–382CrossRefGoogle Scholar
  41. 41.
    Cohen J (1960) A Coefficient of Agreement for Nominal Scales. Educ Psychol Meas 20(1):37–46CrossRefGoogle Scholar
  42. 42.
    de Vet HC, Terwee CB, Knol DL, Bouter LM (2006) When to use agreement versus reliability measures. J Clin Epidemiol 59(10):1033–1039CrossRefPubMedGoogle Scholar
  43. 43.
    Cicchetti DV (1994) Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychol Assess 6(4):284–290CrossRefGoogle Scholar
  44. 44.
    Cicchetti DV, Volkmar F, Klin A, Showalter D (1995) Diagnosing Autism using ICD-10 criteria: A comparison of neural networks and standard multivariate procedures. Child Neuropsychol 1(1):26–37CrossRefGoogle Scholar
  45. 45.
    Volkmar FR, Klin A, Siegel B, Szatmari P, Lord C, Campbell M, Freeman BJ, Cicchetti DV, Rutter M, Kline W et al (1994) Field trial for autistic disorder in DSM-IV. Am J Psychiatry 151(9):1361–1367CrossRefPubMedGoogle Scholar
  46. 46.
    Freedman R, Lewis DA, Michels R, Pine DS, Schultz SK, Tamminga CA, Gabbard GO, Gau SS, Javitt DC, Oquendo MA, Shrout PE, Vieta E, Yager J (2013) The initial field trials of DSM-5: new blooms and old thorns. Am J Psychiatry 170(1):1–5CrossRefPubMedGoogle Scholar
  47. 47.
    Charman T, Gotham K (2013) Measurement Issues: Screening and diagnostic instruments for autism spectrum disorders—lessons from research and practice. Child Adolesc Ment Health 18(1):52–63CrossRefPubMedGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2015

Authors and Affiliations

  • Eric Zander
    • 1
    • 2
    • 3
  • Charlotte Willfors
    • 1
    • 2
  • Steve Berggren
    • 1
    • 2
  • Nora Choque-Olsson
    • 1
    • 2
    • 4
  • Christina Coco
    • 1
    • 2
    • 5
  • Anna Elmund
    • 6
    • 7
  • Åsa Hedfors Moretti
    • 8
  • Anette Holm
    • 9
  • Ida Jifält
    • 10
    • 11
  • Renata Kosieradzki
    • 12
    • 13
  • Jenny Linder
    • 14
  • Viviann Nordin
    • 5
  • Karin Olafsdottir
    • 15
  • Lina Poltrago
    • 6
  • Sven Bölte
    • 1
    • 2
  1. 1.Pediatric Neuropsychiatry Unit, Department of Women’s and Children’s Health, Center of Neurodevelopmental Disorders (KIND)Karolinska InstitutetStockholmSweden
  2. 2.Child and Adolescent Psychiatry, Center for Psychiatry ResearchStockholm County CouncilStockholmSweden
  3. 3.Neurodevelopmental Psychiatry Unit South East, Child and Adolescent PsychiatryStockholm County CouncilStockholmSweden
  4. 4.BUP Södertälje, Child and Adolescent PsychiatryStockholm County CouncilSödertäljeSweden
  5. 5.Neuropediatric Unit, Sachs’ Children and Youth HospitalStockholm County CouncilStockholmSweden
  6. 6.PRIMA Child and Adolescent PsychiatryStockholmSweden
  7. 7.CitypsykologhusStockholmSweden
  8. 8.BUP Sollentuna, Child and Adolescent PsychiatryStockholm County CouncilSollentunaSweden
  9. 9.Astrid Lindgrens Children’s HospitalKarolinska University Hospital Solna, Stockholm County CouncilSolnaSweden
  10. 10.Astrid Lindgrens Children’s HospitalKarolinska University Hospital Huddinge, Stockholm County CouncilHuddingeSweden
  11. 11.Pupil Health UnitTiohundraSweden
  12. 12.BUP Malmö, Child and Adolescent PsychiatryRegion SkåneMalmöSweden
  13. 13.Pupil Health Unit, Resource Team for Learning DisabledCity of MalmöSweden
  14. 14.Södra Älvsborg Hospital (SÄS)Child and Adolescent PsychiatryBoråsSweden
  15. 15.BUP Lund, Child and Adolescent PsychiatryRegion SkåneLundSweden

Personalised recommendations