Abstract
This paper describes a process to define a comprehensive list of exemplars for seven core Diagnostic and Statistical Manual (DSM) diagnostic criteria for autism spectrum disorder (ASD), and report on interrater reliability in applying these exemplars to determine ASD case classification. Clinicians completed an iterative process to map specific exemplars from the CDC Autism and Developmental Disabilities Monitoring (ADDM) Network criteria for ASD surveillance, DSM-5 text, and diagnostic assessments to each of the core DSM-5 ASD criteria. Clinicians applied the diagnostic exemplars to child behavioral descriptions in existing evaluation records to establish initial reliability standards and then for blinded clinician review in one site (phase 1) and for two ADDM Network surveillance years (phase 2). Interrater reliability for each of the DSM-5 diagnostic categories and overall ASD classification was high (defined as very good .60–.79 to excellent ≥ .80 Kappa values) across sex, race/ethnicity, and cognitive levels for both phases. Classification of DSM-5 ASD by mapping specific exemplars from evaluation records by a diverse group of clinician raters is feasible and reliable. This framework provides confidence in the consistency of prevalence classifications of ASD and may be further applied to improve consistency of ASD diagnoses in clinical settings.
Similar content being viewed by others
Change history
29 January 2022
A Correction to this paper has been published: https://doi.org/10.1007/s10803-022-05452-y
References
Aiello, R., Ruble, L., & Esler, A. (2017). National study of school psychologists’ use of evidence-based assessment in autism spectrum disorder. Journal of Applied School Psychology, 33(1), 67–88. https://doi.org/10.1080/15377903.2016.1236307
American Psychiatric Association. (1980). Diagnostic and statistical manual of mental disorders (3rd ed.). APA.
American Psychiatric Association. (2000). Diagnostic and statistical manual of mental disorders (4th ed, text revision). APA.
American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed.). https://doi.org/10.1176/appi.books.978089425596
Arvidsson, O., Gillberg, C., Lichtenstein, P., & Lundström, S. (2018). Secular changes in the symptom level of clinically diagnosed autism. Journal of Child Psychology and Psychiatry, 59(7), 744–751. https://doi.org/10.1111/jcpp.12864
Avchen, R. N., Wiggins, L. D., Devine, O., Braun, K. V. N., Rice, C., Hobson, N. C., Schendel, D., & Yeargin-Allsopp, M. (2011). Evaluation of a records-review surveillance system used to determine the prevalence of autism spectrum disorders. Journal of Autism and Developmental Disorders, 41(2), 227–236. https://doi.org/10.1007/s10803-010-1050-7
Baio J., Wiggins L., Christensen D. L., Maenner, M. J., Daniels, J., Warren, Z., Kurzius-Spencer, M., Zahorodny, W., Rosenberg, C. R., White, T., Durkin, M. S., Imm, P., Nikolaou, L., Yeargin-Allsopp, M., Lee, L. C., Harrington, R., Lopez, M., Fitzgerald, R. T., Hewitt, A., … Dowling, N. F. (2018). Prevalence of autism spectrum disorder among children aged 8 years – Autism and Developmental Disabilities Monitoring Network, 11 sites, United States, 2014. Morbidity and Mortality Weekly Report, Surveilliance Summeries, 67(SS-6), 1–23.
Bertrand, J., Mars, A., Boyle, C., Bove, F., Yeargin-Allsopp, M., & Decoufle, P. (2001). Prevalence of autism in a United States population: The Brick Township, New Jersey, investigation. Pediatrics, 108(5), 1155–1161. https://doi.org/10.1542/peds.108.5.1155
Bishop, S. L., Havdahl, K. A., Huerta, M., & Lord, C. (2016). Subdimensions of social-communication impairment in autism spectrum disorder. Journal of Child Psychology and Psychiatry, 57(8), 909–916. https://doi.org/10.1111/jcpp.12510
Bishop, S. L., Huerta, M., Gotham, K., Havdahl, K. A., Pickles, A., Duncan, A., Bal, V. H., & Lord, C. (2017). The autism symptom interview, school-age: A brief telephone interview to identify autism spectrum disorders in 5-to-12-year-old children. Autism Research, 10(1), 78–88. https://doi.org/10.1002/aur.1645
Bolte, E. E., & Diehl, J. J. (2013). Measurement tools and target symptoms/skills used to assess treatment response for individuals with autism spectrum disorder. Journal of Autism and Developmental Disorders, 43(11), 2491–2501. https://doi.org/10.1007/s10803-013-1798-7
Bone, D., Goodwin, M. S., Black, M. P., Lee, C. C., Audhkhasi, K., & Narayanan, S. (2015). Applying machine learning to facilitate autism diagnostics: Pitfalls and promises. Journal of Autism and Developmental Disorders, 45(5), 1121–1136. https://doi.org/10.1007/s10803-014-2268-6
Carpenter, L. A., Boan, A. D., Wahlquist, A. E., Cohen, A., Charles, J., Jenner, W., & Bradley, C. C. (2016). Screening and direct assessment methodology to determine the prevalence of autism spectrum disorders. Annals of Epidemiology, 26(6), 395–400. https://doi.org/10.1016/j.annepidem.2016.04.003
Centers for Disease Control and Prevention. (2007). Prevalence of autism spectrum disorders—Autism and developmental disabilities monitoring network, six sites, United States, 2000. Morbidity and Mortality Weekly Report, Surveill Summ, 56(SS-1), 1–11.
Christensen, D. L., Maenner, M. J., Bilder, D., Constantino, J. N., Daniels, J., Durkin, M. S., ... & Shenouda, J. (2019). Prevalence and characteristics of autism spectrum disorder among children aged 4 years—Early Autism and Developmental Disabilities Monitoring Network, seven sites, United States, 2010, 2012, and 2014. Morbidity and Mortality Weekly Report, Surveillance Summaries, 68(2), 1–19
Clarke, D. E., Narrow, W. E., Regier, D. A., Kuramoto, S. J., Kupfer, D. J., Kuhl, E. A., Greiner, L., & Kraemer, H. C. (2013). DSM-5 field trials in the United States and Canada. I. Study design, sampling strategy, implementation, and analytic approaches. American Journal of Psychiatry, 170(1), 43–58. https://doi.org/10.1176/appi.ajp.2012.12070998
Constantino, J. N., & Charman, T. (2016). Diagnosis of autism spectrum disorder: Reconciling the syndrome, its diverse origins, and variation in expression. The Lancet Neurology, 15(3), 279–291. https://doi.org/10.1016/S1474-4422(15)00151-9
Constantino, J. N., & Todd, R. D. (2005). Intergenerational transmission of subthreshold autistic traits in the general population. Biological Psychiatry, 57(6), 655–660. https://doi.org/10.1016/j.biopsych.2004.12.014
Craddock, N., & Mynors-Wallis, L. (2014). Psychiatric diagnosis: Impersonal, imperfect and important. The British Journal of Psychiatry, 204(2), 93–95. https://doi.org/10.1192/bjp.bp.113.133090
Cuthbert, B. N., & Insel, T. R. (2013). Toward the future of psychiatric diagnosis: The seven pillars of RDoC. BMC Medicine, 11(1), 126. https://doi.org/10.1186/1741-7015-11-126
Daniels, A. M., Rosenberg, R. E., Law, J. K., Lord, C., Kaufmann, W. E., & Law, P. A. (2011). Stability of initial autism spectrum disorder diagnoses in community settings. Journal of Autism and Developmental Disorders, 41(1), 110–121. https://doi.org/10.1007/s10803-010-1031-x
Di Rezze, B., Rosenbaum, P., Zwaigenbaum, L., Hidecker, M. J., Stratford, P., Cousins, M., Camden, C., & Law, M. (2016). Developing a classification system of social communication functioning of preschool children with autism spectrum disorder. Developmental Medicine & Child Neurology, 58(9), 942–948. https://doi.org/10.1111/dmcn.13152
Durkin, M. S., Bilder, D. A., Pettygrove, S., & Zahorodny, W. (2015). The validity and usefulness of public health surveillance of autism spectrum disorder. Autism, 19(1), 118–119. https://doi.org/10.1177/1362361314548732
Durkin, M. S., Maenner, M. J., Baio, J., Christensen, D., Daniels, J., Fitzgerald, R., Imm, P., Lee, L. C., Schieve, L. A., Braun, K. V. N., Wingate, M. S., & Yeargin-Allsopp, M. (2017). Autism spectrum disorder among US children (2002–2010): Socioeconomic, racial, and ethnic disparities. American Journal of Public Health, 107(11), 1818–1826. https://doi.org/10.2105/AJPH.2017.304032
Frazier, T. W., Youngstrom, E. A., Speer, L., Embacher, R., Law, P., Constantino, J., Findling, R. L., Hardan, A. Y., & Eng, C. (2012). Validation of proposed DSM-5 criteria for autism spectrum disorder. Journal of the American Academy of Child & Adolescent Psychiatry, 51(1), 28–40. https://doi.org/10.1016/j.jaac.2011.09.021
Gardner, L. M., Campbell, J. M., Keisling, B., & Murphy, L. (2018). Correlates of DSM-5 autism spectrum disorder levels of support ratings in a clinical sample. Journal of Autism and Developmental Disorders, 29, 1–11. https://doi.org/10.1007/s10803-018-3620-z
Hausman-Kedem, M., Kosofsky, B. E., Ross, G., Yohay, K., Forrest, E., Dennin, M. H., Patel, R., Bennett, K., Holahan, J. P., & Ward, M. J. (2018). Accuracy of reported community diagnosis of autism spectrum disorder. Journal of Psychopathology and Behavioral Assessment, 40(3), 367–375. https://doi.org/10.1007/s10862-018-9642-1
Hayes, J., McCabe, R., Ford, T., & Russell, G. (2020). Drawing a line in the sand: Affect and testimony in autism assessment teams in the UK. Sociology of Health & Illness, 42(4), 825–843. https://doi.org/10.1111/1467-9566.13063
Huerta, M., Bishop, S. L., Duncan, A., Hus, V., & Lord, C. (2012). Application of DSM-5 criteria for autism spectrum disorder to three samples of children with DSM-IV diagnoses of pervasive developmental disorders. American Journal of Psychiatry, 169(10), 1056–1064. https://doi.org/10.1176/appi.ajp.2012.12020276
Imm, P., White, T., & Durkin, M. S. (2019). Assessment of racial and ethnic bias in autism spectrum disorder prevalence estimates from a US surveillance system. Autism, 23(8), 1927–1935. https://doi.org/10.1177/1362361319827510
Kanner, L. (1943). Autistic disturbances of affective contact. Nervous Child, 2(3), 217–250.
Klin, A., Lang, J., Cicchetti, D. V., & Volkmar, F. R. (2000). Brief report: Interrater reliability of clinical diagnosis and DSM-IV criteria for autistic disorder: Results of the DSM-IV autism field trial. Journal of Autism and Developmental Disorders, 30(2), 163–167.
Lord, C., Petkova, E., Hus, V., Gan, W., Lu, F., Martin, D. M., Ousley, O., Guy, L., Bernier, R., Gerdts, J., Algermissen, M., Whitaker, A., Sutcliffe, J. S., Warren, Z., Klin, A., Saulnier, C., Hanson, E., Hundley, R., Piggot, P., et al. (2012a). A multisite study of the clinical diagnosis of different autism spectrum disorders. Archives of General Psychiatry, 69(3), 306–313. https://doi.org/10.1001/archgenpsychiatry.2011.148
Lord, C., Risi, S., Lambrecht, L., Cook, E. H., Jr., Leventhal, B. L., DiLavore, P. C., Pickles, A., & Rutter, M. (2000). The autism diagnostic observation schedule – generic: A standard measure of social and communication deficits associated with the spectrum of autism. Journal of Autism and Developmental Disorders, 30(3), 205–223.
Lord, C., Rutter, M., DiLavore, P. C., Risi, S., Gotham, K., & Bishop, S. L. (2012). Autism Diagnostic Observation Schedule (2nd Edn.) (ADOS-2) Manual (Part I): Modules 1–4. Western Psychological Services.
Lord, C., Rutter, M., & Le Couteur, A. (1994). Autism diagnostic interview-revised: A revised version of a diagnostic inter-view for caregivers of individuals with possible pervasive developmental disorders. Journal of Autism and Developmental Disorders, 24(5), 659–685.
Maenner, M. J., Shaw, K. A., Baio, J., Washington, A., Patrick, M., DiRienzo, M., Christensen, D. L., Wiggins, L. D., Pettygrove, S., Andrews, J. G., Lopez, M., Hudson, A., Baroud, T., Schwenk, Y., White, T., Rosenberg, C. R., Lee, L. C., Harrington, R. A., Huston, M., et al. (2020). Prevalence of autism spectrum disorder among children aged 8 years—autism and developmental disabilities monitoring network, 11 sites, United States, 2016. Morbidity and Mortality Weekly Report, Surveillance Summaries, 69(4), 1–12.
Maenner, M. J., Rice, C. E., Arneson, C. L., Cunniff, C., Schieve, L. A., Carpenter, L. A., Van Naarden Braun, K., Kirby, R. S., Bakian, A. V., & Durkin, M. S. (2014). Potential impact of DSM-5 criteria on autism spectrum disorder prevalence estimates. JAMA Psychiatry, 71(3), 292–300. https://doi.org/10.1001/jamapsychiatry.2013.3893
McDonnell, C. G., Bradley, C. C., Kanne, S. M., Lajonchere, C., Warren, Z., & Carpenter, L. A. (2018). When are we sure? Predictors of clinician certainty in the diagnosis of autism spectrum disorder. Journal of Autism and Developmental Disorders, 49(4), 1391–1401. https://doi.org/10.1007/s10803-018-3831-3
McPartland, J. C., Reichow, B., & Volkmar, F. R. (2012). Sensitivity and specificity of proposed DSM-5 diagnostic criteria for autism spectrum disorder. Journal of the American Academy of Child & Adolescent Psychiatry, 51(4), 368–383.
Miller, J. S., Bilder, D., Farley, M., Coon, H., Pinborough-Zimmerman, J., Jenson, W., Rice, C. E., Fombonne, E., Pingree, C. B., Ritvo, E., Ritvo, R. A., & McMahon, W. M. (2013). Autism spectrum disorder reclassified: A second look at the 1980s Utah/UCLA autism epidemiologic study. Journal of Autism and Developmental Disorders, 43(1), 200–210. https://doi.org/10.1007/s10803-012-1566-0
Nicholas, J. S., Carpenter, L. A., King, L. B., Jenner, W., Wahlquist, A., Logan, S., & Charles, J. M. (2012). Completeness of case ascertainment for surveillance of autism spectrum disorders using the autism developmental disabilities monitoring network methodology. Disability and Health Journal, 5(3), 185–189.
Rice, C. E., Baio, J., Van Naarden Braun, K., Doernberg, N., Meaney, F. J., Kirby, R. S., & Network, A. D. D. M. (2007). A public health collaboration for the surveillance of autism spectrum disorders. Paediatric and Perinatal Epidemiology, 21(2), 179–190. https://doi.org/10.1111/j.1365-3016.2007.00801
Rice, C. E., Rosanoff, M., Dawson, G., Durkin, M. S., Croen, L. A., Singer, A., & Yeargin-Allsopp, M. (2012). Evaluating changes in the prevalence of the autism spectrum disorders (ASDs). Public Health Reviews, 34(2), 1–22. https://doi.org/10.1007/BF03391685
Rutter, M., Le Couteur, A., & Lord, C. (2003). Autism diagnostic interview-revised. Los Angeles, CA: Western Psychological Services.
Rutherford, M., McKenzie, K., McClure, I., Forsyth, K., O’Hare, A., McCartney, D., & Finlayson, I. (2016). A national study to investigate the clinical use of standardized instruments in autism spectrum disorder assessment of children and adults in Scotland. Research in Autism Spectrum Disorders, 29, 93–100. https://doi.org/10.1016/j.rasd.2016.05.003
Shulman, C., Esler, A., Morrier, M. J., & Rice, C. E. (2020a). Diagnosis of autism spectrum disorder across the lifespan. Child and Adolescent Psychiatric Clinics, 29(2), 253–273. https://doi.org/10.1016/j.chc.2020.01.001
Shulman, C., Rice, C. E., Morrier, M. J., & Esler, A. (2020b). The role of diagnostic instruments in dual and differential diagnosis in autism spectrum disorder across the lifespan. Child and Adolescent Psychiatric Clinics, 29(2), 275–299. https://doi.org/10.1016/j.chc.2020.01.002
Thurm, A., Farmer, C., Salzman, E., Lord, C., & Bishop, S. (2019). State of the field: Differentiating intellectual disability from autism spectrum disorder. Frontiers in Psychiatry, 10, 526.
Uljarević, M., Arnott, B., Carrington, S. J., Meins, E., Fernyhough, C., McConachie, H., Le Couteur, A., & Leekam, S. R. (2017). Development of restricted and repetitive behaviors from 15 to 77 months: Stability of two distinct subtypes?. Developmental Psychology, 53(10), 1859–1868. https://psycnet.apa.org/doi/https://doi.org/10.1037/dev000032.
Van Daalen, E., Kemner, C., Dietz, C., Swinkels, S. H., Buitelaar, J. K., & Van Engeland, H. (2009). Inter-rater reliability and stability of diagnoses of autism spectrum disorder in children identified through screening at a very young age. European Child & Adolescent Psychiatry, 18(11), 663–674. https://doi.org/10.1007/s00787-009-0025-8
Volkmar, F. R., & McPartland, J. C. (2014). From Kanner to DSM-5: Autism as an evolving diagnostic concept. Annual Review of Clinical Psychology, 10, 193–212. https://doi.org/10.1146/annurev-clinpsy-032813-153710
Wiggins, L. D., Durkin, M., Esler, A., Lee, L. C., Zahorodny, W., Rice, C., Yeargin-Allsopp, M., Dowling, N. F., Hall-Lande, J., Morrier, M. J., Christensen, D., Shenouda, J., & Baio, J. (2020). Disparities in documented diagnoses of autism spectrum disorder based on demographic, individual, and service factors. Autism Research, 13(3), 464–473. https://doi.org/10.1002/aur.2255
Wiggins, L. D., Reynolds, A., Rice, C. E., Moody, E. J., Bernal, P., Blaskey, L., Rosenberg, S., & A., Lee, L. C., & Levy, S. E. (2015). Using standardized diagnostic instruments to classify children with autism in the study to explore early development. Journal of Autism and Developmental Disorders, 45(5), 1271–1280. https://doi.org/10.1007/s10803-014-2287-3
Yeargin-Allsopp, M., Rice, C., Karapurkar, T., Doernberg, N., Boyle, C., & Murphy, C. (2003). Prevalence of autism in a US metropolitan area. Journal of the American Medical Association, 289(1), 49–55. https://doi.org/10.1001/jama.289.1.49
Zheng, S., Kaat, A., Farmer, C., Kanne, S., Georgiades, S., Lord, C., Esler, A., & Bishop, S. L. (2020). Extracting latent subdimensions of social communication: A cross-measure factor analysis. Journal of the American Academy of Child & Adolescent Psychiatry, 60, 768–782.
Funding
Funding for the SC SUCCESS project was provided by Autism Speaks (Grant No. 7793 and 8408), Foundation for the National Institutes of Health (Grant No. UL1TR001450), and the Centers for Disease Control and Prevention (Grant No. DD10-1002). Funding for co-author (Thurm), Intramural Research Program of the NIMH (1ZICMH002961). The ADDM Network is supported by the Department of Health and Human Services, Centers for Disease Control and Prevention, National Center on Birth Defects and Developmental Disabilities.
Author information
Authors and Affiliations
Contributions
CER and LAC led the first phase of the DSM-5 ASD criteria mapping and all co-authors contributed to subsequent refinements. LAC led the SC SUCCESS Project. AB analyzed the SC data and JB the ADDM Network data. The first draft of the manuscript was written by CER. All authors commented drafts of the manuscript and approved the final manuscript.
Corresponding author
Ethics declarations
Conflict of interest
C. Lord acknowledges royalties from diagnostic instruments from Western Psychological Services. The findings and conclusions in this report are those of the authors and do not necessarily represent the views of CDC, NIH, or Autism Speaks.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Rice, C.E., Carpenter, L.A., Morrier, M.J. et al. Defining in Detail and Evaluating Reliability of DSM-5 Criteria for Autism Spectrum Disorder (ASD) Among Children. J Autism Dev Disord 52, 5308–5320 (2022). https://doi.org/10.1007/s10803-021-05377-y
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10803-021-05377-y