Skip to main content
Log in

Defining in Detail and Evaluating Reliability of DSM-5 Criteria for Autism Spectrum Disorder (ASD) Among Children

  • Original Paper
  • Published:
Journal of Autism and Developmental Disorders Aims and scope Submit manuscript

A Correction to this article was published on 29 January 2022

This article has been updated

Abstract

This paper describes a process to define a comprehensive list of exemplars for seven core Diagnostic and Statistical Manual (DSM) diagnostic criteria for autism spectrum disorder (ASD), and report on interrater reliability in applying these exemplars to determine ASD case classification. Clinicians completed an iterative process to map specific exemplars from the CDC Autism and Developmental Disabilities Monitoring (ADDM) Network criteria for ASD surveillance, DSM-5 text, and diagnostic assessments to each of the core DSM-5 ASD criteria. Clinicians applied the diagnostic exemplars to child behavioral descriptions in existing evaluation records to establish initial reliability standards and then for blinded clinician review in one site (phase 1) and for two ADDM Network surveillance years (phase 2). Interrater reliability for each of the DSM-5 diagnostic categories and overall ASD classification was high (defined as very good .60–.79 to excellent ≥ .80 Kappa values) across sex, race/ethnicity, and cognitive levels for both phases. Classification of DSM-5 ASD by mapping specific exemplars from evaluation records by a diverse group of clinician raters is feasible and reliable. This framework provides confidence in the consistency of prevalence classifications of ASD and may be further applied to improve consistency of ASD diagnoses in clinical settings.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1

Similar content being viewed by others

Change history

References

  • Aiello, R., Ruble, L., & Esler, A. (2017). National study of school psychologists’ use of evidence-based assessment in autism spectrum disorder. Journal of Applied School Psychology, 33(1), 67–88. https://doi.org/10.1080/15377903.2016.1236307

    Article  Google Scholar 

  • American Psychiatric Association. (1980). Diagnostic and statistical manual of mental disorders (3rd ed.). APA.

    Google Scholar 

  • American Psychiatric Association. (2000). Diagnostic and statistical manual of mental disorders (4th ed, text revision). APA.

  • American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed.). https://doi.org/10.1176/appi.books.978089425596

  • Arvidsson, O., Gillberg, C., Lichtenstein, P., & Lundström, S. (2018). Secular changes in the symptom level of clinically diagnosed autism. Journal of Child Psychology and Psychiatry, 59(7), 744–751. https://doi.org/10.1111/jcpp.12864

    Article  PubMed  Google Scholar 

  • Avchen, R. N., Wiggins, L. D., Devine, O., Braun, K. V. N., Rice, C., Hobson, N. C., Schendel, D., & Yeargin-Allsopp, M. (2011). Evaluation of a records-review surveillance system used to determine the prevalence of autism spectrum disorders. Journal of Autism and Developmental Disorders, 41(2), 227–236. https://doi.org/10.1007/s10803-010-1050-7

    Article  PubMed  Google Scholar 

  • Baio J., Wiggins L., Christensen D. L., Maenner, M. J., Daniels, J., Warren, Z., Kurzius-Spencer, M., Zahorodny, W., Rosenberg, C. R., White, T., Durkin, M. S., Imm, P., Nikolaou, L., Yeargin-Allsopp, M., Lee, L. C., Harrington, R., Lopez, M., Fitzgerald, R. T., Hewitt, A., … Dowling, N. F. (2018). Prevalence of autism spectrum disorder among children aged 8 years – Autism and Developmental Disabilities Monitoring Network, 11 sites, United States, 2014. Morbidity and Mortality Weekly Report, Surveilliance Summeries, 67(SS-6), 1–23.

  • Bertrand, J., Mars, A., Boyle, C., Bove, F., Yeargin-Allsopp, M., & Decoufle, P. (2001). Prevalence of autism in a United States population: The Brick Township, New Jersey, investigation. Pediatrics, 108(5), 1155–1161. https://doi.org/10.1542/peds.108.5.1155

    Article  PubMed  Google Scholar 

  • Bishop, S. L., Havdahl, K. A., Huerta, M., & Lord, C. (2016). Subdimensions of social-communication impairment in autism spectrum disorder. Journal of Child Psychology and Psychiatry, 57(8), 909–916. https://doi.org/10.1111/jcpp.12510

    Article  PubMed  Google Scholar 

  • Bishop, S. L., Huerta, M., Gotham, K., Havdahl, K. A., Pickles, A., Duncan, A., Bal, V. H., & Lord, C. (2017). The autism symptom interview, school-age: A brief telephone interview to identify autism spectrum disorders in 5-to-12-year-old children. Autism Research, 10(1), 78–88. https://doi.org/10.1002/aur.1645

    Article  PubMed  Google Scholar 

  • Bolte, E. E., & Diehl, J. J. (2013). Measurement tools and target symptoms/skills used to assess treatment response for individuals with autism spectrum disorder. Journal of Autism and Developmental Disorders, 43(11), 2491–2501. https://doi.org/10.1007/s10803-013-1798-7

    Article  PubMed  Google Scholar 

  • Bone, D., Goodwin, M. S., Black, M. P., Lee, C. C., Audhkhasi, K., & Narayanan, S. (2015). Applying machine learning to facilitate autism diagnostics: Pitfalls and promises. Journal of Autism and Developmental Disorders, 45(5), 1121–1136. https://doi.org/10.1007/s10803-014-2268-6

    Article  PubMed  PubMed Central  Google Scholar 

  • Carpenter, L. A., Boan, A. D., Wahlquist, A. E., Cohen, A., Charles, J., Jenner, W., & Bradley, C. C. (2016). Screening and direct assessment methodology to determine the prevalence of autism spectrum disorders. Annals of Epidemiology, 26(6), 395–400. https://doi.org/10.1016/j.annepidem.2016.04.003

    Article  PubMed  Google Scholar 

  • Centers for Disease Control and Prevention. (2007). Prevalence of autism spectrum disorders—Autism and developmental disabilities monitoring network, six sites, United States, 2000. Morbidity and Mortality Weekly Report, Surveill Summ, 56(SS-1), 1–11.

    Google Scholar 

  • Christensen, D. L., Maenner, M. J., Bilder, D., Constantino, J. N., Daniels, J., Durkin, M. S., ... & Shenouda, J. (2019). Prevalence and characteristics of autism spectrum disorder among children aged 4 years—Early Autism and Developmental Disabilities Monitoring Network, seven sites, United States, 2010, 2012, and 2014. Morbidity and Mortality Weekly Report, Surveillance Summaries, 68(2), 1–19

  • Clarke, D. E., Narrow, W. E., Regier, D. A., Kuramoto, S. J., Kupfer, D. J., Kuhl, E. A., Greiner, L., & Kraemer, H. C. (2013). DSM-5 field trials in the United States and Canada. I. Study design, sampling strategy, implementation, and analytic approaches. American Journal of Psychiatry, 170(1), 43–58. https://doi.org/10.1176/appi.ajp.2012.12070998

    Article  PubMed  Google Scholar 

  • Constantino, J. N., & Charman, T. (2016). Diagnosis of autism spectrum disorder: Reconciling the syndrome, its diverse origins, and variation in expression. The Lancet Neurology, 15(3), 279–291. https://doi.org/10.1016/S1474-4422(15)00151-9

    Article  PubMed  Google Scholar 

  • Constantino, J. N., & Todd, R. D. (2005). Intergenerational transmission of subthreshold autistic traits in the general population. Biological Psychiatry, 57(6), 655–660. https://doi.org/10.1016/j.biopsych.2004.12.014

    Article  PubMed  Google Scholar 

  • Craddock, N., & Mynors-Wallis, L. (2014). Psychiatric diagnosis: Impersonal, imperfect and important. The British Journal of Psychiatry, 204(2), 93–95. https://doi.org/10.1192/bjp.bp.113.133090

    Article  PubMed  Google Scholar 

  • Cuthbert, B. N., & Insel, T. R. (2013). Toward the future of psychiatric diagnosis: The seven pillars of RDoC. BMC Medicine, 11(1), 126. https://doi.org/10.1186/1741-7015-11-126

    Article  PubMed  PubMed Central  Google Scholar 

  • Daniels, A. M., Rosenberg, R. E., Law, J. K., Lord, C., Kaufmann, W. E., & Law, P. A. (2011). Stability of initial autism spectrum disorder diagnoses in community settings. Journal of Autism and Developmental Disorders, 41(1), 110–121. https://doi.org/10.1007/s10803-010-1031-x

    Article  PubMed  PubMed Central  Google Scholar 

  • Di Rezze, B., Rosenbaum, P., Zwaigenbaum, L., Hidecker, M. J., Stratford, P., Cousins, M., Camden, C., & Law, M. (2016). Developing a classification system of social communication functioning of preschool children with autism spectrum disorder. Developmental Medicine & Child Neurology, 58(9), 942–948. https://doi.org/10.1111/dmcn.13152

    Article  Google Scholar 

  • Durkin, M. S., Bilder, D. A., Pettygrove, S., & Zahorodny, W. (2015). The validity and usefulness of public health surveillance of autism spectrum disorder. Autism, 19(1), 118–119. https://doi.org/10.1177/1362361314548732

    Article  PubMed  Google Scholar 

  • Durkin, M. S., Maenner, M. J., Baio, J., Christensen, D., Daniels, J., Fitzgerald, R., Imm, P., Lee, L. C., Schieve, L. A., Braun, K. V. N., Wingate, M. S., & Yeargin-Allsopp, M. (2017). Autism spectrum disorder among US children (2002–2010): Socioeconomic, racial, and ethnic disparities. American Journal of Public Health, 107(11), 1818–1826. https://doi.org/10.2105/AJPH.2017.304032

    Article  PubMed  PubMed Central  Google Scholar 

  • Frazier, T. W., Youngstrom, E. A., Speer, L., Embacher, R., Law, P., Constantino, J., Findling, R. L., Hardan, A. Y., & Eng, C. (2012). Validation of proposed DSM-5 criteria for autism spectrum disorder. Journal of the American Academy of Child & Adolescent Psychiatry, 51(1), 28–40. https://doi.org/10.1016/j.jaac.2011.09.021

    Article  Google Scholar 

  • Gardner, L. M., Campbell, J. M., Keisling, B., & Murphy, L. (2018). Correlates of DSM-5 autism spectrum disorder levels of support ratings in a clinical sample. Journal of Autism and Developmental Disorders, 29, 1–11. https://doi.org/10.1007/s10803-018-3620-z

    Article  Google Scholar 

  • Hausman-Kedem, M., Kosofsky, B. E., Ross, G., Yohay, K., Forrest, E., Dennin, M. H., Patel, R., Bennett, K., Holahan, J. P., & Ward, M. J. (2018). Accuracy of reported community diagnosis of autism spectrum disorder. Journal of Psychopathology and Behavioral Assessment, 40(3), 367–375. https://doi.org/10.1007/s10862-018-9642-1

    Article  PubMed  PubMed Central  Google Scholar 

  • Hayes, J., McCabe, R., Ford, T., & Russell, G. (2020). Drawing a line in the sand: Affect and testimony in autism assessment teams in the UK. Sociology of Health & Illness, 42(4), 825–843. https://doi.org/10.1111/1467-9566.13063

    Article  Google Scholar 

  • Huerta, M., Bishop, S. L., Duncan, A., Hus, V., & Lord, C. (2012). Application of DSM-5 criteria for autism spectrum disorder to three samples of children with DSM-IV diagnoses of pervasive developmental disorders. American Journal of Psychiatry, 169(10), 1056–1064. https://doi.org/10.1176/appi.ajp.2012.12020276

    Article  PubMed  Google Scholar 

  • Imm, P., White, T., & Durkin, M. S. (2019). Assessment of racial and ethnic bias in autism spectrum disorder prevalence estimates from a US surveillance system. Autism, 23(8), 1927–1935. https://doi.org/10.1177/1362361319827510

    Article  PubMed  PubMed Central  Google Scholar 

  • Kanner, L. (1943). Autistic disturbances of affective contact. Nervous Child, 2(3), 217–250.

    Google Scholar 

  • Klin, A., Lang, J., Cicchetti, D. V., & Volkmar, F. R. (2000). Brief report: Interrater reliability of clinical diagnosis and DSM-IV criteria for autistic disorder: Results of the DSM-IV autism field trial. Journal of Autism and Developmental Disorders, 30(2), 163–167.

    Article  PubMed  Google Scholar 

  • Lord, C., Petkova, E., Hus, V., Gan, W., Lu, F., Martin, D. M., Ousley, O., Guy, L., Bernier, R., Gerdts, J., Algermissen, M., Whitaker, A., Sutcliffe, J. S., Warren, Z., Klin, A., Saulnier, C., Hanson, E., Hundley, R., Piggot, P., et al. (2012a). A multisite study of the clinical diagnosis of different autism spectrum disorders. Archives of General Psychiatry, 69(3), 306–313. https://doi.org/10.1001/archgenpsychiatry.2011.148

    Article  PubMed  Google Scholar 

  • Lord, C., Risi, S., Lambrecht, L., Cook, E. H., Jr., Leventhal, B. L., DiLavore, P. C., Pickles, A., & Rutter, M. (2000). The autism diagnostic observation schedule – generic: A standard measure of social and communication deficits associated with the spectrum of autism. Journal of Autism and Developmental Disorders, 30(3), 205–223.

    Article  PubMed  Google Scholar 

  • Lord, C., Rutter, M., DiLavore, P. C., Risi, S., Gotham, K., & Bishop, S. L. (2012). Autism Diagnostic Observation Schedule (2nd Edn.) (ADOS-2) Manual (Part I): Modules 1–4. Western Psychological Services.

  • Lord, C., Rutter, M., & Le Couteur, A. (1994). Autism diagnostic interview-revised: A revised version of a diagnostic inter-view for caregivers of individuals with possible pervasive developmental disorders. Journal of Autism and Developmental Disorders, 24(5), 659–685.

    Article  PubMed  Google Scholar 

  • Maenner, M. J., Shaw, K. A., Baio, J., Washington, A., Patrick, M., DiRienzo, M., Christensen, D. L., Wiggins, L. D., Pettygrove, S., Andrews, J. G., Lopez, M., Hudson, A., Baroud, T., Schwenk, Y., White, T., Rosenberg, C. R., Lee, L. C., Harrington, R. A., Huston, M., et al. (2020). Prevalence of autism spectrum disorder among children aged 8 years—autism and developmental disabilities monitoring network, 11 sites, United States, 2016. Morbidity and Mortality Weekly Report, Surveillance Summaries, 69(4), 1–12.

    Google Scholar 

  • Maenner, M. J., Rice, C. E., Arneson, C. L., Cunniff, C., Schieve, L. A., Carpenter, L. A., Van Naarden Braun, K., Kirby, R. S., Bakian, A. V., & Durkin, M. S. (2014). Potential impact of DSM-5 criteria on autism spectrum disorder prevalence estimates. JAMA Psychiatry, 71(3), 292–300. https://doi.org/10.1001/jamapsychiatry.2013.3893

    Article  PubMed  PubMed Central  Google Scholar 

  • McDonnell, C. G., Bradley, C. C., Kanne, S. M., Lajonchere, C., Warren, Z., & Carpenter, L. A. (2018). When are we sure? Predictors of clinician certainty in the diagnosis of autism spectrum disorder. Journal of Autism and Developmental Disorders, 49(4), 1391–1401. https://doi.org/10.1007/s10803-018-3831-3

    Article  Google Scholar 

  • McPartland, J. C., Reichow, B., & Volkmar, F. R. (2012). Sensitivity and specificity of proposed DSM-5 diagnostic criteria for autism spectrum disorder. Journal of the American Academy of Child & Adolescent Psychiatry, 51(4), 368–383.

    Article  Google Scholar 

  • Miller, J. S., Bilder, D., Farley, M., Coon, H., Pinborough-Zimmerman, J., Jenson, W., Rice, C. E., Fombonne, E., Pingree, C. B., Ritvo, E., Ritvo, R. A., & McMahon, W. M. (2013). Autism spectrum disorder reclassified: A second look at the 1980s Utah/UCLA autism epidemiologic study. Journal of Autism and Developmental Disorders, 43(1), 200–210. https://doi.org/10.1007/s10803-012-1566-0

    Article  PubMed  PubMed Central  Google Scholar 

  • Nicholas, J. S., Carpenter, L. A., King, L. B., Jenner, W., Wahlquist, A., Logan, S., & Charles, J. M. (2012). Completeness of case ascertainment for surveillance of autism spectrum disorders using the autism developmental disabilities monitoring network methodology. Disability and Health Journal, 5(3), 185–189.

    Article  PubMed  PubMed Central  Google Scholar 

  • Rice, C. E., Baio, J., Van Naarden Braun, K., Doernberg, N., Meaney, F. J., Kirby, R. S., & Network, A. D. D. M. (2007). A public health collaboration for the surveillance of autism spectrum disorders. Paediatric and Perinatal Epidemiology, 21(2), 179–190. https://doi.org/10.1111/j.1365-3016.2007.00801

    Article  PubMed  Google Scholar 

  • Rice, C. E., Rosanoff, M., Dawson, G., Durkin, M. S., Croen, L. A., Singer, A., & Yeargin-Allsopp, M. (2012). Evaluating changes in the prevalence of the autism spectrum disorders (ASDs). Public Health Reviews, 34(2), 1–22. https://doi.org/10.1007/BF03391685

    Article  PubMed  Google Scholar 

  • Rutter, M., Le Couteur, A., & Lord, C. (2003). Autism diagnostic interview-revised. Los Angeles, CA: Western Psychological Services.

  • Rutherford, M., McKenzie, K., McClure, I., Forsyth, K., O’Hare, A., McCartney, D., & Finlayson, I. (2016). A national study to investigate the clinical use of standardized instruments in autism spectrum disorder assessment of children and adults in Scotland. Research in Autism Spectrum Disorders, 29, 93–100. https://doi.org/10.1016/j.rasd.2016.05.003

    Article  Google Scholar 

  • Shulman, C., Esler, A., Morrier, M. J., & Rice, C. E. (2020a). Diagnosis of autism spectrum disorder across the lifespan. Child and Adolescent Psychiatric Clinics, 29(2), 253–273. https://doi.org/10.1016/j.chc.2020.01.001

    Article  Google Scholar 

  • Shulman, C., Rice, C. E., Morrier, M. J., & Esler, A. (2020b). The role of diagnostic instruments in dual and differential diagnosis in autism spectrum disorder across the lifespan. Child and Adolescent Psychiatric Clinics, 29(2), 275–299. https://doi.org/10.1016/j.chc.2020.01.002

    Article  Google Scholar 

  • Thurm, A., Farmer, C., Salzman, E., Lord, C., & Bishop, S. (2019). State of the field: Differentiating intellectual disability from autism spectrum disorder. Frontiers in Psychiatry, 10, 526.

  • Uljarević, M., Arnott, B., Carrington, S. J., Meins, E., Fernyhough, C., McConachie, H., Le Couteur, A., & Leekam, S. R. (2017). Development of restricted and repetitive behaviors from 15 to 77 months: Stability of two distinct subtypes?. Developmental Psychology, 53(10), 1859–1868. https://psycnet.apa.org/doi/https://doi.org/10.1037/dev000032.

  • Van Daalen, E., Kemner, C., Dietz, C., Swinkels, S. H., Buitelaar, J. K., & Van Engeland, H. (2009). Inter-rater reliability and stability of diagnoses of autism spectrum disorder in children identified through screening at a very young age. European Child & Adolescent Psychiatry, 18(11), 663–674. https://doi.org/10.1007/s00787-009-0025-8

    Article  Google Scholar 

  • Volkmar, F. R., & McPartland, J. C. (2014). From Kanner to DSM-5: Autism as an evolving diagnostic concept. Annual Review of Clinical Psychology, 10, 193–212. https://doi.org/10.1146/annurev-clinpsy-032813-153710

    Article  PubMed  Google Scholar 

  • Wiggins, L. D., Durkin, M., Esler, A., Lee, L. C., Zahorodny, W., Rice, C., Yeargin-Allsopp, M., Dowling, N. F., Hall-Lande, J., Morrier, M. J., Christensen, D., Shenouda, J., & Baio, J. (2020). Disparities in documented diagnoses of autism spectrum disorder based on demographic, individual, and service factors. Autism Research, 13(3), 464–473. https://doi.org/10.1002/aur.2255

    Article  PubMed  Google Scholar 

  • Wiggins, L. D., Reynolds, A., Rice, C. E., Moody, E. J., Bernal, P., Blaskey, L., Rosenberg, S., & A., Lee, L. C., & Levy, S. E. (2015). Using standardized diagnostic instruments to classify children with autism in the study to explore early development. Journal of Autism and Developmental Disorders, 45(5), 1271–1280. https://doi.org/10.1007/s10803-014-2287-3

    Article  PubMed  PubMed Central  Google Scholar 

  • Yeargin-Allsopp, M., Rice, C., Karapurkar, T., Doernberg, N., Boyle, C., & Murphy, C. (2003). Prevalence of autism in a US metropolitan area. Journal of the American Medical Association, 289(1), 49–55. https://doi.org/10.1001/jama.289.1.49

    Article  PubMed  Google Scholar 

  • Zheng, S., Kaat, A., Farmer, C., Kanne, S., Georgiades, S., Lord, C., Esler, A., & Bishop, S. L. (2020). Extracting latent subdimensions of social communication: A cross-measure factor analysis. Journal of the American Academy of Child & Adolescent Psychiatry, 60, 768–782.

    Article  Google Scholar 

Download references

Funding

Funding for the SC SUCCESS project was provided by Autism Speaks (Grant No. 7793 and 8408), Foundation for the National Institutes of Health (Grant No. UL1TR001450), and the Centers for Disease Control and Prevention (Grant No. DD10-1002). Funding for co-author (Thurm), Intramural Research Program of the NIMH (1ZICMH002961). The ADDM Network is supported by the Department of Health and Human Services, Centers for Disease Control and Prevention, National Center on Birth Defects and Developmental Disabilities.

Author information

Authors and Affiliations

Authors

Contributions

CER and LAC led the first phase of the DSM-5 ASD criteria mapping and all co-authors contributed to subsequent refinements. LAC led the SC SUCCESS Project. AB analyzed the SC data and JB the ADDM Network data. The first draft of the manuscript was written by CER. All authors commented drafts of the manuscript and approved the final manuscript.

Corresponding author

Correspondence to C. E. Rice.

Ethics declarations

Conflict of interest

C. Lord acknowledges royalties from diagnostic instruments from Western Psychological Services. The findings and conclusions in this report are those of the authors and do not necessarily represent the views of CDC, NIH, or Autism Speaks.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (PDF 862 kb)

Supplementary file2 (PDF 557 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rice, C.E., Carpenter, L.A., Morrier, M.J. et al. Defining in Detail and Evaluating Reliability of DSM-5 Criteria for Autism Spectrum Disorder (ASD) Among Children. J Autism Dev Disord 52, 5308–5320 (2022). https://doi.org/10.1007/s10803-021-05377-y

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10803-021-05377-y

Keywords

Navigation