Skip to main content

Reliability of the Commonly Used and Newly-Developed Autism Measures

Abstract

Purpose

The aim of the present study was to compare scale and conditional reliability derived from item response theory analyses among the most commonly used, as well as several newly developed, observation, interview, and parent-report autism instruments.

Methods

When available, data sets were combined to facilitate large sample evaluation. Scale reliability (internal consistency, average corrected item-total correlations, and model reliability) and conditional reliability estimates were computed for total scores and for measure subscales.

Results

Generally good to excellent scale reliability was observed for total scores for all measures, scale reliability was weaker for RRB subscales of the ADOS and ADI-R, reflecting the relatively small number of items for these measures. For diagnostic measures, conditional reliability tended to be very good (> 0.80) in the regions of the latent trait where ASD and non-ASD developmental disability cases would be differentiated. For parent-report scales, conditional reliability of total scores tended to be excellent (> 0.90) across very wide ranges of autism symptom levels, with a few notable exceptions.

Conclusions

These findings support the use of all of the clinical observation, interview, and parent-report autism symptom measures examined, but also suggest specific limitations that warrant consideration when choosing measures for specific clinical or research applications.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3

References

  • Alexander, L. M., Escalera, J., Ai, L., Andreotti, C., Febre, K., Mangone, A., Vega Potler, N., Langer, N., Alexander, A., Kovacs, M., Litke, S., O’Hagan, B., Andersen, J., Bronstein, B., Bui, A., Bushey, M., Butler, H., Castagna, V., Camacho, N., Chan, E., Citera, D., Clucas, J., Cohen, S., Dufek, S., Eaves, M., & Gregory, C. (2017). The healthy Brain Network Biobank: An open resource for transdiagnostic research in pediatric mental health and learning disorders. Cold Spring Harbor Laboratory. https://doi.org/10.1101/149369.

    Article  Google Scholar 

  • Baron-Cohen, S., Wheelwright, S., Skinner, R., Martin, J., & Clubley, E. (2001). The autism-spectrum quotient (AQ): Evidence from Asperger syndrome/high-functioning autism, males and females, scientists and mathematicians. Journal of Autism and Developmental Disorders, 31(1), 5–17. https://doi.org/10.1023/a:1005653411471.

    Article  PubMed  Google Scholar 

  • Barrett, S. L., Uljarevic, M., Baker, E. K., Richdale, A. L., Jones, C. R., & Leekam, S. R. (2015). The adult repetitive Behaviours Questionnaire-2 (RBQ-2A): A self-report measure of restricted and repetitive behaviours. Journal of Autism and Developmental Disorders, 45(11), 3680–3692. https://doi.org/10.1007/s10803-015-2514-6.

    Article  PubMed  PubMed Central  Google Scholar 

  • Boateng, G. O., Neilands, T. B., Frongillo, E. A., Melgar-Quinonez, H. R., & Young, S. L. (2018). Best Practices for developing and validating Scales for Health, Social, and behavioral research: A primer. Front Public Health, 6, 149. https://doi.org/10.3389/fpubh.2018.00149.

    Article  PubMed  PubMed Central  Google Scholar 

  • Bryson, S. E., Zwaigenbaum, L., McDermott, C., Rombough, V., & Brian, J. (2008). The Autism Observation Scale for Infants: Scale development and reliability data. Journal of Autism and Developmental Disorders, 38(4), 731–738. https://doi.org/10.1007/s10803-007-0440-y.

    Article  PubMed  Google Scholar 

  • Carrington, S., Leekam, S., Kent, R., Maljaars, J., Gould, J., Wing, L., Le Couteur, A., Van Berckelaer-Onnes, I., & Noens, I. (2015). Signposting for diagnosis of Autism Spectrum disorder using the diagnostic interview for Social and Communication Disorders (DISCO). Research in Autism Spectrum Disorders, 9, 45–52. https://doi.org/10.1016/j.rasd.2014.10.003.

    Article  Google Scholar 

  • Carrington, S. J., Barrett, S. L., Sivagamasundari, U., Fretwell, C., Noens, I., Maljaars, J., & Leekam, S. R. (2019). Describing the Profile of Diagnostic features in autistic adults using an Abbreviated Version of the diagnostic interview for Social and Communication Disorders (DISCO-Abbreviated). Journal of Autism and Developmental Disorders, 49(12), 5036–5046. https://doi.org/10.1007/s10803-019-04214-7.

    Article  PubMed  PubMed Central  Google Scholar 

  • Carrington, S. J., Kent, R. G., Maljaars, J., Le Couteur, A., Gould, J., Wing, L., Noens, I., Van Berckelaer-Onnes, I., & Leekam, S. R. (2014). DSM-5 Autism Spectrum Disorder: In search of essential behaviours for diagnosis. Research in Autism Spectrum Disorders, 8(6), 701–715. https://doi.org/10.1016/j.rasd.2014.03.017.

    Article  Google Scholar 

  • Charman, T., & Gotham, K. (2013). Measurement issues: Screening and diagnostic instruments for autism spectrum disorders - lessons from research and practice. Child Adolesc Ment Health, 18(1), 52–63. https://doi.org/10.1111/j.1475-3588.2012.00664.x.

    Article  PubMed  Google Scholar 

  • Constantino, J. N., & Gruber, C. P. (2012). The social responsiveness scale manual, second edition (SRS-2). Western Psychological Services.

  • Corona, L. L., Wagner, L., Wade, J., Weitlauf, A. S., Hine, J., Nicholson, A., Stone, C., Vehorn, A., & Warren, Z. (2021). Toward Novel Tools for Autism Identification: Fusing computational and clinical expertise. Journal of Autism and Developmental Disorders, 51(11), 4003–4012. https://doi.org/10.1007/s10803-020-04857-x.

    Article  PubMed  PubMed Central  Google Scholar 

  • Embretson, S. E., & Reise, S. P. (2000). Item response theory for psychologists. Lawrence Erlbaum Associates.

  • Evans, D. W., Uljarevic, M., Lusk, L. G., Loth, E., & Frazier, T. (2017). Development of two dimensional measures of restricted and repetitive behavior in parents and children. Journal of the American Academy of Child and Adolescent Psychiatry, 56(1), 51–58. https://doi.org/10.1016/j.jaac.2016.10.014.

    Article  PubMed  Google Scholar 

  • Fischbach, G. D., & Lord, C. (2010). The Simons Simplex Collection: A resource for identification of autism genetic risk factors. Neuron, 68(2), 192–195. https://doi.org/10.1016/j.neuron.2010.10.006.

    Article  PubMed  Google Scholar 

  • Frazier, T. W., Dimitropoulos, A., Abbeduto, L., Armstrong-Brine, M., Kralovic, S., Shih, A., Hardan, A. Y., Youngstrom, E. A., & Uljarevic, M. (2023). The Autism Symptom Dimensions Questionnaire: Development and psychometric evaluation of a new, open-source measure of autism symptomatology. Developmental Medicine and Child Neurology. https://doi.org/10.1111/dmcn.15497. & Quadrant Biosciences - As You Are

  • Frazier, T. W., & Hardan, A. Y. (2017). Equivalence of symptom dimensions in females and males with autism. Autism, 21(6), 749–759. https://doi.org/10.1177/1362361316660066.

    Article  PubMed  Google Scholar 

  • Frazier, T. W., Youngstrom, E. A., Embacher, R., Hardan, A. Y., Constantino, J. N., Law, P., Findling, R. L., & Eng, C. (2014). Demographic and clinical correlates of autism symptom domains and autism spectrum diagnosis. Autism, 18(5), 571–582. https://doi.org/10.1177/1362361313481506.

    Article  PubMed  Google Scholar 

  • Geschwind, D. H., Sowinski, J., Lord, C., Iversen, P., Shestack, J., Jones, P., Ducat, L., & Spence, S. (2001). The autism genetic resource exchange: A resource for the study of autism and related neuropsychiatric conditions. American Journal of Human Genetics, 69, 463–466.

    Article  PubMed  PubMed Central  Google Scholar 

  • Grzadzinski, R., Carr, T., Colombi, C., McGuire, K., Dufek, S., Pickles, A., & Lord, C. (2016). Measuring changes in Social Communication Behaviors: Preliminary Development of the brief Observation of Social Communication Change (BOSCC). Journal of Autism and Developmental Disorders, 46(7), 2464–2479. https://doi.org/10.1007/s10803-016-2782-9.

    Article  PubMed  Google Scholar 

  • Hall, D., Huerta, M. F., McAuliffe, M. J., & Farber, G. K. (2012). Sharing heterogeneous data: The national database for autism research. Neuroinformatics, 10(4), 331–339. https://doi.org/10.1007/s12021-012-9151-4.

    Article  PubMed  PubMed Central  Google Scholar 

  • Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory. Sage.

  • Hayes, A. F., & Coutts, J. J. (2020). Use omega rather than Cronbach’s alpha for estimating reliability. But… Communication Methods and Measures, 14(1), 1–24.

    Article  Google Scholar 

  • Huerta, M., Bishop, S. L., Duncan, A., Hus, V., & Lord, C. (2012). Application of DSM-5 criteria for autism spectrum disorder to three samples of children with DSM-IV diagnoses of pervasive developmental disorders. American Journal of Psychiatry, 169(10), 1056–1064. https://doi.org/10.1176/appi.ajp.2012.12020276.

    Article  PubMed  Google Scholar 

  • IBM Corp (2021). IBM SPSS Statistics for Windows. In (Version 28.0) IBM Corp.

  • Janvier, D., Choi, Y. B., Klein, C., Lord, C., & Kim, S. H. (2022). Brief report: Examining test-retest reliability of the Autism Diagnostic Observation schedule (ADOS-2) calibrated severity scores (CSS). Journal of Autism and Developmental Disorders, 52(3), 1388–1394. https://doi.org/10.1007/s10803-021-04952-7.

    Article  PubMed  Google Scholar 

  • Kanne, S. M., Mazurek, M. O., Sikora, D., Bellando, J., Branum-Martin, L., Handen, B., Katz, T., Freedman, B., Powell, M. P., & Warren, Z. (2014). The Autism Impact measure (AIM): Initial development of a new tool for treatment outcome measurement. Journal of Autism and Developmental Disorders, 44(1), 168–179. https://doi.org/10.1007/s10803-013-1862-3.

    Article  PubMed  Google Scholar 

  • Kent, R. G., Carrington, S. J., Le Couteur, A., Gould, J., Wing, L., Maljaars, J., Noens, I., van Berckelaer-Onnes, I., & Leekam, S. R. (2013). Diagnosing autism spectrum disorder: Who will get a DSM-5 diagnosis? Journal of Child Psychology and Psychiatry and Allied Disciplines, 54(11), 1242–1250. https://doi.org/10.1111/jcpp.12085.

    Article  PubMed  Google Scholar 

  • Kitzerow, J., Teufel, K., Wilker, C., & Freitag, C. M. (2016). Using the brief observation of social communication change (BOSCC) to measure autism-specific development. Autism Research, 9(9), 940–950. https://doi.org/10.1002/aur.1588.

    Article  PubMed  Google Scholar 

  • Lam, K. S., & Aman, M. G. (2007). The repetitive behavior Scale-Revised: Independent validation in individuals with autism spectrum disorders. Journal of Autism and Developmental Disorders, 37(5), 855–866. https://doi.org/10.1007/s10803-006-0213-z.

    Article  PubMed  Google Scholar 

  • Leekam, S., Libby, S., Wing, L., Gould, J., & Gillberg, C. (2000). Comparison of ICD-10 and Gillberg’s criteria for Asperger syndrome. Autism, 4(1), 11–28.

    Article  Google Scholar 

  • Leekam, S., Tandos, J., McConachie, H., Meins, E., Parkinson, K., Wright, C., Turner, M., Arnott, B., Vittorini, L., & Le Couteur, A. (2007). Repetitive behaviours in typically developing 2-year-olds. Journal of Child Psychology and Psychiatry, 48(11), 1131–1138. https://doi.org/10.1111/j.1469-7610.2007.01778.x.

    Article  PubMed  Google Scholar 

  • Leekam, S. R., Libby, S. J., Wing, L., Gould, J., & Taylor, C. (2002a). The diagnostic interview for Social and Communication Disorders: Algorithms for ICD-10 childhood autism and Wing and Gould autistic spectrum disorder. Journal of Child Psychology and Psychiatry and Allied Disciplines, 43(3), 327–342. https://doi.org/10.1111/1469-7610.00024.

    Article  PubMed  Google Scholar 

  • Leekam, S. R., Libby, S. J., Wing, L., Gould, J., & Taylor, C. (2002b). The diagnostic interview for Social and Communication Disorders: Algorithms for ICD-10 childhood autism and Wing and Gould autistic spectrum disorder. Journal of Child Psychology and Psychiatry and Allied Disciplines, 43(3), 327–342. http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?cmd=Retrieve &db=PubMed&dopt=Citation&list_uids=11944875.

    Article  PubMed  Google Scholar 

  • Leekam, S. R., Nieto, C., Libby, S. J., Wing, L., & Gould, J. (2007). Describing the sensory abnormalities of children and adults with autism. Journal of Autism and Developmental Disorders, 37(5), 894–910. https://doi.org/10.1007/s10803-006-0218-7.

    Article  PubMed  Google Scholar 

  • Leon, A. C., Marzuk, P. M., & Portera, L. (1995). More reliable outcome measures can reduce sample size requirements. Archives of General Psychiatry, 52(10), 867–871. https://doi.org/10.1001/archpsyc.1995.03950220077014.

    Article  PubMed  Google Scholar 

  • Lidstone, J., Uljarevic, M., Sullivan, J. P., Rodgers, J., McConachie, H., Freeston, M., Le Couteur, A., Prior, M., & Leekam, S. (2014). Relations among restricted and repetitive behviors, anxiety and sensory features in children with autism spectrum disorders. Research in Autism Spectrum Disorders, 8(2), 82–92.

    Article  Google Scholar 

  • Lord, C., Brugha, T. S., Charman, T., Cusack, J., Dumas, G., Frazier, T., Jones, E. J. H., Jones, R. M., Pickles, A., State, M. W., Taylor, J. L., & Veenstra-VanderWeele, J. (2020). Autism spectrum disorder. Nat Rev Dis Primers, 6(1), 5. https://doi.org/10.1038/s41572-019-0138-4.

    Article  PubMed  PubMed Central  Google Scholar 

  • Lord, C., Pickles, A., McLennan, J., Rutter, M., Bregman, J., Folstein, S., Fombonne, E., Leboyer, M., & Minshew, N. (1997). Diagnosing autism: Analyses of data from the Autism Diagnostic interview. Journal of Autism and Developmental Disorders, 27(5), 501–517.

    Article  PubMed  Google Scholar 

  • Lord, C., Rutter, M., DiLavore, P. C., & Risi, S. (2002). Autism Diagnostic Observation schedule: ADOS manual. Western Psychological Services.

  • Lord, C., Rutter, M., DiLavore, P. C., Risi, S., Gotham, K., & Bishop, S. L. (2012). Autism Diagnostic Observation schedule, Second Edition (ADOS-2) manual (part 1): Modules 1–4. Western Psychological Services.

  • Lord, C., Rutter, M., & LeCouteur, A. (1994). ADI-Revised: A revised version of a diagnostic interview for caregivers of individuals with possible pervasive developmental disorders. Journal of Autism and Developmental Disorders, 24, 569–685.

    Article  Google Scholar 

  • Maljaars, J., Noens, I., Scholte, E., & van Berckelaer-Onnes, I. (2012). Evaluation of the criterion and convergent validity of the diagnostic interview for Social and Communication Disorders in young and low-functioning children. Autism, 16(5), 487–497. https://doi.org/10.1177/1362361311402857.

    Article  PubMed  Google Scholar 

  • Murray, A. L., McKenzie, K., Kuenssberg, R., & Booth, T. (2017). Do the Autism Spectrum Quotient (AQ) and Autism Spectrum Quotient Short Form (AQ-S) primarily reflect General ASD Traits or specific ASD traits? A Bi-Factor Analysis. Assessment, 24(4), 444–457. https://doi.org/10.1177/1073191115611230.

    Article  PubMed  Google Scholar 

  • Muthén, L. K., & Muthén, B. O. (1998–2017). Mplus User’s Guide. Eighth Edition Muthén & Muthén.

  • Nguyen, P. H., Ocansey, M. E., Miller, M., Le, D. T. K., Schmidt, R. J., & Prado, E. L. (2019). The reliability and validity of the social responsiveness scale to measure autism symptomology in vietnamese children. Autism Research, 12(11), 1706–1718. https://doi.org/10.1002/aur.2179.

    Article  PubMed  PubMed Central  Google Scholar 

  • Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). McGraw-Hill, Inc.

  • Nygren, G., Hagberg, B., Billstedt, E., Skoglund, A., Gillberg, C., & Johansson, M. (2009). The swedish version of the diagnostic interview for Social and Communication Disorders (DISCO-10). Psychometric properties. Journal of Autism and Developmental Disorders, 39(5), 730–741. https://doi.org/10.1007/s10803-008-0678-z.

    Article  PubMed  Google Scholar 

  • Phillips, J. M., Uljarevic, M., Schuck, R. K., Schapp, S., Solomon, E. M., Salzman, E., Allerhand, L., Libove, R. A., Frazier, T. W., & Hardan, A. Y. (2019). Development of the Stanford Social Dimensions Scale: Initial validation in autism spectrum disorder and in neurotypicals. Mol Autism, 10, 48. https://doi.org/10.1186/s13229-019-0298-9.

    Article  PubMed  PubMed Central  Google Scholar 

  • Reise, S. P., Widaman, K. F., & Pugh, R. H. (1993). Confirmatory factor analysis and item response theory: Two approaches for exploring measurement invariance. Psychological Bulletin, 114(3), 552–566.

    Article  PubMed  Google Scholar 

  • Revelle, W., & Condon, D. M. (2019). Reliability from alpha to omega: A tutorial. Psychological Assessment, 31(12), 1395–1411. https://doi.org/10.1037/pas0000754.

    Article  PubMed  Google Scholar 

  • Robins, D. L., Casagrande, K., Barton, M., Chen, C. M., Dumont-Mathieu, T., & Fein, D. (2014). Validation of the modified checklist for Autism in toddlers, revised with follow-up (M-CHAT-R/F). Pediatrics, 133(1), 37–45. https://doi.org/10.1542/peds.2013-1813.

    Article  PubMed  PubMed Central  Google Scholar 

  • Rutter, M., Bailey, A., & Lord, C. (2003). The Social Communication Questionnaire Manual. Western Psychological Services.

  • Sanchez, M. J., & Constantino, J. N. (2020). Expediting clinician assessment in the diagnosis of autism spectrum disorder. Developmental Medicine and Child Neurology, 62(7), 806–812. https://doi.org/10.1111/dmcn.14530.

    Article  PubMed  PubMed Central  Google Scholar 

  • Schopler, E., Van Bourgondien, M., Wellman, G., & Love, S. (2010). Childhood Autism Rating Scale – 2nd Edition. Western Psychological Services.

  • Simms, L. J., Zelazny, K., Williams, T. F., & Bernstein, L. (2019). Does the number of Response Options Matter? Psychometric perspectives using personality Questionnaire Data. Psychological Assessment, 31(4), 557–566. https://doi.org/10.1037/pas0000648.

    Article  PubMed  Google Scholar 

  • Stone, W. L., McMahon, C. R., & Henderson, L. M. (2008). Use of the Screening Tool for Autism in Two-Year-Olds (STAT) for children under 24 months: An exploratory study. Autism, 12(5), 557–573. https://doi.org/10.1177/1362361308096403.

    Article  PubMed  Google Scholar 

  • Streiner, D. L., & Norman, G. R. (1995). Health Measurement Scales: A practical guide to their development and use (2nd ed.). Oxford University Press.

  • Streiner, D. L., & Norman, G. R. (2008). Health measurement scales: A practical guide to their use (4th ed.). Oxford University Press. https://doi.org/10.1093/acprof:oso/9780199231881.001.0001.

  • Taylor, B. P., Liu, J., Mowrey, W., Eule, E., Bolognani, F., & Hollander, E. (2022). Development, administration, reliability, and validity in child and adult Autism Spectrum Disorder (ASD). Journal of Psychiatric Research, 147, 142–147. https://doi.org/10.1016/j.jpsychires.2021.12.055. The Montefiore-Einstein Rigidity Scale-Revised (MERS-R).

  • Thissen, D. (2000). Reliability and measurement precision (2nd ed.). Lawrence Erlbaum Associates.

  • Uljarevic, M., Arnott, B., Carrington, S. J., Meins, E., Fernyhough, C., McConachie, H., Le Couteur, A., & Leekam, S. R. (2017). Development of restricted and repetitive behaviors from 15 to 77 months: Stability of two distinct subtypes? Developmental Psychology, 53(10), 1859–1868. https://doi.org/10.1037/dev0000324.

    Article  PubMed  Google Scholar 

  • Uljarević, M., Frazier, T. W., Jo, B., Scahill, L., Youngstrom, E. A., Spackman, E., Phillips, J. M., Billingham, W., & Hardan, A. Y. (2022). Dimensional Assessment of Restricted and repetitive behaviors: Development and preliminary validation of a new measure. Journal of American Academy of Child and Adolescent Psychiatry. https://doi.org/10.1016/j.jaac.2022.07.863.

    Article  Google Scholar 

  • Uljarevic, M., Frazier, T. W., Phillips, J. M., Jo, B., Littlefield, S., & Hardan, A. Y. (2019). Mapping the Research Domain Criteria social processes constructs to the Social Responsiveness Scale. Journal of the American Academy of Child and Adolescent Psychiatry, 58(10S), S311. https://doi.org/10.1016/j.jaac.2019.07.938.

    Article  Google Scholar 

  • Uljarevic, M., Frazier, T. W., Phillips, J. M., Jo, B., Littlefield, S., & Hardan, A. Y. (2020). Quantifying Research Domain Criteria Social Communication Subconstructs using the Social Communication Questionnaire in Youth. Journal Of Clinical Child And Adolescent Psychology : The Official Journal For The Society Of Clinical Child And Adolescent Psychology, American Psychological Association, Division 53, 1–11. https://doi.org/10.1080/15374416.2019.1669156.

  • Uljarevic, M., Jo, B., Frazier, T. W., Scahill, L., Youngstrom, E. A., & Hardan, A. Y. (2021). Using the big data approach to clarify the structure of restricted and repetitive behaviors across the most commonly used autism spectrum disorder measures. Mol Autism, 12(1), 39. https://doi.org/10.1186/s13229-021-00419-9.

    Article  PubMed  PubMed Central  Google Scholar 

  • Uljarević, M., Jo, B., Frazier, T. W., Scahill, L., Youngstrom, E. A., & Hardan, A. Y. (2021). Using the big data approach to clarify the structure of restricted repetitive behaviors across the most commonly used autism spectrum disorder measures. Molecular Autism.

  • Wetherby, A. M., & Prizant, B. M. (2003). Communication and symbolic behavior scales (CSBS), Normed Edition. Paul A. Brookes Publishing Co.

  • Wing, L., Leekam, S. R., Libby, S. J., Gould, J., & Larcombe, M. (2002). The diagnostic interview for Social and Communication Disorders: background, inter-rater reliability and clinical use. Journal of Child Psychology and Psychiatry and Allied Disciplines, 43(3), 307–325. https://doi.org/10.1111/1469-7610.00023.

    Article  PubMed  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Thomas W. Frazier.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Frazier, T.W., Whitehouse, A., Leekam, S. et al. Reliability of the Commonly Used and Newly-Developed Autism Measures. J Autism Dev Disord (2023). https://doi.org/10.1007/s10803-023-05967-y

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10803-023-05967-y

Keywords