Identifying Autism with a Brief and Low-Cost Screening Instrument—OERA: Construct Validity, Invariance Testing, and Agreement Between Judges
Simple and low-cost observational-tools to detect symptoms of Autism Spectrum Disorder (ASD) are still necessary. The OERA is a new assessment tool to screen children eliciting observable behaviors with no substantial knowledge on ASD required. The sample was 99 children aged 3–10: 76 with ASD and 23 without ASD (11/23 had intellectual disability). The 13 remained items exhibited high interrater agreement and high reliability loaded onto a single latent trait. Such model showed excellent fit indices evaluated via confirmatory factor analysis and no item showed differential function in terms of age/sex/IQ. A cutoff of five points or higher resulted in the highest sensitivity (92.75) and specificity (90.91) percentages. OERA is a brief, stable, low-cost standardized observational-screening to identify ASD children.
KeywordsAutism Screening Construct validity Sensitivity Specificity Public health
This study was funded by the State of São Paulo Funding Agency-FAPESP under a special agreement with Maria Cecília Souto Vidigal Foundation [Grant No. 2012/51584-0], and by the National Research Council—CNPQ [Grant No. 401468/2010-0]. CSP and JJM are researchers from the National Research Council (CNPQ).
CSP conceived of the study, participated in its design, coordination, data collection, interpretation of data, and drafted the manuscript; GRC was one of the expert clinicians responsible for the ASD diagnosis, helped in the statistical analysis, and helped to draft the manuscript; DB was one of the expert clinicians responsible for the ASD diagnosis; participated in the study design, and helped to draft the manuscript; DB was one of the expert clinicians responsible for the ASD diagnosis, and helped to coordinate the data collection; ACM helped to coordinate the data collection, and participated in the interpretation of data; CAB was responsible for the ADI-R evaluations, and helped to draft the manuscript; JJM participated in the coordination of the study, and helped to draft the manuscript; HCM performed statistical analysis, and helped to draft the manuscript. All authors read and approved the final manuscript.
Compliance with Ethical Standards
Conflict of interest
All authors declare that they have no conflict of interest.
All procedures performed in this study involving human participants were in accordance with the ethical standards of the two institutional research committees and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.
Informed consent was obtained from all individual participants included in the study.
- Agency for Healthcare Research and Quality, A. E. H. C. P. (2014). Therapies for children with autism spectrum disorder: Behavioral interventions update. Rockville, MD.Google Scholar
- Bagaiolo, L., Mari, JdeJ., Bordini, D., Ribeiro, T. C., Martone, M. C. C., Brunoni, D., … Paula, C. S. (2017). Procedures and compliance of a videomodeling ABA intervention for Brazilian parents of children with Autism Spectrum Disorders. Autism: The International Journal of Research and Practice, 21(5), 603–610.CrossRefGoogle Scholar
- Baker, F. B., & Kim, S.-H. (2004). Item response theory: Parameter estimation techniques (2nd edn.). CRC Press: Boca Raton, FL.Google Scholar
- Becker, M. M., Wagner, M. B., Bosa, C., Schmidt, C., Longo, D., Papaleo, C., & Riesgo, R. S. (2012). Translation and validation of autism diagnostic interview-revised (ADI-R) for autism diagnosis in Brazil. Arquivos de Neuro-Psiquiatria, 70(3), 185–190. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/22392110.
- Bishop, S. L., Huerta, M., Gotham, K., Havdahl, A. K., Pickles, A., Duncan, A., … Lord, C. (2017). The autism symptom interview, school-age: A brief telephone interview to identify autism spectrum disorders in 5-to-12-year-old children. Autism Research, 10(1), 78–88. https://doi.org/10.1002/aur.1645.CrossRefPubMedGoogle Scholar
- Bollen, K. (1998). Structural equations with latent variables. New York: Wiley.Google Scholar
- Bölte, S., Westerwald, E., Holtmann, M., Freitag, C., & Poustka, F. (2011). Autistic traits and autism spectrum disorders: The clinical validity of two measures presuming a continuum of social communication skills. Journal of Autism and Developmental Disorders, 41(1), 66–72. https://doi.org/10.1007/s10803-010-1024-9.CrossRefPubMedGoogle Scholar
- Brown, T. (2015). Confirmatory factor analysis for applied research. New York: Guilford Publications.Google Scholar
- Cohen, J. (2013). Statistical power analysis for the behavioral sciences. New York: Academic Press.Google Scholar
- Cruz, M. (2005). WISC III: Escala de Inteligência Wechsler para crianças: Manual. Avaliação Psicológica, 4(2), 199–201.Google Scholar
- de Bildt, A., Mulder, E. J., Hoekstra, P. J., van Lang, N. D. J., Minderaa, R. B., & Hartman, C. A. (2009). Validity of the Children’s Social Behavior Questionnaire (CSBQ) in children with intellectual disability: Comparing the CSBQ with ADI-R, ADOS, and clinical DSM-IV-TR classification. Journal of Autism and Developmental Disorders, 39(10), 1464–1470. https://doi.org/10.1007/s10803-009-0764-x.CrossRefPubMedPubMedCentralGoogle Scholar
- de Macedo, E. C., Mecca, T. P., Valentini, F., Laros, J. A., de Lima, R. M. F., & Schwartzman, J. S. (2013). Utilizando o teste não verbal de inteligência SON-R 2 1/2—7 [a] para avaliar crianças com Transtornos do Espectro do Autismo. Revista Educação Especial, 26(47), 603–618. https://doi.org/10.5902/1984686X9779.CrossRefGoogle Scholar
- Du Toit, M. (2003). IRT from SSI: Bilog-MG, multilog, parscale, testfact. Lincolnwood: Scientific Software International.Google Scholar
- Feinstein, A. R., & Cicchetti, D. V. (1990). High agreement but low kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology, 43(6), 543–549. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/2189948.
- Feldman, H. M., Dale, P. S., Campbell, T. F., Colborn, D. K., Kurs-Lasky, M., Rockette, H. E., & Paradise, J. L. (2005). Concurrent and predictive validity of parent reports of child language at ages 2 and 3 years. Child Development, 76(4), 856–868. https://doi.org/10.1111/j.1467-8624.2005.00882.x.CrossRefPubMedPubMedCentralGoogle Scholar
- Figueiredo, V. (2002). WISC-III: Escala de Inteligência Weschler para crianças—adaptação brasileira da 3° edição. São Paulo: Casa do Psicólogo.Google Scholar
- Garcia, A. H. C., Viveiros, M. M., Schwartzman, J. S., & Brunoni, D. (2016). Transtornos do Espectro do Autismo: Avaliação e Comorbidades em Alunos de Barueri., São Paulo. Psicologia—Teoria E Prática, 18(1), 166–177. https://doi.org/10.15348/1980-6906/psicologia.v18n1p166-177.CrossRefGoogle Scholar
- Garfin, D. G., McCallon, D., & Cox, R. (1988). Validity and reliability of the Childhood Autism Rating Scale with autistic adolescents. Journal of Autism and Developmental Disorders, 18(3), 367–378. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/3170455.
- Gollenberg, A. L., Lynch, C. D., Jackson, L. W., McGuinness, B. M., & Msall, M. E. (2010). Concurrent validity of the parent-completed Ages and Stages Questionnaires, 2nd Ed. with the Bayley Scales of Infant Development II in a low-risk sample. Child: Care, Health and Development, 36(4), 485–490. https://doi.org/10.1111/j.1365-2214.2009.01041.x.Google Scholar
- Grodberg, D., Weinger, P. M., Halpern, D., Parides, M., Kolevzon, A., & Buxbaum, J. D. (2014). The autism mental status exam: Sensitivity and specificity using DSM-5 criteria for autism spectrum disorder in verbally fluent adults. Journal of Autism and Developmental Disorders, 44(3), 609–614. https://doi.org/10.1007/s10803-013-1917-5.CrossRefPubMedPubMedCentralGoogle Scholar
- Grodberg, D., Weinger, P. M., Kolevzon, A., Soorya, L., & Buxbaum, J. D. (2012). Brief report: The autism mental status examination: Development of a brief autism-focused exam. Journal of Autism and Developmental Disorders, 42(3), 455–459. https://doi.org/10.1007/s10803-011-1255-4.CrossRefPubMedGoogle Scholar
- Harstad, E. B., Fogler, J., Sideridis, G., Weas, S., Mauras, C., & Barbaresi, W. J. (2015). Comparing diagnostic outcomes of autism spectrum disorder using DSM-IV-TR and DSM-5 criteria. Journal of Autism and Developmental Disorders, 45(5), 1437–1450. https://doi.org/10.1007/s10803-014-2306-4.CrossRefPubMedGoogle Scholar
- Kim, M. A., & Lord, C. (2012). Combining information from multiple sources for the diagnosis of autism spectrum disorders for toddlers and young preschoolers from 12 to 47 months of age. Journal of Child Psychology and Psychiatry, 53(2), 143–151. https://doi.org/10.1111/j.1469-7610.2011.02458.x.CrossRefPubMedGoogle Scholar
- Krug, D. A., Arick, J., & Almond, P. (1980). Behavior checklist for identifying severely handicapped individuals with high levels of autistic behavior. Journal of Child Psychology and Psychiatry, and Allied Disciplines, 21(3), 221–229. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/7430288.
- Lai, M.-C., Lombardo, M. V., Auyeung, B., Chakrabarti, B., & Baron-Cohen, S. (2015). Sex/gender differences and autism: Setting the scene for future research. Journal of the American Academy of Child & Adolescent Psychiatry, 54(1), 11–24. https://doi.org/10.1016/j.jaac.2014.10.003.CrossRefGoogle Scholar
- Le Couteur, A., Rutter, M., Lord, C., Rios, P., Robertson, S., Holdgrafer, M., & McLennan, J. (1989). Autism diagnostic interview: A standardized investigator-based instrument. Journal of Autism and Developmental Disorders, 19(3), 363–387. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/2793783.
- Lord, C., Risi, S., Lambrecht, L., Cook, E. H., Leventhal, B. L., DiLavore, P. C., … Rutter, M. (2000). The autism diagnostic observation schedule-generic: A standard measure of social and communication deficits associated with the spectrum of autism. Journal of Autism and Developmental Disorders, 30(3), 205–223. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/11055457.
- Luyster, R., Gotham, K., Guthrie, W., Coffing, M., Petrak, R., Pierce, K., … Lord, C. (2009). The autism diagnostic observation schedule—Toddler module: A new module of a standardized diagnostic measure for autism spectrum disorders. Journal of Autism and Developmental Disorders, 39(9), 1305–1320. https://doi.org/10.1007/s10803-009-0746-z.CrossRefPubMedPubMedCentralGoogle Scholar
- McEwen, F. S., Stewart, C. S., Colvert, E., Woodhouse, E., Curran, S., Gillan, N., … Bolton, P. (2016). Diagnosing autism spectrum disorder in community settings using the Development and Well-Being Assessment: Validation in a UK population-based twin sample. Journal of Child Psychology and Psychiatry, 57(2), 161–170. https://doi.org/10.1111/jcpp.12447.CrossRefPubMedGoogle Scholar
- Muthen, B., & Asparouhov, T. (2004). IRT studies of many groups: The alignment method. Frontiers in Psychology, 11, 320–341.Google Scholar
- Muthén, B., Toit, D. S. H., & Spisic, D. (1997). Robust inference using weighted least squares and quadratic estimating equations in latent variable modeling with categorical and continuous outcomes. Psychometrika, 75, 1–45.Google Scholar
- Ozonoff, S., Goodlin-Jones, B. L., & Solomon, M. (2005). Evidence-based assessment of autism spectrum disorders in children and adolescents. Journal of Clinical Child and Adolescent Psychology: The Official Journal for the Society of Clinical Child and Adolescent Psychology, American Psychological Association, Division 53, 34(3), 523–540. https://doi.org/10.1207/s15374424jccp3403_8.CrossRefGoogle Scholar
- Schopler, E., Reichler, R. J., DeVellis, R. F., & Daly, K. (1980). Toward objective classification of childhood autism: Childhood Autism Rating Scale (CARS). Journal of Autism and Developmental Disorders, 10(1), 91–103. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/6927682.
- Shattuck, P. T., Durkin, M., Maenner, M., Newschaffer, C., Mandell, D. S., Wiggins, L., … Cuniff, C. (2009). Timing of identification among children with an autism spectrum disorder: Findings from a population-based surveillance study. Journal of the American Academy of Child and Adolescent Psychiatry, 48(5), 474–483. https://doi.org/10.1097/CHI.0b013e31819b3848.CrossRefPubMedPubMedCentralGoogle Scholar
- The Psychological Corporation. (1992). Weschler Individual Achievement Test. San Antonio, CA: The Psychological Corporation.Google Scholar
- Van Bourgondien, M. E., Marcus, L. M., & Schopler, E. (1992). Comparison of DSM-III-R and childhood autism rating scale diagnoses of autism. Journal of Autism and Developmental Disorders, 22(4), 493–506. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/1483973.
- Van de Schoot, R., Kluytmans, A., Tummers, L., Lugtig, P., Hox, J., & Muthen, B. (2013). Facing off with Scylla and Charybdis: A comparison of scalar, partial, and the novel possibility of approximate measurement invariance. Frontiers in Psychology, 23(4), 770. https://doi.org/10.3389/fpsyg.2013.00770.Google Scholar
- Yu, C.-Y. (2002). Evaluating cutoff criteria of model fit indices for latent variable models with binary and continuous outcomes. University of California, Los Angeles.Google Scholar
- Zwaigenbaum, L., Bauman, M. L., Choueiri, R., Kasari, C., Carter, A., Granpeesheh, D., … Natowicz, M. R. (2015b). Early intervention for children with autism spectrum disorder under 3 years of age: Recommendations for practice and research. Pediatrics, 136(Supplement), S60–S81. https://doi.org/10.1542/peds.2014-3667E.CrossRefPubMedGoogle Scholar
- Zwaigenbaum, L., Bryson, S., Lord, C., Rogers, S., Carter, A., Carver, L., … Yirmiya, N. (2009). Clinical assessment and management of toddlers with suspected autism spectrum disorder: Insights from studies of high-risk infants. Pediatrics, 123(5), 1383–1391. https://doi.org/10.1542/peds.2008-1606.CrossRefPubMedPubMedCentralGoogle Scholar