Journal of Autism and Developmental Disorders

, Volume 48, Issue 5, pp 1780–1791 | Cite as

Identifying Autism with a Brief and Low-Cost Screening Instrument—OERA: Construct Validity, Invariance Testing, and Agreement Between Judges

  • Cristiane S. Paula
  • Graccielle Rodrigues Cunha
  • Daniela Bordini
  • Decio Brunoni
  • Ana Claudia Moya
  • Cleonice Alves Bosa
  • Jair J. Mari
  • Hugo Cogo-Moreira
Original Paper

Abstract

Simple and low-cost observational-tools to detect symptoms of Autism Spectrum Disorder (ASD) are still necessary. The OERA is a new assessment tool to screen children eliciting observable behaviors with no substantial knowledge on ASD required. The sample was 99 children aged 3–10: 76 with ASD and 23 without ASD (11/23 had intellectual disability). The 13 remained items exhibited high interrater agreement and high reliability loaded onto a single latent trait. Such model showed excellent fit indices evaluated via confirmatory factor analysis and no item showed differential function in terms of age/sex/IQ. A cutoff of five points or higher resulted in the highest sensitivity (92.75) and specificity (90.91) percentages. OERA is a brief, stable, low-cost standardized observational-screening to identify ASD children.

Keywords

Autism Screening Construct validity Sensitivity Specificity Public health 

Notes

Acknowledgments

This study was funded by the State of São Paulo Funding Agency-FAPESP under a special agreement with Maria Cecília Souto Vidigal Foundation [Grant No. 2012/51584-0], and by the National Research Council—CNPQ [Grant No. 401468/2010-0]. CSP and JJM are researchers from the National Research Council (CNPQ).

Author Contributions

CSP conceived of the study, participated in its design, coordination, data collection, interpretation of data, and drafted the manuscript; GRC was one of the expert clinicians responsible for the ASD diagnosis, helped in the statistical analysis, and helped to draft the manuscript; DB was one of the expert clinicians responsible for the ASD diagnosis; participated in the study design, and helped to draft the manuscript; DB was one of the expert clinicians responsible for the ASD diagnosis, and helped to coordinate the data collection; ACM helped to coordinate the data collection, and participated in the interpretation of data; CAB was responsible for the ADI-R evaluations, and helped to draft the manuscript; JJM participated in the coordination of the study, and helped to draft the manuscript; HCM performed statistical analysis, and helped to draft the manuscript. All authors read and approved the final manuscript.

Compliance with Ethical Standards

Conflict of interest

All authors declare that they have no conflict of interest.

Ethical approval

All procedures performed in this study involving human participants were in accordance with the ethical standards of the two institutional research committees and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed consent

Informed consent was obtained from all individual participants included in the study.

References

  1. Abrahams, B. S., & Geschwind, D. H. (2008). Advances in autism genetics: On the threshold of a new neurobiology. Nature Reviews. Genetics, 9(5), 341–355.  https://doi.org/10.1038/nrg2346.CrossRefPubMedPubMedCentralGoogle Scholar
  2. Agency for Healthcare Research and Quality, A. E. H. C. P. (2014). Therapies for children with autism spectrum disorder: Behavioral interventions update. Rockville, MD.Google Scholar
  3. Alckmin-Carvalho, F., Teixeira, M. C. T. V., Brunoni, D., Strauss, V. G., & Paula, C. S. (2014). Identification of early signs of autism according to a structured observational protocol: A follow-up study. Psico, 45(4), 502.  https://doi.org/10.15448/1980-8623.2014.4.15873.CrossRefGoogle Scholar
  4. American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders, 5th edn. Arlington, VA: American Psychiatric Publishing.CrossRefGoogle Scholar
  5. Ashwood, K. L., Buitelaar, J., Murphy, D., Spooren, W., & Charman, T. (2015). European clinical network: Autism spectrum disorder assessments and patient characterisation. European Child & Adolescent Psychiatry, 24(8), 985–995.  https://doi.org/10.1007/s00787-014-0648-2.CrossRefGoogle Scholar
  6. Bagaiolo, L., Mari, JdeJ., Bordini, D., Ribeiro, T. C., Martone, M. C. C., Brunoni, D., … Paula, C. S. (2017). Procedures and compliance of a videomodeling ABA intervention for Brazilian parents of children with Autism Spectrum Disorders. Autism: The International Journal of Research and Practice, 21(5), 603–610.CrossRefGoogle Scholar
  7. Baker, F. B., & Kim, S.-H. (2004). Item response theory: Parameter estimation techniques (2nd edn.). CRC Press: Boca Raton, FL.Google Scholar
  8. Becker, M. M., Wagner, M. B., Bosa, C., Schmidt, C., Longo, D., Papaleo, C., & Riesgo, R. S. (2012). Translation and validation of autism diagnostic interview-revised (ADI-R) for autism diagnosis in Brazil. Arquivos de Neuro-Psiquiatria, 70(3), 185–190. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/22392110.
  9. Bishop, S. L., Huerta, M., Gotham, K., Havdahl, A. K., Pickles, A., Duncan, A., … Lord, C. (2017). The autism symptom interview, school-age: A brief telephone interview to identify autism spectrum disorders in 5-to-12-year-old children. Autism Research, 10(1), 78–88.  https://doi.org/10.1002/aur.1645.CrossRefPubMedGoogle Scholar
  10. Bollen, K. (1998). Structural equations with latent variables. New York: Wiley.Google Scholar
  11. Bölte, S., Westerwald, E., Holtmann, M., Freitag, C., & Poustka, F. (2011). Autistic traits and autism spectrum disorders: The clinical validity of two measures presuming a continuum of social communication skills. Journal of Autism and Developmental Disorders, 41(1), 66–72.  https://doi.org/10.1007/s10803-010-1024-9.CrossRefPubMedGoogle Scholar
  12. Brown, T. (2015). Confirmatory factor analysis for applied research. New York: Guilford Publications.Google Scholar
  13. Charman, T. G. K., & Gotham, K. (2013). Measurment Issues: Screening and diagnostic instruments for autism spectrum disorders—lessons from research and practicve. Child and Adolescent Mental Health, 18(1), 52–63.  https://doi.org/10.1111/j.1475-3588.2012.00664.x.CrossRefPubMedGoogle Scholar
  14. Chen, F., Curran, P. J., Bollen, K. A., Kirby, J., & Paxton, P. (2008). An empirical evaluation of the use of fixed cutoff points in RMSEA test statistic in structural equation models. Sociological Methods & Research, 36(4), 462–494.  https://doi.org/10.1177/0049124108314720.CrossRefGoogle Scholar
  15. Cohen, J. (2013). Statistical power analysis for the behavioral sciences. New York: Academic Press.Google Scholar
  16. Cruz, M. (2005). WISC III: Escala de Inteligência Wechsler para crianças: Manual. Avaliação Psicológica, 4(2), 199–201.Google Scholar
  17. Dawson, G., Rogers, S., Munson, J., Smith, M., Winter, J., Greenson, J., … Varley, J. (2010). Randomized, controlled trial of an intervention for toddlers with autism: The Early Start Denver Model. Pediatrics, 125(1), e17–e23.  https://doi.org/10.1542/peds.2009-0958.CrossRefPubMedGoogle Scholar
  18. de Bildt, A., Mulder, E. J., Hoekstra, P. J., van Lang, N. D. J., Minderaa, R. B., & Hartman, C. A. (2009). Validity of the Children’s Social Behavior Questionnaire (CSBQ) in children with intellectual disability: Comparing the CSBQ with ADI-R, ADOS, and clinical DSM-IV-TR classification. Journal of Autism and Developmental Disorders, 39(10), 1464–1470.  https://doi.org/10.1007/s10803-009-0764-x.CrossRefPubMedPubMedCentralGoogle Scholar
  19. de Macedo, E. C., Mecca, T. P., Valentini, F., Laros, J. A., de Lima, R. M. F., & Schwartzman, J. S. (2013). Utilizando o teste não verbal de inteligência SON-R 2 1/2—7 [a] para avaliar crianças com Transtornos do Espectro do Autismo. Revista Educação Especial, 26(47), 603–618.  https://doi.org/10.5902/1984686X9779.CrossRefGoogle Scholar
  20. Du Toit, M. (2003). IRT from SSI: Bilog-MG, multilog, parscale, testfact. Lincolnwood: Scientific Software International.Google Scholar
  21. Duda, M., Kosmicki, J. A., & Wall, D. P. (2014). Testing the accuracy of an observation-based classifier for rapid detection of autism risk. Translational Psychiatry, 4(8), e424.  https://doi.org/10.1038/tp.2014.65.CrossRefPubMedPubMedCentralGoogle Scholar
  22. Falkmer, T., Anderson, K., Falkmer, M., & Horlin, C. (2013). Diagnostic procedures in autism spectrum disorders: A systematic literature review. European Child & Adolescent Psychiatry, 22(6), 329–340.  https://doi.org/10.1007/s00787-013-0375-0.CrossRefGoogle Scholar
  23. Feinstein, A. R., & Cicchetti, D. V. (1990). High agreement but low kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology, 43(6), 543–549. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/2189948.
  24. Feldman, H. M., Dale, P. S., Campbell, T. F., Colborn, D. K., Kurs-Lasky, M., Rockette, H. E., & Paradise, J. L. (2005). Concurrent and predictive validity of parent reports of child language at ages 2 and 3 years. Child Development, 76(4), 856–868.  https://doi.org/10.1111/j.1467-8624.2005.00882.x.CrossRefPubMedPubMedCentralGoogle Scholar
  25. Figueiredo, V. (2002). WISC-III: Escala de Inteligência Weschler para crianças—adaptação brasileira da 3° edição. São Paulo: Casa do Psicólogo.Google Scholar
  26. Gabrielsen, T. P., Farley, M., Speer, L., Villalobos, M., Baker, C. N., & Miller, J. (2015). Identifying autism in a brief observation. Pediatrics, 135(2), e330–e338.  https://doi.org/10.1542/peds.2014-1428.CrossRefPubMedGoogle Scholar
  27. Garcia, A. H. C., Viveiros, M. M., Schwartzman, J. S., & Brunoni, D. (2016). Transtornos do Espectro do Autismo: Avaliação e Comorbidades em Alunos de Barueri., São Paulo. Psicologia—Teoria E Prática, 18(1), 166–177.  https://doi.org/10.15348/1980-6906/psicologia.v18n1p166-177.CrossRefGoogle Scholar
  28. Garfin, D. G., McCallon, D., & Cox, R. (1988). Validity and reliability of the Childhood Autism Rating Scale with autistic adolescents. Journal of Autism and Developmental Disorders, 18(3), 367–378. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/3170455.
  29. Gollenberg, A. L., Lynch, C. D., Jackson, L. W., McGuinness, B. M., & Msall, M. E. (2010). Concurrent validity of the parent-completed Ages and Stages Questionnaires, 2nd Ed. with the Bayley Scales of Infant Development II in a low-risk sample. Child: Care, Health and Development, 36(4), 485–490.  https://doi.org/10.1111/j.1365-2214.2009.01041.x.Google Scholar
  30. Grodberg, D., Weinger, P. M., Halpern, D., Parides, M., Kolevzon, A., & Buxbaum, J. D. (2014). The autism mental status exam: Sensitivity and specificity using DSM-5 criteria for autism spectrum disorder in verbally fluent adults. Journal of Autism and Developmental Disorders, 44(3), 609–614.  https://doi.org/10.1007/s10803-013-1917-5.CrossRefPubMedPubMedCentralGoogle Scholar
  31. Grodberg, D., Weinger, P. M., Kolevzon, A., Soorya, L., & Buxbaum, J. D. (2012). Brief report: The autism mental status examination: Development of a brief autism-focused exam. Journal of Autism and Developmental Disorders, 42(3), 455–459.  https://doi.org/10.1007/s10803-011-1255-4.CrossRefPubMedGoogle Scholar
  32. Hanley, J. A., & McNeil, B. J. (1983). A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology, 148(3), 839–843.  https://doi.org/10.1148/radiology.148.3.6878708.CrossRefPubMedGoogle Scholar
  33. Harstad, E. B., Fogler, J., Sideridis, G., Weas, S., Mauras, C., & Barbaresi, W. J. (2015). Comparing diagnostic outcomes of autism spectrum disorder using DSM-IV-TR and DSM-5 criteria. Journal of Autism and Developmental Disorders, 45(5), 1437–1450.  https://doi.org/10.1007/s10803-014-2306-4.CrossRefPubMedGoogle Scholar
  34. Hu, L., & Bentler, P. (1999). No cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1–55.  https://doi.org/10.1080/10705519909540118.CrossRefGoogle Scholar
  35. Kim, M. A., & Lord, C. (2012). Combining information from multiple sources for the diagnosis of autism spectrum disorders for toddlers and young preschoolers from 12 to 47 months of age. Journal of Child Psychology and Psychiatry, 53(2), 143–151.  https://doi.org/10.1111/j.1469-7610.2011.02458.x.CrossRefPubMedGoogle Scholar
  36. Krug, D. A., Arick, J., & Almond, P. (1980). Behavior checklist for identifying severely handicapped individuals with high levels of autistic behavior. Journal of Child Psychology and Psychiatry, and Allied Disciplines, 21(3), 221–229. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/7430288.
  37. Lai, M.-C., Lombardo, M. V., Auyeung, B., Chakrabarti, B., & Baron-Cohen, S. (2015). Sex/gender differences and autism: Setting the scene for future research. Journal of the American Academy of Child & Adolescent Psychiatry, 54(1), 11–24.  https://doi.org/10.1016/j.jaac.2014.10.003.CrossRefGoogle Scholar
  38. Landa, R. J. (2008). Diagnosis of autism spectrum disorders in the first 3 years of life. Nature Clinical Practice Neurology, 4(3), 138–147.  https://doi.org/10.1038/ncpneuro0731.CrossRefPubMedGoogle Scholar
  39. Laros, J. A., Almeida, G. O. M., Valentini, F., & Lima, R. M. F. (2015). Dimensionalidade e evidências de validade convergente do SON-R 6-40. Temas Em Psicologia, 23(4), 929–945.  https://doi.org/10.9788/TP2015.4-10.CrossRefGoogle Scholar
  40. Le Couteur, A., Rutter, M., Lord, C., Rios, P., Robertson, S., Holdgrafer, M., & McLennan, J. (1989). Autism diagnostic interview: A standardized investigator-based instrument. Journal of Autism and Developmental Disorders, 19(3), 363–387. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/2793783.
  41. Lord, C., Risi, S., Lambrecht, L., Cook, E. H., Leventhal, B. L., DiLavore, P. C., … Rutter, M. (2000). The autism diagnostic observation schedule-generic: A standard measure of social and communication deficits associated with the spectrum of autism. Journal of Autism and Developmental Disorders, 30(3), 205–223. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/11055457.
  42. Luyster, R., Gotham, K., Guthrie, W., Coffing, M., Petrak, R., Pierce, K., … Lord, C. (2009). The autism diagnostic observation schedule—Toddler module: A new module of a standardized diagnostic measure for autism spectrum disorders. Journal of Autism and Developmental Disorders, 39(9), 1305–1320.  https://doi.org/10.1007/s10803-009-0746-z.CrossRefPubMedPubMedCentralGoogle Scholar
  43. Marsh, H., Hau, K., & Wen, Z. (2004). In search of golden rules: Comment on hypothesis-testing approaches to setting cutoff values for fit indexes and dangers in overgeneralizing Hu and Bentler’s 1999 findings. Structural Equation Modeling, 11, 320–341.CrossRefGoogle Scholar
  44. Marteleto, M. R. F., & Pedromônico, M. R. M. (2005). Validity of autism behavior checklist (ABC): Preliminary study. Revista Brasileira de Psiquiatria, 27(4), 295–301.  https://doi.org/10.1590/S1516-44462005000400008.CrossRefPubMedGoogle Scholar
  45. McEwen, F. S., Stewart, C. S., Colvert, E., Woodhouse, E., Curran, S., Gillan, N., … Bolton, P. (2016). Diagnosing autism spectrum disorder in community settings using the Development and Well-Being Assessment: Validation in a UK population-based twin sample. Journal of Child Psychology and Psychiatry, 57(2), 161–170.  https://doi.org/10.1111/jcpp.12447.CrossRefPubMedGoogle Scholar
  46. Muthen, B., & Asparouhov, T. (2004). IRT studies of many groups: The alignment method. Frontiers in Psychology, 11, 320–341.Google Scholar
  47. Muthén, B., Toit, D. S. H., & Spisic, D. (1997). Robust inference using weighted least squares and quadratic estimating equations in latent variable modeling with categorical and continuous outcomes. Psychometrika, 75, 1–45.Google Scholar
  48. Muthén, L. K., & Muthén, B. O. (2002). How to use a monte carlo study to decide on sample size and determine power. Structural Equation Modeling: A Multidisciplinary Journal, 9(4), 599–620.  https://doi.org/10.1207/S15328007SEM0904_8.CrossRefGoogle Scholar
  49. Obuchowski, N., McClish, D. (1997). Sample size determination for diagnostic accuracy studies involving binormal roc curve indices. Statistics in Medicine, 16(13), 1529–1542.CrossRefPubMedGoogle Scholar
  50. Ozonoff, S., Goodlin-Jones, B. L., & Solomon, M. (2005). Evidence-based assessment of autism spectrum disorders in children and adolescents. Journal of Clinical Child and Adolescent Psychology: The Official Journal for the Society of Clinical Child and Adolescent Psychology, American Psychological Association, Division 53, 34(3), 523–540.  https://doi.org/10.1207/s15374424jccp3403_8.CrossRefGoogle Scholar
  51. Pereira, A., Riesgo, R. S., & Wagner, M. B. (2008). Childhood autism: Translation and validation of the Childhood Autism Rating Scale for use in Brazil. Jornal de Pediatria, 84(6), 487–494.  https://doi.org/10.2223/JPED.1828.CrossRefPubMedGoogle Scholar
  52. Raykov, T. (1997). Scale reliability, Cronbach’s coefficient alpha, and violations of essential tau-equivalence for fixed congeneric components. Multivariate Behavioral Research, 32, 329–354.CrossRefPubMedGoogle Scholar
  53. Raykov, T. (2001). Bias of coefficient afor fixed congeneric measures with correlated errors. Applied Psychological Measurement, 25(1), 69–76.  https://doi.org/10.1177/01466216010251005.CrossRefGoogle Scholar
  54. Samms-Vaughan, M. E. (2014). The status of early identification and early intervention in autism spectrum disorders in lower- and middle-income countries. International Journal of Speech-Language Pathology, 16(1), 30–35.  https://doi.org/10.3109/17549507.2013.866271.CrossRefPubMedGoogle Scholar
  55. Schopler, E., Reichler, R. J., DeVellis, R. F., & Daly, K. (1980). Toward objective classification of childhood autism: Childhood Autism Rating Scale (CARS). Journal of Autism and Developmental Disorders, 10(1), 91–103. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/6927682.
  56. Shattuck, P. T., Durkin, M., Maenner, M., Newschaffer, C., Mandell, D. S., Wiggins, L., … Cuniff, C. (2009). Timing of identification among children with an autism spectrum disorder: Findings from a population-based surveillance study. Journal of the American Academy of Child and Adolescent Psychiatry, 48(5), 474–483.  https://doi.org/10.1097/CHI.0b013e31819b3848.CrossRefPubMedPubMedCentralGoogle Scholar
  57. Soto, S., Linas, K., Jacobstein, D., Biel, M., Migdal, T., & Anthony, B. J. (2015). A review of cultural adaptations of screening tools for autism spectrum disorders. Autism, 19(6), 646–661.  https://doi.org/10.1177/1362361314541012.CrossRefPubMedGoogle Scholar
  58. The Psychological Corporation. (1992). Weschler Individual Achievement Test. San Antonio, CA: The Psychological Corporation.Google Scholar
  59. Van Bourgondien, M. E., Marcus, L. M., & Schopler, E. (1992). Comparison of DSM-III-R and childhood autism rating scale diagnoses of autism. Journal of Autism and Developmental Disorders, 22(4), 493–506. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/1483973.
  60. Van de Schoot, R., Kluytmans, A., Tummers, L., Lugtig, P., Hox, J., & Muthen, B. (2013). Facing off with Scylla and Charybdis: A comparison of scalar, partial, and the novel possibility of approximate measurement invariance. Frontiers in Psychology, 23(4), 770.  https://doi.org/10.3389/fpsyg.2013.00770.Google Scholar
  61. Virués-Ortega, J. (2010). Applied behavior analytic intervention for autism in early childhood: Meta-analysis, meta-regression and dose-response meta-analysis of multiple outcomes. Clinical Psychology Review, 30(4), 387–399.  https://doi.org/10.1016/j.cpr.2010.01.008.CrossRefPubMedGoogle Scholar
  62. Wall, D. P., Kosmicki, J., DeLuca, T. F., Harstad, E., & Fusaro, V. A. (2012). Use of machine learning to shorten observation-based screening and diagnosis of autism. Translational Psychiatry, 2(4), e100.  https://doi.org/10.1038/tp.2012.10.CrossRefPubMedPubMedCentralGoogle Scholar
  63. Yu, C.-Y. (2002). Evaluating cutoff criteria of model fit indices for latent variable models with binary and continuous outcomes. University of California, Los Angeles.Google Scholar
  64. Zwaigenbaum, L., Bauman, M. L., Choueiri, R., Fein, D., Kasari, C., Pierce, K., … Wetherby, A. (2015a). Early identification and interventions for autism spectrum disorder: Executive summary. Pediatrics, 136(Supplement), S1–S9.  https://doi.org/10.1542/peds.2014-3667B.CrossRefPubMedGoogle Scholar
  65. Zwaigenbaum, L., Bauman, M. L., Choueiri, R., Kasari, C., Carter, A., Granpeesheh, D., … Natowicz, M. R. (2015b). Early intervention for children with autism spectrum disorder under 3 years of age: Recommendations for practice and research. Pediatrics, 136(Supplement), S60–S81.  https://doi.org/10.1542/peds.2014-3667E.CrossRefPubMedGoogle Scholar
  66. Zwaigenbaum, L., Bryson, S., Lord, C., Rogers, S., Carter, A., Carver, L., … Yirmiya, N. (2009). Clinical assessment and management of toddlers with suspected autism spectrum disorder: Insights from studies of high-risk infants. Pediatrics, 123(5), 1383–1391.  https://doi.org/10.1542/peds.2008-1606.CrossRefPubMedPubMedCentralGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2017

Authors and Affiliations

  • Cristiane S. Paula
    • 1
    • 2
  • Graccielle Rodrigues Cunha
    • 2
  • Daniela Bordini
    • 2
  • Decio Brunoni
    • 1
    • 3
  • Ana Claudia Moya
    • 2
  • Cleonice Alves Bosa
    • 4
  • Jair J. Mari
    • 2
  • Hugo Cogo-Moreira
    • 2
  1. 1.Development Disorders ProgramMackenzie UniversitySão PauloBrazil
  2. 2.Departament of PsychiatryFederal University of Sao Paulo (UNIFESP)São PauloBrazil
  3. 3.Departament of GeneticsFederal University of Sao Paulo (UNIFESP)São PauloBrazil
  4. 4.Deparment of PsychologyUniversidade Federal do Rio Grande do SulPorto AlegreBrazil

Personalised recommendations