Skip to main content

Reliability Generalization for Childhood Autism Rating Scale

Journal of Autism and Developmental Disorders volume 43, pages 2855–2865 (2013)Cite this article

Abstract

The Childhood Autism Rating Scale (CARS) is a popular behavior-observation instrument that was developed more than 34 years ago and has since been adopted in a wide variety of contexts for assessing the presence and severity of autism symptomatology in both children and adolescents. This investigation of the reliability of CARS scores involves meta-analysis and meta-regression of empirical data from reports of original research that made use of CARS between 1980 and 2012. Findings of good internal consistency (.896, 95 % CI .877–.913) and good interrater reliability (.796, 95 % CI .736–.844) support use of CARS at least in early-phase, exploratory research. Evidence of heterogeneity among literature data indicates that reliability is a property of CARS scores and is not intrinsic to the instrument itself. As the first of its kind pertaining to autism, this investigation provides guidance for reviews of other instruments’ ratings.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

43,34 €

Price includes VAT (Finland)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2

References

  • Arbelle, S., Sigman, M. D., & Kasari, C. (1994). Compliance with parental prohibition in autistic children. Journal of Autism and Developmental Disorders, 24, 693–702.

    PubMed  Article  Google Scholar 

  • Arizona Software. (2008). GraphClick graph-digitization software: Release 3.0 [computer software]. Neuchâtel: Arizona Software.

    Google Scholar 

  • Bagby, R. M., Ryder, A. G., Schuller, D. B., & Marshall, M. B. (2004). The Hamilton Depression Rating Scale: Has the gold standard become a lead weight? American Journal of Psychiatry, 161, 2163–2177.

    PubMed  Article  Google Scholar 

  • Baghdadli, A., Assouline, B., Sonié, S., Pernon, E., Darrou, C., Michelon, C., et al. (2012). Developmental trajectories of adaptive behaviors from early childhood to adolescence in a cohort of 152 children with autism spectrum disorders. Journal of Autism and Developmental Disorders, 42, 1314–1325.

    PubMed  Article  Google Scholar 

  • Baker, W. L., White, C. M., Cappelleri, J. C., Kluger, J., & Coleman, C. I. (2009). Understanding heterogeneity in meta-analysis: The role of meta-regression. International Journal of Clinical Practice, 63, 1426–1434.

    PubMed  Article  Google Scholar 

  • Bebko, J. M., Konstantareas, M. M., & Springer, J. (1987). Parent and professional evaluations of family stress associated with characteristics of autism. Journal of Autism and Developmental Disorders, 17, 565–576.

    PubMed  Article  Google Scholar 

  • Bernabei, P., Cerquiglini, A., Cortesi, F., & D’Ardia, C. (2007). Regression versus no regression in the autistic disorder: Developmental trajectories. Journal of Autism and Developmental Disorders, 37, 580–588.

    PubMed  Article  Google Scholar 

  • Bornmann, L., Mutz, R., & Daniel, H. D. (2010). A reliability-generalization study of journal peer reviews: A multilevel meta-analysis of inter-rater reliability and its determinants. PLoS One, 5, 1–10.

    Google Scholar 

  • Botella, J., Suero, M., & Gambara, H. (2010). Psychometric inferences from a meta-analysis of reliability and internal consistency coefficients. Psychological Methods, 4, 386–397.

    Article  Google Scholar 

  • Bristol, M. M. (1987). Mothers of children with autism or communication disorders: Successful adaptation and the Double ABCX model. Journal of Autism and Developmental Disorders, 17, 469–486.

    PubMed  Article  Google Scholar 

  • Campbell, J. S. (2003). Meta-analysis and reliability generalization of the Psychopathy Checklist applied to youth. Unpublished doctoral dissertation from University of Northern Colorado.

  • Chen, G. M., Yoder, J. J., Ganzei, B. L., Goodwin, M. S., & Belmonte, M. K. (2012). Harnessing repetitive behaviours to engage attention and learning in a novel therapy for autism: An exploratory analysis. Frontiers in Psychology, doi:10.3389/fpsyg.2012.00012.

    Google Scholar 

  • Chez, M. G., & Buchanan, C. P. (2000). Comment: Reply: Reply to B. Rimland’s “Comments on ‘Secretin and autism: A two-part clinical investigation’”. Journal of Autism and Developmental Disorders, 30, 97–98.

    Article  Google Scholar 

  • Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6, 284–290.

    Article  Google Scholar 

  • Connelly, B. S. (2008). The reliability, convergence, and predictive validity of personality ratings: An other perspective. Unpublished doctoral dissertation from University of Minnesota.

  • Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297–334.

    Article  Google Scholar 

  • Duhachek, A., & Iacobucci, D. (2004). Alpha’s standard error (ASE): An accurate and precise confidence interval estimate. Journal of Applied Psychology, 5, 792–808.

    Article  Google Scholar 

  • Fein, D., Skoff, B., & Mirsky, A. F. (1981). Clinical correlates of brainstem dysfunction in autistic children. Journal of Autism and Developmental Disorders, 11, 303–315.

    PubMed  Article  Google Scholar 

  • Freeman, N. L., Perry, A., & Factor, D. C. (1991). Child behaviours as stressors: Replicating and extending the use of the CARS as a measure of stress: A research note. Journal of Child Psychology and Psychiatry, 32, 1025–1030.

    PubMed  Article  Google Scholar 

  • Hakstian, A. R., & Whalen, T. E. (1976). A k-sample significance test for independent alpha coefficients. Psychometrika, 41, 219–231.

    Article  Google Scholar 

  • Hedges, L. V., Tipton, E., & Johnson, M. C. (2010). Robust variance estimation in meta-regression with dependent effect size estimates. Research Synthesis Methods, 1, 39–65.

    Article  Google Scholar 

  • Hobson, R. P., Lee, A., & Brown, R. (1999). Autism and congenital blindness. Journal of Autism and Developmental Disorders, 29, 45–56.

    PubMed  Article  Google Scholar 

  • Howell, R. T., & Shields, A. L. (2008). The file drawer problem in reliability generalization: A strategy to compute fail-safe N with reliability coefficients. Educational and Psychological Measurement, 68, 120–128.

    Article  Google Scholar 

  • Hranilovic, D., Bujas-Petkovic, Z., Vragovic, R., Vuk, T., Hock, K., & Jernej, B. (2007). Hyperserotonemia in adults with autistic disorder. Journal of Autism and Developmental Disorders, 37, 1934–1940.

    PubMed  Article  Google Scholar 

  • Kasari, C., & Sigman, M. (1997). Linking parental perceptions to interactions in young children with autism. Journal of Autism and Developmental Disorders, 27, 39–57.

    PubMed  Article  Google Scholar 

  • Konstantareas, M. M., & Homatidis, S. (1989). Assessing child symptom severity and stress in parents of autistic children. Journal of Child Psychology and Psychiatry, 30, 459–470.

    PubMed  Article  Google Scholar 

  • Konstantareas, M. M., & Homatidis, S. (1999). Chromosomal abnormalities in a series of children with autistic disorder. Journal of Autism and Developmental Disorders, 29, 275–285.

    PubMed  Article  Google Scholar 

  • Konstantareas, M. M., Homatidis, S., & Plowright, C. M. S. (1992). Assessing resources and stress in parents of severely dysfunctional children through the Clarke modification of Holroyd’s Questionnaire on Resources and Stress. Journal of Autism and Developmental Disorders, 22, 217–234.

    PubMed  Article  Google Scholar 

  • Konstantareas, M. M., & Stewart, K. (2006). Affect regulation and temperament in children with autism spectrum disorder. Journal of Autism and Developmental Disorders, 36, 143–154.

    PubMed  Article  Google Scholar 

  • Kurita, H., Miyake, Y., & Katsuno, K. (1989). Reliability and validity of the Childhood Autism Rating Scale-Tokyo version (CARS–TV). Journal of Autism and Developmental Disorders, 19, 389–396.

    PubMed  Article  Google Scholar 

  • Lance, C. E., Butts, M. M., & Michels, L. C. (2006). The sources of four commonly reported cutoff criteria: What did they really say? Organizational Research Methods, 9, 202–220.

    Article  Google Scholar 

  • Laushey, K. M., & Heflin, L. J. (2000). Enhancing social skills of kindergarten children with autism through the training of multiple peers as tutors. Journal of Autism and Developmental Disorders, 30, 183–193.

    PubMed  Article  Google Scholar 

  • LeBreton, J. M., & Senter, J. L. (2008). Answers to 20 questions about interrater reliability and interrater agreement. Organizational Research Methods, 11, 815–852.

    Article  Google Scholar 

  • Li, A., & Bagger, J. (2007). The balanced inventory of desirable responding (BIDR): A reliability generalization study. Educational and Psychological Measurement, 67, 525–544.

    Article  Google Scholar 

  • Lord, C., Schopler, E., & Revicki, D. (1982). Sex differences in autism. Journal of Autism and Developmental Disorders, 12, 317–330.

    PubMed  Article  Google Scholar 

  • Matese, M., Matson, J. L., & Sevin, J. (1994). Comparison of psychotic and autistic children using behavioral observation. Journal of Autism and Developmental Disorders, 24, 83–94.

    PubMed  Article  Google Scholar 

  • Matson, J. L., Smiroldo, B. B., & Hastings, T. L. (1998). Validity of the autism/pervasive developmental disorder subscale of the Diagnostic Assessment for the Severely Handicapped-II. Journal of Autism and Developmental Disorders, 28, 77–81.

    PubMed  Article  Google Scholar 

  • Mokkink, L. B., Terwee, C. B., Stratford, P. W., Alonso, J., Patrick, D. L., Riphagen, I., et al. (2009). Evaluation of the methodological quality of systematic reviews of health status measurement instruments. Quality of Life Research, 18, 313–333.

    PubMed  Article  Google Scholar 

  • Nassen, L. M. (2008). An internal consistency reliability generalization of the Child Behavior Checklist. Unpublished doctoral dissertation from University of Northern Colorado.

  • Njardvik, U., Matson, J. L., & Cherry, K. E. (1999). A comparison of social skills in adults with autistic disorder, pervasive developmental disorder not otherwise specified, and mental retardation. Journal of Autism and Developmental Disorders, 29, 287–295.

    PubMed  Article  Google Scholar 

  • Nordin, V., Gillberg, C., & Nyden, A. (1998). The Swedish version of the Childhood Autism Rating Scale in a clinical setting. Journal of Autism and Developmental Disorders, 28, 69–75.

    PubMed  Article  Google Scholar 

  • Pereira, T. V., Patsopoulos, N. A., Salanti, G., & Ioannidis, J. P. A. (2010). Critical interpretation of Cochran’s Q test depends on power and prior assumptions about heterogeneity. Research Synthesis Methods, 1, 149–161.

    Article  Google Scholar 

  • Perry, A., Condillac, R. A., Freeman, N. L., Dunn-Geier, J., & Belair, J. (2005). Multi-site study of the Childhood Autism Rating Scale (CARS) in five clinical groups of young children. Journal of Autism and Developmental Disorders, 35, 625–634.

    PubMed  Article  Google Scholar 

  • Ponterotto, J. G., & Charter, R. A. (2009). Statistical extensions of Ponterotto and Ruckdeschel’s (2007) reliability matrix for estimating the adequacy of internal consistency coefficients. Perceptual and Motor Skills, 108, 878–886.

    PubMed  Article  Google Scholar 

  • Reichler, R. J., & Schopler, E. (1971). Observations on the nature of human relatedness. Journal of Autism and Childhood Schizophrenia, 1, 283–296.

    PubMed  Article  Google Scholar 

  • Rodriguez, M. C., & Jung, T. (2003). The replicability of reliability generalization. Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL. Accessed May 27, 2011 from http://www.edmeasurement.net/aera/papers/rg%20replication%20aera.pdf.

  • Rodriguez, M. C., & Maeda, Y. (2006). Meta-analysis of coefficient alpha. Psychological Methods, 11, 306–322.

    PubMed  Article  Google Scholar 

  • Rousson, V., Gasser, T., & Seifert, B. (2002). Assessing intrarater, interrater and test-retest reliability of continuous measurements. Statistics in Medicine, 21, 3431–3446.

    PubMed  Article  Google Scholar 

  • Ryngala, D. J., Shields, A. L., & Caruso, J. C. (2005). Reliability generalization of the revised Children’s Manifest Anxiety Scale. Educational and Psychological Measurement, 65, 259–271.

    Article  Google Scholar 

  • Sæmundsen, E., Magnússon, P., Smári, J., & Sigurdardóttir, S. (2003). Autism Diagnostic Interview-Revised and the Childhood Autism Rating Scale: Convergence and discrepancy in diagnosing autism. Journal of Autism and Developmental Disorders, 33, 319–328.

    PubMed  Article  Google Scholar 

  • Samtani, A., Sterling-Lewis, K., Scholten, R. J. P. M., Woolfenden, S., Hooft, L., & Williams, K. (2011). Diagnostic tests for autism spectrum disorder (ASD) in preschool children. Cochrane Database for Systematic Reviews Collaboration,. doi:10.1002/14651858.cd009044.

    Google Scholar 

  • Sawilowsky, S. S. (2000). Psychometrics versus datametrics: Comment on Vacha-Haase’s “reliability generalization” method and some Epm [sic] editorial policies. Educational and Psychological Measurement, 60, 157–173.

    Article  Google Scholar 

  • Schlosser, R. W., Wendt, O., & Sigafoos, J. (2007). Not all systematic reviews are created equal: Considerations for appraisal. Evidence-Based Communication Assessment and Intervention, 1, 138–150.

    Article  Google Scholar 

  • Schopler, E., Andrews, C. E., & Strupp, K. (1979a). Do autistic children come from upper-middle-class parents? Journal of Autism and Developmental Disorders, 9, 139–152.

    PubMed  Article  Google Scholar 

  • Schopler, E., & Reichler, R. J. (1971). Parents as cotherapists in the treatment of psychotic children. Journal of Autism and Childhood Schizophrenia, 1, 87–102.

    PubMed  Article  Google Scholar 

  • Schopler, E., & Reichler, R. J. (1972). How well do parents understand their own psychotic child? Journal of Autism and Childhood Schizophrenia, 2, 387–400.

    PubMed  Article  Google Scholar 

  • Schopler, E., Reichler, R. J., DeVellis, R. F., & Daly, K. (1980). Toward objective classification of childhood autism: Childhood Autism Rating Scale (CARS). Journal of Autism and Developmental Disorders, 10, 91–103.

    PubMed  Article  Google Scholar 

  • Schopler, E., Reichler, R. J., & Renner, B. R. (1988). The Childhood Autism Rating Scale (CARS). Los Angeles, CA: Western Psychological Services.

    Google Scholar 

  • Schopler, E., Rutter, M., & Chess, S. (1979b). Editorial: Change of journal scope and title. Journal of Autism and Developmental Disorders, 9, 1–10.

    PubMed  Article  Google Scholar 

  • Sevin, J. A., Matson, J. L., Coe, D. A., Fee, V. E., & Sevin, B. M. (1991). A comparison and evaluation of three commonly used autism scales. Journal of Autism and Developmental Disorders, 21, 417–432.

    PubMed  Article  Google Scholar 

  • Sloan, J. L. (1978). Differential development of autistic symptoms in a pair of fraternal twins. Journal of Autism and Childhood Schizophrenia, 8, 191–202.

    PubMed  Article  Google Scholar 

  • Sloan, J. L., & Marcus, L. (1981). Some findings on the use of the adaptive behavior scale with autistic children. Journal of Autism and Developmental Disorders, 11, 191–199.

    PubMed  Article  Google Scholar 

  • Stanley, G. C., & Konstantareas, M. M. (2007). Symbolic play in children with autism spectrum disorder. Journal of Autism and Developmental Disorders, 37, 1215–1223.

    PubMed  Article  Google Scholar 

  • StataCorp. (2012). Stata statistical software: Release 12.1 [computer software]. College Station, TX: StataCorp LP.

    Google Scholar 

  • Sun, S. (2011). Meta-analysis of Cohen’s kappa. Health Services Outcomes Research Methodology, 11, 145–163.

    Article  Google Scholar 

  • Teal, M. B., & Wiebe, M. J. (1986). A validity analysis of selected instruments used to assess autism. Journal of Autism and Developmental Disorders, 16, 485–494.

    PubMed  Article  Google Scholar 

  • Thompson, B., & Vacha-Haase, T. (2000). Psychometrics is datametrics: The test is not reliable. Educational and Psychological Measurement, 60, 174–195.

    Google Scholar 

  • Trajković, G., Starčević, V., Latas, M., Leštarević, M., Ille, T., Bukumirić, Z., et al. (2011). Reliability of the Hamilton Rating Scale for Depression: A meta-analysis over a period of 49 years. Psychiatry Research, 189, 1–9.

    PubMed  Article  Google Scholar 

  • Travis, L., Sigman, M., & Ruskin, E. (2001). Links between social understanding and social behavior in verbally able children with autism. Journal of Autism and Developmental Disorders, 31, 119–130.

    PubMed  Article  Google Scholar 

  • Vacha-Haase, T., Kogan, L. R., & Thompson, B. (2000). Sample compositions and variabilities in published studies versus those in test manuals: Validity of score reliability inductions. Educational and Psychological Measurement, 60, 509–522.

    Article  Google Scholar 

  • Vacha-Haase, T., & Thompson, B. (2011). Score reliability: A retrospective look back at 12 years of reliability generalization studies. Measurement and Evaluation in Counseling and Development, 44, 159–168.

    Article  Google Scholar 

  • Van Bourgondien, M. E., Marcus, L. M., & Schopler, E. (1992). Comparison of DSM-III-R and Childhood Autism Rating Scale diagnoses of autism. Journal of Autism and Developmental Disorders, 22, 493–506.

    PubMed  Article  Google Scholar 

  • Vassar, M., & Hale, W. (2007). Reliability reporting practices in youth life satisfaction research. Social Indicators Research, 83, 487–496.

    Article  Google Scholar 

  • Vassar, M., Ridge, J. W., & Hill, A. D. (2008). Inducing score reliability from previous reports: An examination of life satisfaction studies. Social Indicators Research, 87, 27–45.

    Article  Google Scholar 

  • Viding, E., & Blakemore, S. J. (2007). Endophenotype approach to developmental psychopathology: Implications for autism research. Behavioral Genetics, 37, 51–60.

    Article  Google Scholar 

  • Wiggins, L. D., Robins, D. L., Adamson, L. B., Bakeman, R., & Henrich, C. C. (2012). Support for a dimensional view of autism spectrum disorders in toddlers. Journal of Autism and Developmental Disorders, 42, 191–200.

    PubMed  Article  Google Scholar 

  • Wilkinson, L., & American Psychological Association (APA) Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594–604.

  • Wilson, R., Pascalis, O., & Blades, M. (2007). Familiar face recognition in children with autism: The differential use of inner and outer face parts. Journal of Autism and Developmental Disorders, 37, 314–320.

    PubMed  Article  Google Scholar 

  • Woolfenden, S., Sarkozy, V., Ridley, G., & Williams, K. (2012). A systematic review of the diagnostic stability of autism spectrum disorder. Research in Autism Spectrum Disorders, 6, 345–354.

    Article  Google Scholar 

  • Yamashita, Y., Fujimoto, C., Nakajima, E., Isagai, T., & Matsuishi, T. (2003). Possible association between congenital cytomegalovirus infection and autistic disorder. Journal of Autism and Developmental Disorders, 33, 455–459.

    PubMed  Article  Google Scholar 

  • Zheng, Y. P., Zhao, J. P., Phillips, M., Liu, J. B., Cai, M. F., Sun, S. Q., et al. (1988). Validity and reliability of the Chinese Hamilton Depression Rating Scale. British Journal of Psychiatry, 152, 660–664.

    PubMed  Article  Google Scholar 

Download references

Acknowledgments

Authors thank anonymous reviewers for thoughtful feedback.

Author information

Affiliations

  1. Wolfson College, Cambridge, CB3 9BB, UK

    Jonathan Breidbord

  2. Hull York Medical School, Centre for Health and Population Sciences/Mental Health and Addiction Research Group, Department of Health Sciences, University of York, York, YO10 5DD, UK

    Tim J. Croudace

  3. Developmental and Lifecourse Research Group, Department of Psychiatry, University of Cambridge, Douglas House, 18b Trumpington Road, Cambridge, CB2 8AH, UK

    Tim J. Croudace

Authors
  1. Jonathan Breidbord
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tim J. Croudace
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jonathan Breidbord.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Breidbord, J., Croudace, T.J. Reliability Generalization for Childhood Autism Rating Scale. J Autism Dev Disord 43, 2855–2865 (2013). https://doi.org/10.1007/s10803-013-1832-9

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10803-013-1832-9

Keywords

  • Internal consistency
  • Meta-analysis
  • Psychometrics
  • Questionnaires
  • Reliability generalisation
  • Reproducibility of results
  • Score stability