Using Expert Panels to Examine the Content Validity and Inter-Rater Reliability of the ABLLS-R

  • Jennifer Usry
  • Scott W. Partington
  • James W. Partington
ORIGINAL ARTICLE

Abstract

The assessment literature cites several instruments used to assess the skills of children with an autism spectrum disorder (ASD) diagnosis, but many lack adequate empirical support for their psychometric properties. The Assessment of Basic Language and Learning Skills-Revised (ABLLS-R) is a popular assessment used by clinicians to measure the skills of children with ASD. Despite its widespread use, the ABLLS-R contains limited research on its psychometric properties. The current study sought to extend the recent research on the psychometric properties of the ABLLS-R by using data obtained from two separate panels of expert raters to evaluate its content validity and the inter-rater reliability of its scores. Our results demonstrate evidence of content validity as at least five out of six expert panel members rated 441 out of the 544 ABLLS-R items (or 81% of the items in the assessment) as “essential.” We also found evidence of excellent inter-rater reliability (intraclass correlation coefficient = .95, p < .001) across the ABLLS-R scores obtained from a second panel of expert raters. These findings extend the existing literature and further document the ABLLS-R as a valid instrument that yields reliable scores.

Keywords

Autism Assessments ABLLS-R Reliability Validity Psychometric properties 

Notes

Compliance with Ethical Standards

Ethical Approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed Consent

Informed consent was obtained from all individual participants included in the study.

Conflict of Interest

The first and second authors declare that they have no conflict of interest. The third author owns stock in the company that publishes the ABLLS-R.

References

  1. Aman, M.G., Novotny, S., Samango-Sprouse, C., Lecavalier, L., Leonard, E., Gadow, K.D…Chez, M. (2004). Outcome measures for clinical drug trials in autism. CNS Spectrums, 9, 36–47.  https://doi.org/10.1017/S1092852900008348.
  2. American Educational Research Association, American Psychological Association, National Council on Measurement in Education, & Joint Committee on Standards for Educational and Psychological Testing. (2014). Standards for educational and psychological testing. Washington, DC: AERA.Google Scholar
  3. American Medical Association. (2014). CPT assistant. Chicago: American Medical Association.Google Scholar
  4. Center for Disease Control and Prevention (2015). Data and statistics. Retrieved from http://www.cdc.gov/NCBDDD/autism/data.html
  5. Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6, 284–290.  https://doi.org/10.1037/1040-3590.6.4.284.CrossRefGoogle Scholar
  6. Every Student Succeeds Act, Pub. L. No. 114–95 § 114 Stat. 1177 (2015).Google Scholar
  7. Guldberg, K. (2010). Educating children on the autism spectrum: Preconditions for inclusion and notions of “best autism practice” in the early years. British Journal of Special Education, 37, 168–170.  https://doi.org/10.1111/j.1467-8578.2010.00482.x.CrossRefGoogle Scholar
  8. Individuals with Disabilities Education Act, 20 U.S.C. § 1400 (2004).Google Scholar
  9. Landers, R. N. (2011). Computing intraclass correlations (ICC) as estimates of inter-rater reliability in SPSS. Retrieved from http://neoacademic.com/2011/11/16/computing-intraclass-correlations-icc-as-estimates-of-inter-rater-reliability-in-spss/#.U6z9uChhmGl. doi:10.15200/winn.143518.81744.
  10. Lawshe, C. H. (1975). A quantitative approach to content validity. Personnel Psychology, 28, 563–575.  https://doi.org/10.1111/j.1744-6570.1975.tb01393.x.CrossRefGoogle Scholar
  11. Lynn, M. R. (1986). Determination and quantification of content validity. Nursing Research, 35, 382–385.  https://doi.org/10.1097/00006199-198611000-00017.CrossRefPubMedGoogle Scholar
  12. Malkin, A., Dixon, M. R., Speelman, R. C., & Luke, N. (2016). Evaluating the relationships between the PEAK Relations Training System-Direct Training Module, Assessment of Basic Language and Learning Skills-Revised, and the Vineland Adaptive Behavior Scales-II. Journal of Developmental and Physical Disabilities.  https://doi.org/10.1007/s10882-016-9527-8.
  13. Partington, J. W. (2010a). The Assessment of Basic Language and Learning Skills-Revised. Pleasant Hill: Behavior Analysts, Inc..Google Scholar
  14. Partington, J. W. (2010b). The Assessment of Basic Language and Learning Skills-Revised: Scoring Instructions and IEP Development Guide. Pleasant Hill: Behavior Analysts, Inc..Google Scholar
  15. Partington, J. W., Bailey, A., & Partington, S. W. (2016). A pilot study examining the test-retest and internal consistency reliability of the ABLLS-R. Journal of Psychoeducational Assessment.  https://doi.org/10.1177/0734282916678348.
  16. Salkind, N. (2013). Encyclopedia of research design: Content validity. Thousand Oaks: SAGE Publications, Inc..Google Scholar
  17. Schwartz, I. S., Boulware, G., McBride, B. J., & Sandall, S. R. (2001). Functional assessment strategies for young children with autism. Focus on Autism and Other Developmental Disabilities, 16, 222–227.  https://doi.org/10.1177/108835760101600404.CrossRefGoogle Scholar
  18. Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420–428.  https://doi.org/10.1037/0033-2909.86.2.420.CrossRefPubMedGoogle Scholar
  19. Sparrow, S. S., Cicchetti, D. V., & Balla, D. A. (2005). Vineland Adaptive Behavior Scales (2nd ed.). Bloomington, MN: Pearson.  https://doi.org/10.1037/t15164-000.Google Scholar
  20. Sundberg, M. (2008). VB-MAPP Verbal Behavior Milestones Assessment and Placement Program: A language and social skills assessment program for children with autism or other developmental disabilities: Guide. Concord: AVB Press.Google Scholar
  21. Thompson, T. (2007). Making Sense of Autism. Baltimore: Paul H. Brookes Publishing Co..Google Scholar
  22. Thompson, T. (2011). Individualized Autism Intervention for Young Children. Baltimore: Paul H. Brookes Publishing Co..Google Scholar
  23. Wilson, F. R., Pan, W., & Schumsky, D. A. (2012). Recalculation of the critical values for Lawshe’s content validity ratio. Measurement and Evaluation in Counseling and Development, 45, 197–210.  https://doi.org/10.1177/0748175612440286.CrossRefGoogle Scholar
  24. Wolak, M. E., Fairbairn, D. J., & Paulsen, Y. R. (2012). Guidelines for estimating repeatability. Methods in Ecology and Evolution, 3, 129–137.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2017

Authors and Affiliations

  • Jennifer Usry
    • 1
  • Scott W. Partington
    • 2
  • James W. Partington
    • 2
  1. 1.Liberty UniversityLynchburgUSA
  2. 2.Behavior Analysts, Inc.Walnut CreekUSA

Personalised recommendations