Development of the Evaluative Method for Evaluating and Determining Evidence-Based Practices in Autism

Abstract

Although research in autism has grown more sophisticated, the gap between research knowledge and applicability of research in real world settings has grown. There have been a number of different reviews of evidence-based practices of treatments for young children with autism. Reviews which have critically evaluated the empirical evidence have not found any treatments that can be considered evidence-based. Reasons for this shortcoming are explored, and a new method for the evaluation of empirical evidence is provided. Future uses of this evaluative method are provided as well as a discussion of how this tool might aid in narrowing the research to practice gap.

This is a preview of subscription content, log in to check access.

Notes

  1. 1.

    With regard to the treatment of single subject research, our concerns are twofold; some definitions did not allow single subject studies to be used in the determination of EBP (Francis 2005; Shavelson and Towne 2002), and no definition of EBP that included single subject research provided a way to synthesize research across both group and single subject research (Kratochwill and Stoiber 2002; Lonigan et al. 1998; Lord et al. 2001; Odom et al. 2005). Because much of the research on treatments for young children with autism uses single subject research methods (see Odom et al. 2003), we feel it is imperative that single subject research be included in a definition of EBP used to review treatments for young children with autism.

References

  1. Baker, B. L., & Feinfield, K. A. (2003). Early intervention. Current Opinion in Psychiatry, 16, 503–509.

    Article  Google Scholar 

  2. Bristol, M. M., Cohen, D. J., Costello, E. J., Denckla, M., Eckberg, T. J., Kallen, R., et al. (1996). State of the science in autism: Report to the National Institutes of Health. Journal of Autism and Developmental Disorders, 26, 121–154.

    PubMed  Article  Google Scholar 

  3. Chorpita, B. F. (2003). The frontier of evidence-based practice. In A. E. Kazdin & J. R. Weisz (Eds.), Evidence-based psychotherapies for children and adolescents (pp. 42–59). New York: Guilford.

    Google Scholar 

  4. Cicchetti, D. V. (2001). The precision of reliability and validity estimates re-visited: Distinguishing between clinical and statistical significance of sample size requirements. Journal of Clinical and Experimental Neuropsychology, 23, 695–700.

    PubMed  Article  Google Scholar 

  5. Cutspec, P. A. (2004). Origins of evidence-based approaches to best practice: Evidence-based medicine. Centerscope, 2(1), 1–12.

    Google Scholar 

  6. Doehring, P., Reichow, B., & Volkmar, F. R. (2007, March). Is it evidenced-based? How to evaluate claims of effectiveness for autism. Paper presented at the International Association for Positive Behavior Support Conference, Boston, MA.

  7. Filipek, P. A., Accardo, P. J., Ashwal, S., Baranek, G. T., Cook, E. H., Jr., Dawson, G., et al. (2000). Practice parameter: Screening and diagnosis of autism. Neurology, 55, 468–479.

    PubMed  Google Scholar 

  8. Francis, K. (2005). Autism interventions: A critical update. Developmental Medicine & Child Neurology, 47, 493–499.

    Article  Google Scholar 

  9. Goodheart, C. D., Kazdin, A. E., & Sternberg, R. J. (2006). Evidence-based psychotherapy: Where practice and research meet. Washington, DC: American Psychological Association.

    Google Scholar 

  10. Greenhalgh, T. (2001). How to read a paper: The basics of evidence based medicine (2nd ed.). London: BMJ Books.

    Google Scholar 

  11. Gresham, F. M., Beebe-Frankenberger, M. E., MacMillan, D. L. (1999). A selective review of treatments for children with autism: Description and methodological considerations. The School Psychology Review, 28, 559–575.

    Google Scholar 

  12. Grice, J. W., Jackson, B. J., & McDaniel, B. L. (2006). Bridging the idiographic-nomothetic divide: A follow-up study. Journal of Personality, 74, 1191–1218.

    PubMed  Article  Google Scholar 

  13. Hamilton, J. (2005). Clinicians’ guide to evidence-based practice. Journal of American Academy of Child and Adolescent Psychiatry, 44, 494–498.

    Article  Google Scholar 

  14. Horner, R. H., Carr, E. G., Halle, J., McGee, G., Odom, S., & Wolery, M. (2005). The use of single-subject research to identify evidence-based practice in special education. Exceptional Children, 71, 165–179.

    Google Scholar 

  15. Kasari, C. (2002). Assessing change in early intervention programs for young children with autism. Journal of Autism and Developmental Disorders, 32, 447–461.

    PubMed  Article  Google Scholar 

  16. Kazdin, A. E. (2001). Bridging the enormous gaps of theory with therapy research and practice. Journal of Clinical Child Psychology, 30, 59–66.

    PubMed  Article  Google Scholar 

  17. Kazdin, A. E., & Weisz, J. R. (Eds.). (2003). Evidence-based psychotherapies for children and adolescents. New York: Guilford.

    Google Scholar 

  18. Klin, A., Lang, J., Cicchetti, D. V., & Volkmar, F. (2000). Inter-rater reliability of clinical diagnosis and DSM-IV criteria for autistic disorder: Results of the DSM-IV autism field trial. Journal of Autism and Developmental Disorders, 30, 163–167.

    PubMed  Article  Google Scholar 

  19. Kratochwill, T. R., & Stoiber, K. C. (2002). Evidence-based interventions in school psychology: Conceptual foundations of the Procedural and Coding Manual of Division 16 and the Society for the Study of School Psychology Task Force. School Psychology Quarterly, 17, 341–389.

    Article  Google Scholar 

  20. Landis, J. R., & Koch, G. G. (1977). The measure of observer agreement for categorical data. Biometrics, 3, 159–174.

    Article  Google Scholar 

  21. Levin, J. R. (2002). How to evaluate the evidence of evidence-based interventions. School Psychology Quarterly, 17, 483–492.

    Article  Google Scholar 

  22. Lonigan, C. J., Elbert, J. C., & Johnson, S. B. (1998). Empirically supported psychosocial interventions for children: An overview. Journal of Clinical Child Psychology, 27, 138–145.

    PubMed  Article  Google Scholar 

  23. Lord, C., Bristol-Power, M., Cafiero, J. M., Filipek, P. A., Gallagher, J. J., Harris, S. L., et al. (2001). Educating children with autism. Washington, DC: National Academy Press.

    Google Scholar 

  24. Lucas, S. M., & Cutspec, P. A. (2005). The role and process of literature searching in the preparation of research synthesis. Centerscope, 4(3), 1–26.

    Google Scholar 

  25. McHugo, G. J., & Drake, R. E. (2003). Finding and evaluating the evidence: A critical step in evidence-based medicine. Psychiatric Clinics of North America, 26, 821–831.

    PubMed  Article  Google Scholar 

  26. Norcross, J. C., Beutler, L. E., & Levant, R. F. (2006). Evidence-based practices in mental health: Debate and dialogue on the fundamental questions. Washington, DC: American Psychological Association.

    Google Scholar 

  27. Odom, S. L., Brantlinger, E., Gersten, R., Horner, R. H., Thompson, B., & Harris, K. R. (2005). Research in special education: Scientific methods and evidence-based practices. Exceptional Children, 71, 137–148.

    Google Scholar 

  28. Odom, S. L., Brown, W. H., Frey, T., Karasu, N., Smith-Canter, L. L., & Strain, P. S. (2003). Evidence-based practices for young children with autism: Contributions for single-subject design research. Focus on Autism and Other Developmental Disorders, 18, 166–175.

    Article  Google Scholar 

  29. Petticrew, M., & Roberts, H. (2006). Systematic reviews in the social sciences: A practical guide. Malden, MA: Blackwell.

    Google Scholar 

  30. Reichow, B., Barton, E. E., Volkmar, F. R., & Cicchetti, D. V. (2007, May). The status of research on interventions for young children with autism spectrum disorders. Poster presented at the International Meeting for Autism Research, Seattle, WA.

  31. Rogers, S. J. (1998). Empirically supported comprehensive treatments for young children with autism. Journal of Clinical Child Psychology, 27, 168–179.

    PubMed  Article  Google Scholar 

  32. Shavelson, R. J., & Towne, L. (Eds.). (2002). Scientific research in education. Washington, DC: National Academy Press.

    Google Scholar 

  33. Urbaniak, G. C., & Plous, S. (2006). Research randomizer (retrieved July 15, 2005 from http://www.randomizer.org).

  34. Wampold, B. E., Lichtenberg, J. W., & Waehler, C. A. (2002). Principles of empirically supported interventions in counseling psychology. The Counseling Psychologist, 30, 197–217.

    Article  Google Scholar 

  35. Weisz, J. R., & Hawley, K. M. (1998). Finding, evaluating, refining, and applying empirically supported treatments for children and adolescents. Journal of Clinical Child Psychology, 27, 202–216.

    Google Scholar 

  36. Weisz, J. R., Hawley, K. M., Pilkonis, P. A., Woody, S. R., & Follette, W. C. (2000). Stressing the (other) three rs in the search for empirically supported treatments: Review procedures, research quality, relevance to practice, and the public interest. Clinical Psychology: Science and Practice, 7, 243–258.

    Article  Google Scholar 

Download references

Acknowledgments

The authors would like to thank Erin Barton and Peter Doehring for their assistance in the development of the reliability of the rubrics. This manuscript was prepared while Brian Reichow was completing an internship at the Yale Child Study Center.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Brian Reichow.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Reichow, B., Volkmar, F.R. & Cicchetti, D.V. Development of the Evaluative Method for Evaluating and Determining Evidence-Based Practices in Autism. J Autism Dev Disord 38, 1311–1319 (2008). https://doi.org/10.1007/s10803-007-0517-7

Download citation

Keywords

  • Autism
  • Evidence-based practice