Replication Research, Publication Bias, and Applied Behavior Analysis
The “replication crisis” describes recent difficulties in replicating studies in various scientific fields, most notably psychology. The available evidence primarily documents replication failures for group research designs. However, we argue that contingencies of publication bias that led to the “replication crisis” also operate on applied behavior analysis (ABA) researchers who use single-case research designs (SCRD). This bias strongly favors publication of SCRD studies that show strong experimental effect, and disfavors publication of studies that show less robust effect. The resulting research literature may unjustifiably inflate confidence about intervention effects, limit researchers’ ability to delineate intervention boundary conditions, and diminish the credibility of our science. To counter problems of publication bias in ABA, we recommend that journals that publish SCRD research establish journal standards for publication of noneffect studies; that our research community adopt open sharing of SCRD protocols and data; and that members of our community routinely publish systematic literature reviews that include gray (i.e., unpublished) research.
KeywordsApplied behavior analysis Replication Publication bias File drawer effect Single-case design Behavior science
- Cooper, J. O., Heron, T. E., & Heward, W. L. (2007). Applied behavior analysis (2nd ed.). Upper Saddle River, NJ: Prentice Hall.Google Scholar
- Council for Exceptional Children. (2014). Council for Exceptional Children standards for evidence-based practices in special education. Arlington, VA: Author.Google Scholar
- Driessen, E., Hollon, S. D., Bockting, C. L., Cuijpers, P., & Turner, E. H. (2015). Does publication bias inflate the apparent efficacy of psychological treatment for major depressive disorder? A systematic review and meta-analysis of US National Institutes of Health-funded trials. PLoS ONE, 10, e0137864.CrossRefGoogle Scholar
- Foxx, R. M., & Mulick, J. A. (Eds.). (2016). Controversial therapies for autism and intellectual disabilities: Fad, fashion, and science in professional practice (2nd ed.). New York, NY: Routledge.Google Scholar
- Gage, N. A., Cook, B. G., & Reichow, B. (2017). Publication bias in special education meta-analyses. Exceptional Children, 83, 428–445.Google Scholar
- Ganz, J. B., Earles-Vollrath, T. L., Mason, R. A., Rispoli, M. J., Heath, A. K., & Parker, R. I. (2011). An aggregate study of single-case research involving aided AAC: Participant characteristics of individuals with autism spectrum disorders. Research in Autism Spectrum Disorders, 5, 1500–1509.CrossRefGoogle Scholar
- Gelman, A., & Fung, K. (2016). The power of the “Power Pose.” Slate Magazine. Retrieved January 20, 2019, from http://www.slate.com/articles/health_and_science/science/2016/01/amy_cuddy_s_power_pose_research_is_the_latest_example_of_scientific_overreach.html.
- Gilroy, S. P., Kaplan, B. A., Reed, D. D., Koffarnus, M. N., & Hantula, D. A. (2018). The demand curve analyzer: Behavioral economic software for applied research. Journal of the Experimental Analysis of Behavior. https://doi.org/10.1002/jeab.479.
- Hales, A. H., Wesselmann, E. D., & Hilgard, J. (2018). Improving psychological science through transparency and openness: An overview. Perspectives on Behavior Science. https://doi.org/10.1007/s40614-018-00186-8.
- Horner, R. H., Carr, E. G., Halle, J., McGee, G., Odom, S., & Wolery, M. (2005). The use of single-subject research to identify evidence-based practice in special education. Exceptional Children, 71, 165–179.Google Scholar
- Jamshidi, L., Heyvaert, M., Declercq, L., Fernández-Castilla, B., Ferron, J. M., Moeyaert, M., et al. (2017). Methodological quality of meta-analyses of single-case experimental studies. Research in Developmental Disabilities. Advance online publication. https://doi.org/10.1016/j.ridd.2017.12.016.
- Kaplan, B. A., Gilroy, S. P., Reed, D. D., Koffarnus, M. N., & Hursh, S. R. (2018). The R package beezdemand: Behavioral economic easy demand. Perspectives on Behavior Science. https://doi.org/10.1007/s40614-018-00187-7.
- Koegel, R. L., & Koegel, L. K. (2006). Pivotal response treatments for autism: Communication, social, and academic development. Baltimore, MD: Paul H. Brookes.Google Scholar
- Mahoney, M. J. (1994). Scientist as subject: The psychological imperative. Clinton Corners, NY: Percheron Press.Google Scholar
- Mitchell, M., Leachman, M., & Masterson, K. (2016). Funding down, tuition up: State Cuts to higher education threaten quality and affordability at public colleges. Washington, DC: Center on Budget & Policy Priorities Retrieved January 20, 2019, from http://www.cbpp.org/research/state-budget-and-tax/funding-down-tuition-up#_ftn33.Google Scholar
- National Autism Center. (2015). Findings and conclusions: National standards project, phase 2. Randolph, MA: Author.Google Scholar
- Perone, M. (2018). How I learned to stop worrying and love replication failures. Perspectives on Behavior Science. Advance online publication. https://doi.org/10.1007/s40614-018-0153-x.
- Sidman, M. (1960). Tactics of scientific research. Cambridge, MA: Cambridge Center for Behavioral Studies.Google Scholar
- Skinner, B. F. (1938). The behavior of organisms: an experimental analysis. Cambridge, MA: B. F. Skinner Foundation.Google Scholar
- Skinner, B. F. (1953). Science and human behavior. Cambridge, MA: B. F. Skinner Foundation.Google Scholar
- UNESCO. (2018). Global open access portal. Retrieved January 20, 2019, from http://www.unesco.org/new/en/communication-and-information/portals-and-platforms/goap/open-science-movement