Advertisement

Perspectives on Behavior Science

, Volume 42, Issue 1, pp 59–75 | Cite as

Replication Research, Publication Bias, and Applied Behavior Analysis

  • Matt TincaniEmail author
  • Jason Travers
Commentary

Abstract

The “replication crisis” describes recent difficulties in replicating studies in various scientific fields, most notably psychology. The available evidence primarily documents replication failures for group research designs. However, we argue that contingencies of publication bias that led to the “replication crisis” also operate on applied behavior analysis (ABA) researchers who use single-case research designs (SCRD). This bias strongly favors publication of SCRD studies that show strong experimental effect, and disfavors publication of studies that show less robust effect. The resulting research literature may unjustifiably inflate confidence about intervention effects, limit researchers’ ability to delineate intervention boundary conditions, and diminish the credibility of our science. To counter problems of publication bias in ABA, we recommend that journals that publish SCRD research establish journal standards for publication of noneffect studies; that our research community adopt open sharing of SCRD protocols and data; and that members of our community routinely publish systematic literature reviews that include gray (i.e., unpublished) research.

Keywords

Applied behavior analysis Replication Publication bias File drawer effect Single-case design Behavior science 

Notes

References

  1. Baer, D. M., Wolf, M. M., & Risley, T. R. (1968). Some current dimensions of applied behavior analysis. Journal of Applied Behavior Analysis, 1, 91–97.CrossRefGoogle Scholar
  2. Barton, E. E., Reichow, B., Schnitz, A., Smith, I. C., & Sherlock, D. (2015). A systematic review of sensory-based treatments for children with disabilities. Research in Developmental Disabilities, 37, 64–80.CrossRefGoogle Scholar
  3. Bloom, S. E., Iwata, B. A., Fritz, J. N., Roscoe, E. M., & Carreau, A. B. (2011). Classroom application of a trial-based functional analysis. Journal of Applied Behavior Analysis, 44, 19–31.CrossRefGoogle Scholar
  4. Bullock, C. E., Fisher, W. W., & Hagopian, L. P. (2017). Description and validation of a computerized behavioral data program: “BDataPro.”. The Behavior Analyst, 40, 275–285.  https://doi.org/10.1007/s40614-016-0079-0.CrossRefGoogle Scholar
  5. Carr, E. G., & McDowell, J. J. (1980). Social control of self-injurious behavior of organic etiology. Behavior Therapy, 11, 402–409.CrossRefGoogle Scholar
  6. Chambless, D. L., & Hollon, S. D. (1998). Defining empirically supported therapies. Journal of Consulting and Clinical Psychology, 66, 7–18.  https://doi.org/10.1037/0022-006X.66.1.7.CrossRefGoogle Scholar
  7. Cook, B. G., Collins, L. W., Cook, S. C., & Cook, L. (2016). A replication by any other name: A systematic review of replicative intervention studies. Remedial & Special Education, 37, 223–234.CrossRefGoogle Scholar
  8. Cooper, J. O., Heron, T. E., & Heward, W. L. (2007). Applied behavior analysis (2nd ed.). Upper Saddle River, NJ: Prentice Hall.Google Scholar
  9. Cooper, L. J., Wacker, D. P., Sasso, G. M., Reimers, T. M., & Donn, L. K. (1990). Using parents as therapists to evaluate appropriate behavior of their children: Application to a tertiary diagnostic clinic. Journal of Applied Behavior Analysis, 23, 285–296.CrossRefGoogle Scholar
  10. Council for Exceptional Children. (2014). Council for Exceptional Children standards for evidence-based practices in special education. Arlington, VA: Author.Google Scholar
  11. Cox, A. L., Gast, D. L., Luscre, D., & Ayres, K. M. (2009). The effects of weighted vests on appropriate in-seat behaviors of elementary-age students with autism and severe to profound intellectual disabilities. Focus on Autism & Other Developmental Disabilities, 24, 17–26.CrossRefGoogle Scholar
  12. Davis, T. N., Dacus, S., Strickland, E., Copeland, D., Chan, J. M., Blenden, K., et al. (2013). The effects of a weighted vest on aggressive and self-injurious behavior in a child with autism. Developmental Neurorehabilitation, 16, 210–215.CrossRefGoogle Scholar
  13. Denton, T. F., & Meindl, J. N. (2016). The effect of colored overlays on reading fluency in individuals with dyslexia. Behavior Analysis in Practice, 9, 191–198.CrossRefGoogle Scholar
  14. Deochand, N., Costello, M. S., & Deochand, M. E. (2018). Behavioral research with planaria. Perspectives on Behavior Science, 41, 447–464.  https://doi.org/10.1007/s40614-018-00176-w.CrossRefGoogle Scholar
  15. Dixon, M. R., Reed, D. D., Smith, T., Belisle, J., & Jackson, R. E. (2015). Research rankings of behavior analytic graduate training programs and their faculty. Behavior Analysis in Practice, 8, 7–15.CrossRefGoogle Scholar
  16. Doyen, S., Klein, O., Pichon, C. L., & Cleeremans, A. (2012). Behavioral priming: it's all in the mind, but whose mind? PloS ONE, 7, e29081.CrossRefGoogle Scholar
  17. Driessen, E., Hollon, S. D., Bockting, C. L., Cuijpers, P., & Turner, E. H. (2015). Does publication bias inflate the apparent efficacy of psychological treatment for major depressive disorder? A systematic review and meta-analysis of US National Institutes of Health-funded trials. PLoS ONE, 10, e0137864.CrossRefGoogle Scholar
  18. Earp, B. D., & Trafimow, D. (2015). Replication, falsification, and the crisis of confidence in social psychology. Frontiers in Psychology, 6, 621.CrossRefGoogle Scholar
  19. Eldevik, S., Hastings, R. P., Hughes, J. C., Jahr, E., Eikeseth, S., & Cross, S. (2009). Meta-analysis of early intensive behavioral intervention for children with autism. Journal of Clinical Child & Adolescent Psychology, 38, 439–450.CrossRefGoogle Scholar
  20. Evanschitzky, H., & Armstrong, J. S. (2010). Replications of forecasting research. International Journal of Forecasting, 26, 4–8.CrossRefGoogle Scholar
  21. Ferster, C. B., & Skinner, B. F. (1957). Schedules of reinforcement. Cambridge, MA: B. F. Skinner Foundation.CrossRefGoogle Scholar
  22. Foxx, R. M., & Mulick, J. A. (Eds.). (2016). Controversial therapies for autism and intellectual disabilities: Fad, fashion, and science in professional practice (2nd ed.). New York, NY: Routledge.Google Scholar
  23. Franco, A., Malhotra, N., & Simonovits, G. (2014). Publication bias in the social sciences: Unlocking the file drawer. Science, 345, 1502–1505.CrossRefGoogle Scholar
  24. Gage, N. A., Cook, B. G., & Reichow, B. (2017). Publication bias in special education meta-analyses. Exceptional Children, 83, 428–445.Google Scholar
  25. Ganz, J. B., Earles-Vollrath, T. L., Mason, R. A., Rispoli, M. J., Heath, A. K., & Parker, R. I. (2011). An aggregate study of single-case research involving aided AAC: Participant characteristics of individuals with autism spectrum disorders. Research in Autism Spectrum Disorders, 5, 1500–1509.CrossRefGoogle Scholar
  26. Gelman, A., & Fung, K. (2016). The power of the “Power Pose.” Slate Magazine. Retrieved January 20, 2019, from http://www.slate.com/articles/health_and_science/science/2016/01/amy_cuddy_s_power_pose_research_is_the_latest_example_of_scientific_overreach.html.
  27. Gilroy, S. P., Franck, C. T., & Hantula, D. A. (2017). The discounting model selector: Statistical software for delay discounting applications. Journal of the Experimental Analysis of Behavior, 107(3), 388–401.  https://doi.org/10.1002/jeab.257.CrossRefGoogle Scholar
  28. Gilroy, S. P., Kaplan, B. A., Reed, D. D., Koffarnus, M. N., & Hantula, D. A. (2018). The demand curve analyzer: Behavioral economic software for applied research. Journal of the Experimental Analysis of Behavior.  https://doi.org/10.1002/jeab.479.
  29. Hales, A. H., Wesselmann, E. D., & Hilgard, J. (2018). Improving psychological science through transparency and openness: An overview. Perspectives on Behavior Science.  https://doi.org/10.1007/s40614-018-00186-8.
  30. Hanley, G. P. (2017). Editor’s note. Journal of Applied Behavior Analysis, 50, 3–7.CrossRefGoogle Scholar
  31. Hantula, D. A. (2016a). Editorial: A very special issue. The Behavior Analyst, 39, 1–5.  https://doi.org/10.1007/s40614-016-0066-5.CrossRefGoogle Scholar
  32. Hantula, D. A. (2016b). Expanding the scope: Beyond the familiar and beyond the page. The Behavior Analyst, 39, 189–196.  https://doi.org/10.1007/s40614-016-0078-1.CrossRefGoogle Scholar
  33. Horner, R. H., Carr, E. G., Halle, J., McGee, G., Odom, S., & Wolery, M. (2005). The use of single-subject research to identify evidence-based practice in special education. Exceptional Children, 71, 165–179.Google Scholar
  34. Hubbard, R., & Armstrong, J. S. (1994). Replications and extensions in marketing: Rarely published but quite contrary. International Journal of Research in Marketing, 11, 233–248.CrossRefGoogle Scholar
  35. Ioannidis, J. P. (2005). Why most published research findings are false. PLoS Medicine, 2, e124.CrossRefGoogle Scholar
  36. Ioannidis, J. P. (2012). Why science is not necessarily self-correcting. Perspectives on Psychological Science, 7, 645–654.CrossRefGoogle Scholar
  37. Iwata, B. A., Dorsey, M. F., Slifer, K. J., Bauman, K. E., & Richman, G. S. (1982). Toward a functional analysis of self-injury. Analysis & Intervention in Developmental Disabilities, 2, 3–20.CrossRefGoogle Scholar
  38. Iwata, B. A., Dorsey, M. F., Slifer, K. J., Bauman, K. E., & Richman, G. S. (1994). Toward a functional analysis of self-injury. Journal of Applied Behavior Analysis, 27, 197–209.CrossRefGoogle Scholar
  39. Jamshidi, L., Heyvaert, M., Declercq, L., Fernández-Castilla, B., Ferron, J. M., Moeyaert, M., et al. (2017). Methodological quality of meta-analyses of single-case experimental studies. Research in Developmental Disabilities. Advance online publication.  https://doi.org/10.1016/j.ridd.2017.12.016.
  40. Jessel, J., Hanley, G. P., & Ghaemmaghami, M. (2016). Interview-informed synthesized contingency analyses: Thirty replications and reanalysis. Journal of Applied Behavior Analysis, 49, 576–595.CrossRefGoogle Scholar
  41. Kaplan, B. A., Gilroy, S. P., Reed, D. D., Koffarnus, M. N., & Hursh, S. R. (2018). The R package beezdemand: Behavioral economic easy demand. Perspectives on Behavior Science.  https://doi.org/10.1007/s40614-018-00187-7.
  42. Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality & Social Psychology Review, 2, 196–217.CrossRefGoogle Scholar
  43. Kittelman, A., Gion, C., Horner, R. H., Levin, J. R., & Kratochwill, T. R. (2018). Establishing journalistic standards for the publication of negative results. Remedial & Special Education., 39, 171–176.CrossRefGoogle Scholar
  44. Koegel, R. L., & Koegel, L. K. (2006). Pivotal response treatments for autism: Communication, social, and academic development. Baltimore, MD: Paul H. Brookes.Google Scholar
  45. Kratochwill, T. R., Hitchcock, J. H., Horner, R. H., Levin, J. R., Odom, S. L., Rindskopf, D. M., & Shadish, W. R. (2013). Single-case intervention research design standards. Remedial & Special Education, 34, 26–38.CrossRefGoogle Scholar
  46. Kratochwill, T. R., Levin, J. R., & Horner, R. H. (2018). Negative results: Conceptual and methodological dimensions in single-case intervention research. Remedial & Special Education, 39, 67–76.CrossRefGoogle Scholar
  47. Lane, J. (2010). Let’s make science metrics more useful. Nature, 464, 488–489.CrossRefGoogle Scholar
  48. Ledford, J. R., Barton, E. E., Hardy, J. K., Elam, K., Seabolt, J., Shanks, M., et al. (2016). What equivocal data from single case comparison studies reveal about evidence-based practices in early childhood special education. Journal of Early Intervention, 38, 79–91.CrossRefGoogle Scholar
  49. Lilienfeld, S. O. (2017). Psychology’s replication crisis and the grant culture: Righting the ship. Perspectives on Psychological Science, 12(4), 660–664.  https://doi.org/10.1177/1745691616687745.CrossRefGoogle Scholar
  50. Lonigan, C. J., Elbert, J. C., & Johnson, S. B. (1998). Empirically supported psychosocial interventions for children: An overview. Journal of Clinical Child Psychology, 27, 138–145.  https://doi.org/10.1207/s15374424jccp2702_1.CrossRefGoogle Scholar
  51. Losinski, M., Cook, K., Hirsch, S., & Sanders, S. (2017). The effects of deep pressure therapies and antecedent exercise on stereotypical behaviors of students with autism spectrum disorders. Behavioral Disorders, 42, 196–208.CrossRefGoogle Scholar
  52. Maggin, D. M., Talbott, E., Van Acker, E. Y., & Kumm, S. (2017). Quality indicators for systematic reviews in behavioral disorders. Behavioral Disorders, 42, 52–64.CrossRefGoogle Scholar
  53. Mahoney, M. J. (1977). Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognitive Therapy & Research, 1, 161–175.CrossRefGoogle Scholar
  54. Mahoney, M. J. (1994). Scientist as subject: The psychological imperative. Clinton Corners, NY: Percheron Press.Google Scholar
  55. Makel, M. C., & Plucker, J. A. (2014). Facts are more important than novelty: Replication in the education sciences. Educational Researcher, 43, 304–316.CrossRefGoogle Scholar
  56. Mitchell, M., Leachman, M., & Masterson, K. (2016). Funding down, tuition up: State Cuts to higher education threaten quality and affordability at public colleges. Washington, DC: Center on Budget & Policy Priorities Retrieved January 20, 2019, from http://www.cbpp.org/research/state-budget-and-tax/funding-down-tuition-up#_ftn33.Google Scholar
  57. National Autism Center. (2015). Findings and conclusions: National standards project, phase 2. Randolph, MA: Author.Google Scholar
  58. Normand, M. P., & Knoll, M. L. (2006). The effects of a stimulus-stimulus pairing procedure on the unprompted vocalizations of a young child diagnosed with autism. Analysis of Verbal Behavior, 22, 81–85.CrossRefGoogle Scholar
  59. Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7, 615–631.CrossRefGoogle Scholar
  60. Open Science Collaboration. (2012). An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspectives on Psychological Science, 7, 657–660.CrossRefGoogle Scholar
  61. Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349, aac4716.CrossRefGoogle Scholar
  62. Pashler, H., & Harris, C. R. (2012). Is the replicability crisis overblown? Three arguments examined. Perspectives on Psychological Science, 7, 531–536.CrossRefGoogle Scholar
  63. Pashler, H., & Wagenmakers, E. J. (2012). Editors’ introduction to the special section on replicability in psychological science: A crisis of confidence? Perspectives on Psychological Science, 7, 528–530.CrossRefGoogle Scholar
  64. Perone, M. (2018). How I learned to stop worrying and love replication failures. Perspectives on Behavior Science. Advance online publication.  https://doi.org/10.1007/s40614-018-0153-x.
  65. Pierce, W. D., & Cheyney, C. D. (2017). Behavior analysis and learning: A biobehavioral approach. New York, NY: Routledge.CrossRefGoogle Scholar
  66. Quigley, S. P., Peterson, L., Frieder, J. E., & Peterson, S. (2011). Effects of a weighted vest on problem behaviors during functional analyses in children with pervasive developmental disorders. Research in Autism Spectrum Disorders, 5, 529–538.CrossRefGoogle Scholar
  67. Reichow, B. (2012). Overview of meta-analyses on early intensive behavioral intervention for young children with autism spectrum disorders. Journal of Autism & Developmental Disorders, 42, 512–520.CrossRefGoogle Scholar
  68. Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86, 638.CrossRefGoogle Scholar
  69. Shillingsburg, M. A., Hollander, D. L., Yosick, R. N., Bowen, C., & Muskat, L. R. (2015). Stimulus-stimulus pairing to increase vocalizations in children with language delays: A review. Analysis of Verbal Behavior, 3, 215–235.CrossRefGoogle Scholar
  70. Shadish, W. R., Zelinsky, N. A., Vevea, J. L., & Kratochwill, T. R. (2016). A survey of publication practices of single-case design researchers when treatments have small or large effects. Journal of Applied Behavior Analysis, 49, 656–673.CrossRefGoogle Scholar
  71. Sham, E., & Smith, T. (2014). Publication bias in studies of an applied behavior-analytic intervention: An initial analysis. Journal of Applied Behavior Analysis, 47, 663–678.CrossRefGoogle Scholar
  72. Sidman, M. (1960). Tactics of scientific research. Cambridge, MA: Cambridge Center for Behavioral Studies.Google Scholar
  73. Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366.CrossRefGoogle Scholar
  74. Skinner, B. F. (1938). The behavior of organisms: an experimental analysis. Cambridge, MA: B. F. Skinner Foundation.Google Scholar
  75. Skinner, B. F. (1953). Science and human behavior. Cambridge, MA: B. F. Skinner Foundation.Google Scholar
  76. Slocum, T. A., Detrich, R., Wilczynski, S. M., Spencer, T. D., Lewis, T., & Wolfe, K. (2014). The evidence-based practice of applied behavior analysis. The Behavior Analyst, 37, 41–56.CrossRefGoogle Scholar
  77. Tincani, M., & De Mers, M. (2016). Meta-analysis of single-case research design studies on instructional pacing. Behavior Modification, 40, 799–824.CrossRefGoogle Scholar
  78. Tincani, M., & Travers, J. C. (2018). Publishing single-case experimental research studies that do not demonstrate experimental control. Remedial & Special Education, 39, 118–128.CrossRefGoogle Scholar
  79. Travers, J. C., Tincani, M. J., & Lang, R. (2014). Facilitated communication denies people with disabilities their voice. Research & Practice in Severe Disabilities, 39, 195–202.CrossRefGoogle Scholar
  80. UNESCO. (2018). Global open access portal. Retrieved January 20, 2019, from http://www.unesco.org/new/en/communication-and-information/portals-and-platforms/goap/open-science-movement
  81. Wong, C., Odom, S. L., Hume, K. A., Cox, A. W., Fettig, A., Kucharczyk, S., et al. (2015). Evidence-based practices for children, youth, and young adults with autism spectrum disorder: A comprehensive review. Journal of Autism & Developmental Disorders, 45, 1951–1966.CrossRefGoogle Scholar
  82. Yoon, S. Y., & Bennett, G. M. (2000). Effects of a stimulus-stimulus pairing procedure on conditioning vocal sounds as reinforcers. Analysis of Verbal Behavior, 1, 75–88.CrossRefGoogle Scholar
  83. Zane, T., Davis, C., & Rosswurm, M. (2014). The cost of fad treatments in autism. Journal of Early & Intensive Behavior Intervention, 5, 44–51.CrossRefGoogle Scholar

Copyright information

© Association for Behavior Analysis International 2019

Authors and Affiliations

  1. 1.Department of Teaching and LearningTemple UniversityPhiladelphiaUSA
  2. 2.Department of Special EducationUniversity of KansasLawrenceUSA

Personalised recommendations