How I Learned to Stop Worrying and Love Replication Failures
- 250 Downloads
Worries about the reproducibility of experiments in the behavioral and social sciences arise from evidence that many published reports contain false positive results. Misunderstanding and misuse of statistical procedures are key sources of false positives. In behavior analysis, however, statistical procedures have not been used much. Instead, the investigator must show that the behavior of an individual is consistent over time within an experimental condition, that the behavior changes systematically across conditions, and that these changes can be reproduced – and then the whole pattern must be shown in additional individuals. These high standards of within- and between-subject replication protect behavior analysis from the publication of false positive findings. When a properly designed and executed experiment fails to replicate a previously published finding, the failure exposes flaws in our understanding of the phenomenon under study – perhaps in recognizing the boundary conditions of the phenomenon, identifying the relevant variables, or bringing the variables under sufficient control. We must accept the contradictory findings as valid and pursue an experimental analysis of the possible reasons. In this way, we resolve the contradiction and advance our science. To illustrate, two research programs are described, each initiated because of a replication failure.
KeywordsReplication Replication failure Open Science Collaboration Statistical significance Fixed-ratio pausing Conditioned reinforcement
Compliance with Ethical Standards
Conflict of Interest
The author declares that he has no conflict of interest.
This paper does not report original empirical research, but rather reviews previous work, almost all of it published in peer-reviewed journals. The research in which I directly participated was reviewed and approved by the responsible committee (IACUC or IRB).
- Baker, M. (2015). Over half of psychology studies fail reproducibility test. Retrieved from Nature News, https://www.nature.com/news/over-half-of-psychology-studies-fail-reproducibility-test-1.18248.
- Barlow, D. H., Hayes, S. C., & Nelson, R. O. (1984). The scientist practitioner: Research and accountability in clinical and educational settings. New York: Pergamon Press.Google Scholar
- Barlow, D. H., Nock, M. K., & Hersen, M. (2009). Single case experimental designs: Strategies for studying behavior change (3rd ed.). New York: Pearson i9780205474554.Google Scholar
- Castillo, M. I., Clark, D. R., Schaller, E. A., Donaldson, J. M., DeLeon, I. G., & Kahng, S. (2018). Descriptive assessment of problem behavior during transitions of children with intellectual and developmental disabilities. Journal of Applied Behavior Analysis, 51, 99–117. https://doi.org/10.1002/jaba.430.CrossRefGoogle Scholar
- Fantino, E. (1977). Conditioned reinforcement: Choice and information. In W. K. Honig & J. E. R. Staddon (Eds.), Handbook of operant behavior (pp. 313–339). Englewood Cliffs: Prentice-Hall.Google Scholar
- Johnston, J. M., & Pennypacker, H. S. (2009). Strategies and tactics of behavioral research (3rd ed.). New York: Routledge.Google Scholar
- Kline, R. B. (2004). Beyond significance testing. Washington, DC: American Psychological Association.Google Scholar
- Kollins, S. H., Newland, M. C., &. Critchfield, T. S. (1997). Human sensitivity to reinforcement in operant choice: How much do consequences matter? Psychonomic Bulletin & Review, 4, 208–220. Erratum: Psychonomic Bulletin & Review, 4, 431. https://doi.org/10.3758/BF03209395
- Kubrick, S. (1964). Dr. Strangelove or: How I learned to stop worrying and love the bomb [motion picture]. United States: Columbia Pictures.Google Scholar
- Morrison, D. E., & Henkel, R. E. (Eds.). (1970). The significance test controversy. Chicago: Aldine.Google Scholar
- Nelson, L. D., Simmons, J., & Simonsohn, U. (2018). Psychology’s renaissance. Annual Review of Psychology, 69, 17.1–17.24. https://doi.org/10.1146/annurev-psych-122216-011836.CrossRefGoogle Scholar
- Perone, M. (1991). Experimental design in the analysis of free-operant behavior. In I. H. Iversen & K. A. Lattal (Eds.), Techniques in the behavioral and neural sciences: Vol. 6. Experimental analysis of behavior, part 1 (pp. 135–171). Amsterdam: Elsevier.Google Scholar
- Sidman, M. (1960). Tactics of scientific research. New York: Basic Books.Google Scholar
- Skinner, B. F. (1938). The behavior of organisms. New York: Appleton-Century-Crofts.Google Scholar
- Weinberg, T. & Cohen, J. (2014). Baseball’s been very, very good to me: The Minnie Miñoso story [Video file]. Retrieved from http://www.pbs.org/video/wttw-documentaries-baseballs-been-very-very-good-me-minnie-minoso-story/.