Improving Psychological Science through Transparency and Openness: An Overview
- 286 Downloads
The ability to independently verify and replicate observations made by other researchers is a hallmark of science. In this article, we provide an overview of recent discussions concerning replicability and best practices in mainstream psychology with an emphasis on the practical benefists to both researchers and the field as a whole. We first review challenges individual researchers face in producing research that is both publishable and reliable. We then suggest methods for producing more accurate research claims, such as transparently disclosing how results were obtained and analyzed, preregistering analysis plans, and publicly posting original data and materials. We also discuss ongoing changes at the institutional level to incentivize stronger research. These include officially recognizing open science practices at the journal level, disconnecting the publication decision from the results of a study, training students to conduct replications, and publishing replications. We conclude that these open science practices afford exciting low-cost opportunities to improve the quality of psychological science.
KeywordsReplication Reproducibility Preregistration Meta-analysis
We thank Thomas Critchfield for valuable comments on a draft of this article.
- Abelson, R. P. (1995). Statistics as principled argument. Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar
- Baron, A., & Perone, M. (1999). Experimental design and analysis in the laboratory study of human operant behavior. In K. Lattal & M. Perone (Eds.), The handbook of research methods in human operant behavior (pp. 45–91). Boston, MA: Springer.Google Scholar
- Branch, M. (2018). The "reproducibility crisis”: Might the methods used frequently in behavior-analysis research help? Perspectives on Behavior Science. doi: https://doi.org/10.1007/s40614-018-0158-5
- Campbell, L. (2016, September 27). Campbell Lab: OSF Research Milestones. Retrieved from osf.io/jrd8f
- Center for Open Science (2018). Registered reports: Peer review before results are known to align scientific values and practices. https://cos.io/rr/
- Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
- Cook, F. L. (2018, March 9). Dear colleague letter: Achieving new insights through replicability and reproducibility. National Science Foundation. Retrieved from https://www.nsf.gov/pubs/2018/nsf18053/nsf18053.jsp
- Davison, M. (1999). Statistical inference in behavior analysis: Having my cake and eating it? Behavior Analyst, 22, 99–103. https://doi.org/10.1007/BF03391986.
- Eastwick, P. (2018, January 9). Two lessons from a registered report. Retrieved from http://pauleastwick.blogspot.com/2018/01/two-lessons-from-registered-report.html
- Faith, M. S., Allison, D. B., & Gorman, B. S. (1996). Meta-analysis of single-case research. In R. D. Franklin, D. B. Allison, & B. S. Gorman (Eds.), Design and analysis of single-case research (pp. 245–277). Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
- Finkel, E. J., Eastwick, P. W., & Reis, H. T. (2015). Best research practices in psychology: Illustrating epistemological and pragmatic considerations with the case of relationship science. Journal of Personality and Social Psychology, 108, 275–297. https://doi.org/10.1037/pspi0000007.CrossRefPubMedGoogle Scholar
- Fiske, S. T. (2016). A call to change science’s culture of shaming. APS Observer. Retrieved from https://www.psychologicalscience.org/observer/a-call-to-change-sciences-culture-of-shaming.
- Funder, D. C., Levine, J. M., Mackie, D. M., Morf, C. C., Sansone, C., Vazire, S., & West, S. G. (2014). Improving the dependability of research in personality and social psychology: Recommendations for research and educational practice. Personality & Social Psychology Review, 18, 3–12. https://doi.org/10.1177/1088868313507536.CrossRefGoogle Scholar
- Hennes, E. P., & Lane, S. P. (2017, April). Power to the people: Simulation methods for conducting power analysis for any model. In Workshop presented at the 89th Annual Meeting of the Midwestern Psychological Association. Chicago: IL.Google Scholar
- John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth-telling. Psychological Science, 23, 524–532. https://doi.org/10.1177/0956797611430953.
- Kanyongo, G. Y., Brook, G. P., Kyei-Blankson, L., & Gocmen, G. (2007). Reliability and statistical power: How measurement fallibility affects power and required sample sizes for several parametric and nonparametric statistics. Journal of Modern Applied Statistical Methods, 6, 81–90. https://doi.org/10.22237/jmasm/1177992480.CrossRefGoogle Scholar
- Kidwell, M. C., Lazarević, L. B., Baranski, E., Hardwicke, T. E., Piechowski, S., Falkenberg, L.-S., et al. (2016). Badges to acknowledge open practices: A simple, low-cost, effective method for increasing transparency. PLoS Biology, 14, e1002456. https://doi.org/10.1371/journal.pbio.1002456.CrossRefPubMedPubMedCentralGoogle Scholar
- Kirkpatrick, K., Marshall, A. T., Steele, C. C., & Peterson, J. R. (2018). Resurrecting the individual in behavioral analysis: Using mixed effects models to address nonsystematic discounting data. Behavior Analysis: Research & Practice, 18, 219–238. https://doi.org/10.1037/bar0000103.CrossRefGoogle Scholar
- Kyonka, E. G. E. (2018). Tutorial: Small-N power analysis. Perspectives on Behavior Science. https://doi.org/10.1007/s40614-018-0167-4.
- Kochari, A., & Ostarek, M. (2018, January 17). Introducing a replication-first rule for PhD projects (BBS commentary on Zwaan et al., Making replication mainstream). Retrieved from psyarxiv.com/6yv45
- Mayer, H. (Producer/Director) & Norris, M. (Writer). (1970). The social animal [Motion picture]. (Available from Indiana University Audio-Visual Center, Bloomington, Indiana).Google Scholar
- Mellor, D. T., Esposito, J., DeHaven, A. C., & Stodden, V. (2018, October 24). Resources. Open Science Foundation. Retrieved from osf.io/kgnvaGoogle Scholar
- Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2017, August 24). Preregistration Revolution. Retrieved from osf.io/2dxu5
- Patil, P., Peng, R. D., & Leek, J. T. (2016). What should researchers expect when they replicate studies? A statistical view on replicability in psychological science. Perspectives on Psychological Science, 11, 539–544. https://doi.org/10.1177/1745691616646366.CrossRefPubMedPubMedCentralGoogle Scholar
- Perone, M. (2018). How I learned to stop worrying and love replication failures. Perspectives on Behavior Science. https://doi.org/10.1007/s40614-018-0153-x.
- Poincaré, H. (1952). Science and method. London, UK: Thomas Nelson (Original work published 1914).Google Scholar
- Rosenthal, R. (1984). Meta-analytic procedures for social research. (Applied social research methods series, Vol. 6). Newbury Park, CA: Sage.Google Scholar
- Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366. https://doi.org/10.1177/0956797611417632.
- Simonsohn, U. (2013). Just post it: The lesson from two cases of fabricated data detected by statistics alone. Psychological Science, 24, 1875–1888. https://doi.org/10.1177/0956797613480366.
- Vasishth, S., & Gelman, A. (2017). The statistical significance filter leads to overconfident expectations of replicability. arXive. Retrieved April 27, 2018, from https://arxiv.org/abs/1702.00556
- Wagenmakers, E., & Dutil, G. (2016). Seven selfish reasons for preregistration. Observer, 29, 13–14.Google Scholar
- Wang, Y. A., Sparks, J., Gonzales, J. E., Hess, Y. D., & Ledgerwood, A. (2017). Using independent covariates in experimental designs: Quantifying the trade-off between power boost and Type I error inflation. Journal of Experimental Social Psychology, 72, 118–124. https://doi.org/10.1016/j.jesp.2017.04.011.CrossRefGoogle Scholar
- Wesselmann, E., Wirth, J. H., & Grahe, J. E. (2018, February 27). “Sources of Ostracism” Hub. https://doi.org/10.17605/OSF.IO/ENV5W
- Zwaan, R. A., Etz, A., Lucas, R. E., & Donnellan, M. B. (2017). Making replication mainstream. Behavioral & Brain Sciences. Advance online publication, 41. https://doi.org/10.1017/S0140525X17001972.