Skip to main content

Advertisement

SpringerLink
Log in

Study Preregistration: An Evaluation of a Method for Transparent Reporting

  • Original Paper
  • Published:
Journal of Business and Psychology Aims and scope Submit manuscript

Abstract

Study preregistration promotes transparency in scientific research by making a clear distinction between a priori and post hoc procedures or analyses. Management and applied psychology have not embraced preregistration in the way other closely related social science fields have. There may be concerns that preregistration does not add value and prevents exploratory data analyses. Using a mixed-method approach, in Study 1, we compared published preregistered samples against published non-preregistered samples. We found that preregistration effectively facilitated more transparent reporting based on criteria (i.e., confirmed hypotheses and a priori analysis plans). Moreover, consistent with concerns that the published literature contains elevated type I error rates, preregistered samples had fewer statistically significant results (48%) than non-preregistered samples (66%). To learn about the perceived advantages, disadvantages, and misconceptions of study preregistration, in Study 2, we surveyed authors of preregistered studies and authors who had never preregistered a study. Participants in both samples had positive inclinations towards preregistration yet expressed concerns about the process. We conclude with a review of best practices for management and applied psychology stakeholders.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. We also originally excluded samples at the team/organizational level, as the expertise of our author team is more at the individual level. However, as one of our reviewers rightly noted, samples at the team/organization level are central to the management and applied psychology fields. As a result, we went back and coded the two samples that we initially excluded for being at the team/organizational level and included them in our analyses/results.

  2. We would like to thank an anonymous reviewer for raising this issue.

References

  • Agnoli, F., Wicherts, J. M., Veldkamp, C. L., Albiero, P., & Cubelli, R. (2017). Questionable research practices among Italian research psychologists. PLoS One, 12(3), e0172792.

    Article  Google Scholar 

  • Aguinis, H., Ramani, R. S., & Alabduljader, N. (2018). What you see is what you get? Enhancing methodological transparency in management research. Academy of Management Annals, 12(1), 83–110.

    Article  Google Scholar 

  • Anderson, M., & Magruder, J. (2017). Split-sample strategies for avoiding false discoveries (no. w23544). https://doi.org/10.3386/w23544.

  • Anderson, M. S., Martinson, B. C., & De Vries, R. (2007). Normative dissonance in science: Results from a national survey of US scientists. Journal of Empirical Research on Human Research Ethics, 2(4), 3–14.

    Article  Google Scholar 

  • Antonakis, J. (2017). On doing better science: From thrill of discovery to policy implications. The Leadership Quarterly, 28(1), 5–21.

    Article  Google Scholar 

  • Banks, G. C., O’Boyle Jr, E. H., Pollack, J. M., White, C. D., Batchelor, J. H., Whelpley, C. E., Abston, K. A., Bennett., A. A, Adkins, C. L. (2016a). Questions about questionable research practices in the field of management: A guest commentary. Journal of Management, 42(1): 5–20.

  • Banks, G. C., Rogelberg, S. G., Woznyj, H. M., Landis, R. S., & Rupp, D. E. (2016b). Evidence on questionable research practices: The good, the bad, and the ugly. Journal of Business and Psychology, 31, 323–338.

    Article  Google Scholar 

  • Banks, G. C., Field, J. G., Oswald, F. L., O’Boyle, E. H., Landis, R. S., Rupp, D. E., & Rogelberg, S. G. (2018). Answers to 18 questions about Open Science practices. Journal of Business and Psychology, 1–14.

  • Bosco, F. A., Aguinis, H., Field, J. G., Pierce, C. A., & Dalton, D. R. (2016). HARKing’s threat to organizational research: Evidence from primary and meta-analytic sources. Personnel Psychology, 69, 709–750.

    Article  Google Scholar 

  • Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101.

    Article  Google Scholar 

  • Butler, N., Delaney, H., & Spoelstra, S. (2017). The gray zone: Questionable research practices in the business school. Academy of Management Learning & Education, 16(1), 94–109.

    Article  Google Scholar 

  • Choi, M. (2011). Employees' attitudes toward organizational change: A literature review. Human Resource Management, 50(4), 479–500.

    Article  Google Scholar 

  • Cunliffe, A. L., & Alcadipani, R. (2016). The politics of access in fieldwork: Immersion, backstage dramas, and deception. Organizational Research Methods, 19(4), 535–561.

    Article  Google Scholar 

  • Dechartres, A., Ravaud, P., Atal, I., Riveros, C., & Boutron, I. (2016). Association between trial registration and treatment effect estimates: A meta-epidemiological study. BMC Medicine, 14(1), 100.

    Article  Google Scholar 

  • Dwork, C., Feldman, V., Hardt, M., Pitassi, T., Reingold, O., & Roth, A. (2015). The reusable holdout: Preserving validity in adaptive data analysis. Science, 349(6248), 636–638.

    Article  Google Scholar 

  • Ebersole, C. R., Axt, J. R., & Nosek, B. A. (2016). Scientists’ reputations are based on getting it right, not being right. PLoS Biology, 14(5), e1002460.

    Article  Google Scholar 

  • Emerson, G. B., Warme, W. J., Wolf, F. M., Heckman, J. D., Brand, R. A., & Leopold, S. S. (2010). Testing for the presence of positive-outcome bias in peer review: A randomized controlled trial. Archives of Internal Medicine, 170(21), 1934–1939.

    Article  Google Scholar 

  • Fafchamps, M., & Labonne, J. (2016). Using split samples to improve inference about causal effects (no. w21842). https://doi.org/10.3386/w21842.

  • Fanelli, D. (2012). Negative results are disappearing from most disciplines and countries. Scientometrics, 90(3), 891–904.

    Article  Google Scholar 

  • Fleiss, J. L. (1981). Statistical methods for rates and proportions. 2nd ed. Wiley.

  • Fraser, H., Parker, T., Nakagawa, S., Barnett, A., & Fidler, F. (2018). Questionable research practices in ecology and evolution. PLoS One, 13(7), e0200303.

    Article  Google Scholar 

  • Goldacre, B., Drysdale, H., Dale, A., Milosevic, I., Slade, E., Hartley, P., Marston, C., Powell-Smith, A., Heneghan, C., & Mahtani, K. R. (2019a). COMPare: A prospective cohort study correcting and monitoring 58 misreported trials in real time. Trials, 20(1), 118.

    Article  Google Scholar 

  • Goldacre, B., Drysdale, H., Marston, C., Mahtani, K. R., Dale, A., Milosevic, I., Slade, E., Hartley, P., & Heneghan, C. (2019b). COMPare: Qualitative analysis of researchers’ responses to critical correspondence on a cohort of 58 misreported trials. Trials, 20(1), 124.

    Article  Google Scholar 

  • Guest, G., MacQueen, K. M., & Namey, E. E. (2012). Validity and reliability (credibility and dependability) in qualitative research and data analysis. Applied Thematic Analysis: 79–106. Sage Publications.

  • Haven, T. L., & Van Grootel, D. L. (2019). Preregistering qualitative research. Accountability in Research, 26(3), 229–244.

    Article  Google Scholar 

  • Hollenbeck, J. R., & Wright, P. M. (2016). Harking, sharking, and tharking: Making the case for post hoc analysis of scientific data. Journal of Management, 43, 5–18.

    Article  Google Scholar 

  • Hsieh, H. F., & Shannon, S. E. (2005). Three approaches to qualitative content analysis. Qualitative Health Research, 15(9), 1277–1288.

    Article  Google Scholar 

  • Hunt, S. T. (2018). If robust science is relevant science, then make IO psychology research more relevant: Thoughts from a practitioner point of view. Industrial and Organizational Psychology, 11(1), 65–70.

    Article  Google Scholar 

  • John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524–532.

    Article  Google Scholar 

  • Kaplan, R. M., & Irvin, V. L. (2015). Likelihood of null effects of large NHLBI clinical trials has increased over time. PLoS One, 10(8), e0132382.

    Article  Google Scholar 

  • Kepes, S., & McDaniel, M. A. (2013). How trustworthy is the scientific literature in I-O psychology? Industrial and Organizational Psychology: Perspectives on Science and Practice, 6(3), 252–268.

    Article  Google Scholar 

  • Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2, 196–217.

    Article  Google Scholar 

  • Koenig, W. D. (2017). Striving for science that is transparent, credible—and enjoyable: A comment on Ihle et al. Behavioral Ecology, 28(2), 358–358.

    Article  Google Scholar 

  • Köhler, T., Gonzàlez-Morales, M. G., Banks, G.C., O’Boyle, E., Allen, J., Sinha, R., Woo, S. E., Gulick, L. (in press). Supporting robust, rigorous, and reliable reviewing as the cornerstone of our profession: Introducing a competency model for peer review. Industrial and Organizational Psychology: Perspectives on Science and Practice.

  • Leavitt, K. (2013). Publication bias might make us untrustworthy, but the solutions may be worse. Industrial and Organizational Psychology, 6(3), 290–295.

    Article  Google Scholar 

  • Makel, M. C., Hodges, J., Cook, B. G., & Plucker, J. (2019, October 31). Questionable and open research practices in education research. https://doi.org/10.35542/osf.io/f7srb.

  • McAbee, S. T., Grubbs, J. B., & Zickar, M. J. (2018). Open science is robust science. Industrial and Organizational Psychology, 11(1), 54–61.

    Article  Google Scholar 

  • Mellor, D. T., & Nosek, B. A. (2018). Easy preregistration will benefit any research. Nature Human Behaviour, 1.

  • Morey. (2019). You must tug that thread: Why treating preregistration as a gold standard might incentivize poor behavior. Psychonomic Society. https://featuredcontent.psychonomic.org/you-must-tug-that-thread-why-treating-preregistration-as-a-gold-standard-might-incentivize-poor-behavior/.

  • Murphy, K. R., & Aguinis, H. (2017). HARKing: How badly can cherry-picking and question trolling produce bias in published results? Journal of Business and Psychology: 1–17.

  • Murray, M. M., & Antonakis, J. (2019). An Introductory Guide to Organizational Neuroscience: SAGE Publications Sage CA, An Introductory Guide to Organizational Neuroscience.

  • Navarro. (2019). Prediction, pre-specification and transparency. Psychonomic Society. https://featuredcontent.psychonomic.org/prediction-pre-specification-and-transparency/

  • Nosek, B. A., & Bar-Anan, Y. (2012). Scientific utopia: I. Opening scientific communication. Psychological Inquiry, 23, 217–243.

    Article  Google Scholar 

  • Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7, 615–631.

    Article  Google Scholar 

  • Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Chistensen, G., Contestabile, M., Dafoe, A., Eich, E., Freese, J., Glennerster, R., Goroff, D., Green, D. P., Hesse, B., Humphreys, M., et al. (2015). Promoting an open research culture: Author guidelines for journals to promote transparency, openness, and reproducibility. Science, 348, 1422–1425.

    Article  Google Scholar 

  • Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences: 201708274.

  • Nosek, B. A., Beck, E. D., Campbell, L., Flake, J. K., Hardwicke, T. E., Mellor, D. T., van't Veer, A. E, & Vazire, S. (2019). Preregistration is hard, and worthwhile. Trends in Cognitive Sciences, 23(10), 815–818.

  • O’Boyle, E., Banks, G. C., Carter, K., Walter, S., & Yuan, Z. (2019). A 20-year review of outcome reporting bias in moderated multiple regression. Journal of Business and Psychology, 34(1), 1–19.

    Article  Google Scholar 

  • Oberauer. (2019). Preregistration of a forking path—what does it add to the garden of evidence? Psychonomic Society. https://featuredcontent.psychonomic.org/preregistration-of-a-forking-path-what-does-it-add-to-the-garden-of-evidence/.

  • O'Boyle, E. H., Banks, G. C., & Gonzalez-Mule, E. (2017). The chrysalis effect: How ugly initial results metamorphosize into beautiful articles. Journal of Management, 43, 400–425.

    Google Scholar 

  • Papageorgiou, S. N., Xavier, G. M., Cobourne, M. T., & Eliades, T. (2018). Registered trials report less beneficial treatment effects than unregistered ones: A meta-epidemiological study in orthodontics. Journal of Clinical Epidemiology, 100, 44–52.

    Article  Google Scholar 

  • Rogelberg, S. G., & Stanton, J. M. (2007). Introduction: Understanding and dealing with organizational survey nonresponse. Organizational Research Methods, 10(2), 195–209.

    Article  Google Scholar 

  • Schwarzkopf, S. (2015, March 21). Some questions about registered reports. https://neuroneurotic.net/2015/03/21/some-questions-about-registered-reports/.

  • Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366.

    Article  Google Scholar 

  • Simonsohn, U., Simmons, J. P., & Nelson, L. D. (2015). Specification curve: Descriptive and inferential statistics on all reasonable specifications. Available at SSRN 2694998.

  • Tonidandel, S., King, E. B., & Cortina, J. M. (2018). Big data methods: Leveraging modern data analytic techniques to build organizational science. Organizational Research Methods, 21(3), 525–547.

    Article  Google Scholar 

  • van Rooij. (2019). Psychological science needs theory development before preregistration. Psychonomic Society. https://featuredcontent.psychonomic.org/psychological-science-needs-theory-development-before-preregistration/.

  • Veldkamp, C. L. S., Bakker, M., van Assen, M. A. L. M., Crompvoets, E. A. V., Ong, H. H., Soderberg, C. K., Mellor, D., Nosek, B. A., & Wicherts, J. M. (in progress)(n.d.). Restriction of opportunistic use of researcher degrees of freedom in pre-registrations on the Open Science Framework.

  • Wagenmakers, E.-J., Wetzels, R., Borsboom, D., van der Maas, H. L., & Kievit, R. A. (2012). An agenda for purely confirmatory research. Perspectives on Psychological Science, 7(6), 632–638.

    Article  Google Scholar 

Download references

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Author information

Authors and Affiliations

  1. Organizational Science, University of North Carolina at Charlotte, 9201 University City Blvd., Charlotte, NC, 28223, USA

    Allison A. Toth, Ashleigh Dickson & Jared Borns

  2. Belk College of Business, University of North Carolina at Charlotte, Charlotte, NC, USA

    George C. Banks

  3. Center for Open Science, Charlottesville, VA, USA

    David Mellor & Alex DeHaven

  4. Indiana University, Bloomington, IN, USA

    Ernest H. O’Boyle

  5. University of South Carolina Upstate, Spartanburg, SC, USA

    Daniel J. Davis

  6. College of Liberal Arts & Sciences, University of North Carolina at Charlotte, Charlotte, NC, USA

    Jaime Bochantin

Authors
  1. Allison A. Toth
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. George C. Banks
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. David Mellor
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ernest H. O’Boyle
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ashleigh Dickson
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Daniel J. Davis
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Alex DeHaven
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Jaime Bochantin
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Jared Borns
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Allison A. Toth.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

We dedicate this article to the memory of Jared Borns whose seminal work on this project provided an important foundation for the final submission.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Toth, A.A., Banks, G.C., Mellor, D. et al. Study Preregistration: An Evaluation of a Method for Transparent Reporting. J Bus Psychol 36, 553–571 (2021). https://doi.org/10.1007/s10869-020-09695-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10869-020-09695-3

Keywords

Search

Navigation