Skip to main content
SpringerLink
Log in

A practical guide to Registered Reports for economists

  • Methodology Paper
  • Published:
Journal of the Economic Science Association Aims and scope Submit manuscript

Abstract

The current publication system in economics has encouraged the inflation of positive results in empirical papers. Registered Reports, also called Pre-Results Reviews, are a new submission format for empirical work that takes pre-registration one step further. In Registered Reports, researchers write their papers before running the study and commit to a detailed data collection process and analysis plan. After a first-stage review, a journal can give an In-Principle-Acceptance guaranteeing that the paper will be published if the authors carry out their data collection and analysis as pre-specified. We here propose a practical guide to Registered Reports for empirical economists. We illustrate the major problems that Registered Reports address (p-hacking, HARKing, forking, and publication bias), and present practical guidelines on how to write and review Registered Reports (e.g., the data-analysis plan, power analysis, and correction for multiple-hypothesis testing), with R and STATA codes. We provide specific examples for experimental economics, and show how research design can be improved to maximize statistical power. Last, we discuss some tools that authors, editors, and referees can use to evaluate Registered Reports (checklist, study-design table, and quality assessment).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Notes

  1. Franco et al. (2014) analyze the results of survey-based experiments funded by a NSF-sponspored program and run on nationally representative samples between 2002 and 2012. They compare the results of the experiments that got eventually published with the results of the experiments that remained unpublished.

  2. See for instance John et al. (2012), Agnoli et al. (2017), Fanelli (2009), Fiedler and Schwarz (2016), LeBel et al. (2013), O’Boyle et al. (2017).

  3. This effect is worsened by non-replicable analyses being cited more than replicable analyses (Serra-Garcia and Gneezy, 2021) and by the fact that a failure to replicate a work does not lead to fewer citations (Schafmeister, 2021). Note that, in economics, Camerer et al. (2016) find a replication rate of 61% in a sample of 18 experiments published in the American Economic Review and the Quarterly Journal of Economics, although the low replication rate might result from imperfect replication conditions (Chen et al., 2021).

  4. This only applies to field experiments. Laboratory experiments have no pre-registration requirements for the moment.

  5. According to the Center for Open Science https://www.cos.io/initiatives/registered-reports.

  6. The current publication system might lead some researchers to avoid high-risk, high-reward protocols that might however be beneficial for science. The pressure for positive results might indeed make risk-averse researchers invest in several small-scale experiments rather than in a large-scale high-risk intervention to ensure that they have at least some positive results to publish. In-principle acceptance could help mitigate this issue by reducing the publication risk associated with high-risk studies.

  7. Scheel et al. (2021) select papers in psychology that include hypothesis testing, and find that 96% report a positive significant result for their first hypothesis, as compared to a figure of only 44% in RRs.

  8. We focus here on the statistical advantages of RRs. Henderson (2022) proposes a similar table where she also discusses the benefits for researchers, such as the reduced stress associated with the publication process.

  9. Henderson et al. (2019) is an example of a Stage-1 manuscript with conditional results. https://osf.io/8rq7k.

  10. Calculating statistical power based on the observed effect size would indeed be a form of tautology: observed effect sizes that are not statistically significant are indeed more likely to have low statistical power. See Althouse (2021) for a brief discussion.

  11. The iterative process stops as soon as the researchers are not able to reject a hull hypothesis.

  12. List et al. (2019) propose a novel and less-restrictive approach to deal with the simultaneous testing of null hypotheses. The results show improvements over the Holm and Bonferroni corrections, but continue to indicate the price of testing an additional hypothesis.

  13. Peer Community In RRs (PCI RR) is a researcher-run, non-profit, and non-commercial platform that reviews and recommends pre-prints RRs.

  14. All of the information for the submission of RRs to the JDE are available on the dedicated website: http://jde-preresultsreview.org/. The Journal of Political Economy and Q-Open have not specified any submission level for RRs at present.

  15. https://www.cos.io/initiatives/registered-reports.

  16. Some journals like Nature Human Behavior require authors to sign a statement confirming that if they withdraw their paper after in-principle acceptance, they agree to the journal publishing a short summary of the pre-registered study under a dedicated section.

  17. Nature Human Behavior and Cortex accept Bayes factor analysis (Dienes, 2020) We do not know of any specific journal in economics policy regarding Bayes factor analysis.

References

  • Abrams, E., Libgober, J., & List, J. A. (2020). Research registries: Facts, myths, and possible improvements. NBER Working Paper.

  • Agnoli, F., Wicherts, J. M., Veldkamp, C. L., Albiero, P., & Cubelli, R. (2017). Questionable research practices among Italian research psychologists. PloS One, 12(3), e0172792.

    Article  Google Scholar 

  • Albers, C., & Lakens, D. (2018). When power analyses based on pilot data are biased: Inaccurate effect size estimators and follow-up bias. Journal of Experimental Social Psychology, 74, 187–195.

    Article  Google Scholar 

  • Althouse, A. D. (2021). Post hoc power: Not empowering, just misleading. Journal of Surgical Research, 259, A3–A6.

    Article  Google Scholar 

  • Altman, D. G., & Bland, J. M. (1995). Statistics notes: Absence of evidence is not evidence of absence. BMJ, 311(7003), 485.

    Article  Google Scholar 

  • Amrhein, V., Greenland, S., & McShane, B. (2019). Scientists rise up against statistical significance. Nature, 567, 305–307.

    Article  Google Scholar 

  • Bakker, M., Veldkamp, C. L., van Assen, M. A., Crompvoets, E. A., Ong, H. H., Nosek, B. A., Soderberg, C. K., Mellor, D., & Wicherts, J. M. (2020). Ensuring the quality and specificity of preregistrations. PLoS Biology, 18(12), e3000937.

    Article  Google Scholar 

  • Beaumont, J.-F., & Rivest, L.-P. (2009). Dealing with outliers in survey data. Handbook of statistics (Vol. 29, pp. 247–279). Elsevier.

    Google Scholar 

  • Bellemare, C., Bissonnette, L., & Kröger, S. (2016). Simulating power of economic experiments: The powerbbk package. Journal of the Economic Science Association, 2(2), 157–168.

    Article  Google Scholar 

  • Brodeur, A., Cook, N., Hartley, J., & Heyes, A. (2022). Do pre-registration and pre-analysis plans reduce p-hacking and publication bias? Available at SSRN.

  • Brodeur, A., Lé, M., Sangnier, M., & Zylberberg, Y. (2016). Star wars: The empirics strike back. American Economic Journal: Applied Economics, 8(1), 1–32.

    Google Scholar 

  • Bruns, S., Deressa, T. K., Stanley, T., Doucouliagos, C., & Ioannidis, J. (2022). Estimating the extent of inflated significance in economics. MetaArXiv—Preprints.

  • Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., et al. (2016). Evaluating replicability of laboratory experiments in economics. Science, 351(6280), 1433–1436.

    Article  Google Scholar 

  • Chambers, C. D., Feredoes, E., Muthukumaraswamy, S. D., & Etchells, P. (2014). Instead of playing the game it is time to change the rules: Registered reports at aims neuroscience and beyond. AIMS Neuroscience, 1(1), 4–17.

    Article  Google Scholar 

  • Chambers, C. D., & Tzavella, L. (2022). The past, present and future of registered reports. Nature Human Behaviour, 6, 29–42.

    Article  Google Scholar 

  • Chen, R., Chen, Y., & Riyanto, Y. E. (2021). Best practices in replication: A case study of common information in coordination games. Experimental Economics, 24(1), 2–30.

    Article  Google Scholar 

  • Christensen, G., & Miguel, E. (2018). Transparency, reproducibility, and the credibility of economics research. Journal of Economic Literature, 56(3), 920–80.

    Article  Google Scholar 

  • Clarke, D., Romano, J. P., & Wolf, M. (2020). The Romano–Wolf multiple-hypothesis correction in Stata. The Stata Journal, 20(4), 812–843.

    Article  Google Scholar 

  • Dienes, Z. (2011). Bayesian versus orthodox statistics: Which side are you on? Perspectives on Psychological Science, 6(3), 274–290.

    Article  Google Scholar 

  • Dienes, Z. (2020). The inner workings of registered reports. PsyArXiv—Preprints.

  • Dienes, Z. (2021). Obtaining evidence for no effect. Collabra: Psychology, 7(1), 28202.

    Article  Google Scholar 

  • Dienes, Z. (2022). Testing theories with Bayes factors. PsyArXiv—Preprints.

  • Dutilh, G., Sarafoglou, A., & Wagenmakers, E.-J. (2021). Flexible yet fair: Blinding analyses in experimental psychology. Synthese, 198(23), 5745–5772.

    Article  Google Scholar 

  • Espinosa, R., Arpinon, T., Maginot, P., Demange, S., & Peureux, F. (2022). Removing barriers to plant-based diets: assisting doctors with vegan patients. Stage-2 Registered Report accepted at PCI RR.

  • Espinosa, R., & Treich, N. (2021). Moderate versus radical NGOs. American Journal of Agricultural Economics, 103(4), 1478–1501.

    Article  Google Scholar 

  • Fanelli, D. (2009). How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PloS One, 4(5), e5738.

    Article  Google Scholar 

  • Fanelli, D. (2010). Positive results increase down the hierarchy of the sciences. PloS One, 5(4), e10068.

    Article  Google Scholar 

  • Ferraro, P. J., & Shukla, P. (2020). Feature—Is a replicability crisis on the horizon for environmental and resource economics? Review of Environmental Economics and Policy, 14(2).

  • Fiedler, K., & Schwarz, N. (2016). Questionable research practices revisited. Social Psychological and Personality Science, 7(1), 45–52.

    Article  Google Scholar 

  • Franco, A., Malhotra, N., & Simonovits, G. (2014). Publication bias in the social sciences: Unlocking the file drawer. Science, 345(6203), 1502–1505.

    Article  Google Scholar 

  • Gelman, A., & Loken, E. (2013). The garden of forking paths: Why multiple comparisons can be a problem, even when there is no fishing expedition or p-hacking and the research hypothesis was posited ahead of time. Department of Statistics, Columbia University, 348, 1–17.

    Google Scholar 

  • Glennerster, R., & Takavarasha, K. (2013). Running randomized evaluations. Running randomized evaluations. Princeton University Press.

    Chapter  Google Scholar 

  • Heckelei, T., Hüttel, S., Odening, M., Rommel, J. (2022). The replicability crisis and the \(p\)-value debate—What are the consequences for the agricultural and food economics community? Preprints.

  • Henderson, E. (2022). A guide to preregistration and registered reports. MetaArXiv.

  • Henderson, E. L., & Chambers, C. D. (2022). Ten simple rules for writing a registered report. PLoS Computational Biology, 18(10), e1010571.

    Article  Google Scholar 

  • Henderson, E., Vallée-Tourangeau, F., & Simons, D. (2019). The effect of concrete wording on truth judgements: A preregistered replication and extension of Hansen & Wänke (2010). Registered Report—Stage 1, retrieved from osf.io/f9jh6.

  • Jasielska, D., Rogoza, R., Zajenkowska, A., & Russa, M. B. (2021). General trust scale: Validation in cross-cultural settings. Current Psychology, 40(10), 5019–5029.

    Article  Google Scholar 

  • John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524–532.

    Article  Google Scholar 

  • Lakens, D. (2014). Performing high-powered studies efficiently with sequential analyses. European Journal of Social Psychology, 44(7), 701–710.

    Article  Google Scholar 

  • Lakens, D. (2017). Equivalence tests: A practical primer for t tests, correlations, and meta-analyses. Social Psychological and Personality Science, 8(4), 355–362.

    Article  Google Scholar 

  • Lakens, D. (2022). Sample size justification. Collabra: Psychology, 8(1), 33267.

    Article  Google Scholar 

  • Lakens, D., Scheel, A. M., & Isager, P. M. (2018). Equivalence testing for psychological research: A tutorial. Advances in Methods and Practices in Psychological Science, 1(2), 259–269.

    Article  Google Scholar 

  • LeBel, E. P., Borsboom, D., Giner-Sorolla, R., Hasselman, F., Peters, K. R., Ratliff, K. A., & Smith, C. T. (2013). Psychdisclosure.org: Grassroots support for reforming reporting standards in psychology. Perspectives on Psychological Science, 8(4), 424–432.

    Article  Google Scholar 

  • Leys, C., Delacre, M., Mora, Y. L., Lakens, D., & Ley, C. (2019). How to classify, detect, and manage univariate and multivariate outliers, with emphasis on pre-registration. International Review of Social Psychology, 32(1).

  • List, J. A., Shaikh, A. M., & Xu, Y. (2019). Multiple hypothesis testing in experimental economics. Experimental Economics, 22(4), 773–793.

    Article  Google Scholar 

  • Loken, E., & Gelman, A. (2017). Measurement error and the replication crisis. Science, 355(6325), 584–585.

    Article  Google Scholar 

  • MacCoun, R. J., & Perlmutter, S. (2017). Blind analysis as a correction for confirmatory bias in physics and in psychology. In Psychological science under scrutiny: Recent challenges and proposed solutions.

  • McCloskey, D. N., & Ziliak, S. T. (1996). The standard error of regressions. Journal of Economic Literature, 34(1), 97–114.

    Google Scholar 

  • Miguel, E. (2021). Evidence on research transparency in economics. Journal of Economic Perspectives, 35(3), 193–214.

    Article  Google Scholar 

  • Necker, S. (2014). Scientific misbehavior in economics. Research Policy, 43(10), 1747–1759.

    Article  Google Scholar 

  • Nosek, B. A., & Lakens, D. (2014). Registered reports: A method to increase the credibility of published results. Social Psychology, 45(3), 137–141

  • O’Boyle, E. H., Jr., Banks, G. C., & Gonzalez-Mulé, E. (2017). The chrysalis effect: How ugly initial results metamorphosize into beautiful articles. Journal of Management, 43(2), 376–399.

    Article  Google Scholar 

  • Ofosu, G. K., & Posner, D. N. (2021). Pre-analysis plans: An early stocktaking. In Perspectives on Politics, pp. 1–17. https://www.cambridge.org/core/journals/perspectives-on-politics/article/preanalysis-plans-an-earlystocktaking/94E7FAE76001C45A04E8F5E272C773CE

  • Olken, B. A. (2015). Promises and perils of pre-analysis plans. Journal of Economic Perspectives, 29(3), 61–80.

    Article  Google Scholar 

  • Page, L., Noussair, C. N., & Slonim, R. (2021). The replication crisis, the rise of new research practices and what it means for experimental economics. Journal of the Economic Science Association, 7(2), 210–225.

    Article  Google Scholar 

  • Romano, J. P., & Wolf, M. (2005). Stepwise multiple testing as formalized data snooping. Econometrica, 73(4), 1237–1282.

    Article  Google Scholar 

  • Romano, J. P., & Wolf, M. (2016). Efficient computation of adjusted p-values for resampling-based stepdown multiple testing. Statistics & Probability Letters, 113, 38–40.

    Article  Google Scholar 

  • Schafmeister, F. (2021). The effect of replications on citation patterns: Evidence from a large-scale reproducibility project. Psychological Science, 32(10), 1537–1548.

    Article  Google Scholar 

  • Scheel, A. M., Schijen, M. R., & Lakens, D. (2021). An excess of positive results: Comparing the standard psychology literature with registered reports. Advances in Methods and Practices in Psychological Science, 4(2), 25152459211007468.

    Article  Google Scholar 

  • Schooler, J. W. (2014). Metascience could rescue the replication crisis. Nature, 515(7525), 9–9.

    Article  Google Scholar 

  • Serra-Garcia, M., & Gneezy, U. (2021). Nonreplicable publications are cited more than replicable ones. Science Advances, 7(21), eabd1705.

    Article  Google Scholar 

  • Swanson, N., Christensen, G., Littman, R., Birke, D., Miguel, E., Paluck, E. L., & Wang, Z. (2020). Research transparency is on the rise in economics. In In AEA Papers and Proceedings, vol. 110, pp. 61–65.

  • Thissen, D., Steinberg, L., & Kuang, D. (2002). Quick and easy implementation of the Benjamini–Hochberg procedure for controlling the false positive rate in multiple comparisons. Journal of Educational and Behavioral Statistics, 27(1), 77–83.

    Article  Google Scholar 

  • Van’t Veer, A. E., & Giner-Sorolla, R. (2016). Pre-registration in social psychology—A discussion and suggested template. Journal of Experimental Social Psychology, 67, 2–12.

    Article  Google Scholar 

  • Wicherts, J. M., Veldkamp, C. L., Augusteijn, H. E., Bakker, M., Van Aert, R., & Van Assen, M. A. (2016). Degrees of freedom in planning, running, analyzing, and reporting psychological studies: A checklist to avoid p-hacking. In Frontiers in psychology, p. 1832.

  • Yamagishi, T., & Yamagishi, M. (1994). Trust and commitment in the United States and Japan. Motivation and Emotion, 18(2), 129–166.

    Article  Google Scholar 

  • Young, C., & Holsteen, K. (2017). Model uncertainty and robustness: A computational framework for multimodel analysis. Sociological Methods & Research, 46(1), 3–40.

    Article  Google Scholar 

  • Ziliak, S. T., & McCloskey, D. N. (2004). Size matters: The standard error of regressions in the American Economic Review. Journal of Socio-Economics, 33(5), 527–546.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. CREM, University of Rennes 1, Rennes, France

    Thibaut Arpinon

  2. CIRED, CNRS, Nogent-sur-Marne, France

    Romain Espinosa

Authors
  1. Thibaut Arpinon
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Romain Espinosa
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Romain Espinosa.

Ethics declarations

Conflict of interest

Romain Espinosa acknowledges financial support from the ANR under Grant ANR-19-CE21- 0005-01. Thibaut Arpinon has no conflicting interests in this study (no paid or unpaid position in an interested organization). Romain Espinosa acts as a recommender at Peer-Community In-Registered Reports.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The authors thank Lionel Page, Emma Henderson, Daniel Lakens, Zoltan Dienes, Jens Rommel, Anna Dreber Almenberg, Andrew Clark, Marianne Lefebvre, and Etienne Dagorn for useful comments.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file 1 (pdf 973 KB)

A Appendix

A Appendix

See Fig. 5.

Fig. 5
figure 5

The decision rule with a smallest effect size of interest (SESOI) with confidence intervals

Full size image

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Arpinon, T., Espinosa, R. A practical guide to Registered Reports for economists. J Econ Sci Assoc 9, 90–122 (2023). https://doi.org/10.1007/s40881-022-00123-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40881-022-00123-1

Keywords

JEL Classification

Search

Navigation