Skip to main content
SpringerLink
Log in

“Reproducible” Research in Mathematical Sciences Requires Changes in our Peer Review Culture and Modernization of our Current Publication Approach

  • Perspectives Article
  • Published:
Bulletin of Mathematical Biology Aims and scope Submit manuscript

Abstract

The nature of scientific research in mathematical and computational biology allows editors and reviewers to evaluate the findings of a scientific paper. Replication of a research study should be the minimum standard for judging its scientific claims and considering it for publication. This requires changes in the current peer review practice and a strict adoption of a replication policy similar to those adopted in experimental fields such as organic synthesis. In the future, the culture of replication can be easily adopted by publishing papers through dynamic computational notebooks combining formatted text, equations, computer algebra and computer code.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. https://www.nibib.nih.gov/research-funding/interagency-modeling-and-analysis-group-imag.

  2. https://www.imagwiki.nibib.nih.gov/content/model-and-data-sharing-working-group.

  3. For August 10, 2018, the BioModels Database (http://www.ebi.ac.uk/biomodels/) has 641 manually curated (fully reproducible) models and 1008 non-curated models.

  4. http://www.orgsyn.org/.

  5. https://www.jove.com/.

  6. http://jupyter.org.

References

  • Alberts B, Stodden V, Young S, Choudhury S (2013) Testimony on scientific integrity and transparency. Available online at https://science.house.gov/legislation/hearings/subcommittee-research-scientific-integrity-transparency. Accessed 10 Aug 2018

  • Association for Computing Machinery (2016) Artifact review and badging. Available online at: https://www.acm.org/publications/policies/artifact-review-badging. Accessed 10 Aug 2018

  • Baker M (2016) 1500 scientists lift the lid on reproducibility. Nature 533:452–454

    Article  Google Scholar 

  • Bakker M, Wicherts JM (2011) The (mis)reporting of statistical results in psychology journals. Behav Res Methods 43:666–678

    Article  Google Scholar 

  • Begley CG, Ellis LM (2012) Drug development: raise standards for preclinical cancer research. Nature 483:531–533

    Article  Google Scholar 

  • Bergman RG, Danheiser DL (2016) Reproducibility in chemical reaction. Angew Chem Int Ed 55:12548–12549

    Article  Google Scholar 

  • Björnmalm M, Faria M, Caruso F (2016) Increasing the impact of materials in and beyond bio-nano science. J Am Chem Soc 138:13449–13456

    Article  Google Scholar 

  • Bollen K, Cacioppo JT, Kaplan R, Krosnick J, Olds JL (2015) Social, behavioral, and economic sciences perspectives on robust and reliable science. National Science Foundation, Arlington, Virginia. Available online at: https://www.nsf.gov/sbe/SBE_Spring_2015_AC_Meeting_Presentations/Bollen_Report_on_Replicability_SubcommitteeMay_2015.pdf. Accessed 10 Aug 2018

  • Button KS, Ioannidis JP, Mokrysz C, Nosek BA, Flint J, Robinson ES, Munafò MR (2013) Power failure: why small sample size undermines the reliability of neuroscience. Nat Rev Neurosci 14:365–376

    Article  Google Scholar 

  • Chelliah V, Juty N, Ajmera I, Ali R, Dumousseau M, Glont M, Hucka M, Jalowicki G, Keating S, Knight-Schrijver V, Lloret-Villas A, Natarajan KN, Pettit JB, Rodriguez N, Schubert M, Wimalaratne SM, Zhao Y, Hermjakob H, Le Novère N, Laibe C (2015) BioModels: 10-year anniversary. Nucleic Acids Res 43:D542–D548

    Article  Google Scholar 

  • Collins FS, Tabak LA (2014) Policy: NIH plans to enhance reproducibility. Nature 505:612–613

    Article  Google Scholar 

  • Crook S, Davison AP, Plesser HE (2013) Learning from the past: approaches for reproducibility in computational neuroscience. In: Bower JM (ed) 20 Years in computational neuroscience. Springer, New York, pp 73–102

    Chapter  Google Scholar 

  • Danheiser DL (2011) Organic syntheses: the “Gold Standard” in experimental synthetic organic chemistry. Org Synth 88:1–3

    Article  Google Scholar 

  • Donoho DL, Maleki A, Rahman IU, Shahram M, Stodden V (2009) 15 Years of reproducible research in computational harmonic analysis. Comput Sci Eng 11:8–18

    Article  Google Scholar 

  • Editorial N (2014) Journals unite for reproducibility. Nature 515:7

    Google Scholar 

  • Fanelli D (2009) How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PLoS ONE 4:e5738

    Article  Google Scholar 

  • Fanelli D (2010) “Positive” results increase down the hierarchy of the sciences. PLoS ONE 5:e10068

    Article  Google Scholar 

  • Fanelli D (2018) Opinion: Is science really facing a reproducibility crisis, and do we need it to? Proc Natl Acad Sci USA 115:2628–2631

    Article  Google Scholar 

  • Fanelli D, Costas R, Ioannidis JPA (2017) Meta-assessment of bias in science. Proc Natl Acad Sci USA 114:3714–3719

    Article  Google Scholar 

  • Goodman SN, Fanelli D, Ioannidis JPA (2016) What does research reproducibility mean? Sci Transl Med 8:341ps12

    Article  Google Scholar 

  • Heroux M (2015) Editorial: ACM TOMS replicated computational results initiative. ACM Trans Math Softw 41:art13

    Article  Google Scholar 

  • Hirschhorn JN, Lohmueller K, Byrne E, Hirschhorn K (2002) A comprehensive review of genetic association studies. Genet Med 4:45–61

    Article  Google Scholar 

  • Hucka M, Finney A, Sauro HM, Bolouri H, Doyle JC, Kitano H, Arkin AP, Bornstein BJ, Bray D, Cornish-Bowden A, Cuellar AA, Dronov S, Gilles ED, Ginkel M, Gor V, Goryanin II, Hedley WJ, Hodgman TC, Hofmeyr JH, Hunter PJ, Juty NS, Kasberger JL, Kremling A, Kummer U, Le Novère N, Loew LM, Lucio D, Mendes P, Minch E, Mjolsness ED, Nakayama Y, Nelson MR, Nielsen PF, Sakurada T, Schaff JC, Shapiro BE, Shimizu TS, Spence HD, Stelling J, Takahashi K, Tomita M, Wagner J, Wang J (2003) The systems biology markup language (SBML): a medium for representation and exchange of biochemical network models. Bioinformatics 19:524–531

    Article  Google Scholar 

  • International Organization for Standardization (1994) Applications of statistical methods. Technical Committee ISO/TC 69, Subcommittee SC 6, Measurement methods and results. Available online at: https://www.iso.org/obp/ui/#iso:std:iso:5725:-1:ed-1:v1:en. Accessed 10 Aug 2018

  • International Union of Pure and Applied Chemistry (1997) IUPAC Compendium of Chemical Terminology. In: McNaught AD, Wilkinson A (eds), 2end. Blackwell Scientific Publications, Oxford. Available online at: https://goldbook.iupac.org/. Accessed 10 Aug 2018

  • Ioannidis JPA (2005) Why most published research findings are false. PLoS Med 2:e124

    Article  Google Scholar 

  • Ioannidis JPA (2014) How to make more published research true. PLoS Med 11:e1001747

    Article  Google Scholar 

  • Ioannidis JPA, Stanley TD, Doucouliagos H (2017) The power bias in economics research. Econ J 127:F236–F265

    Article  Google Scholar 

  • John LK, Loewenstein G, Prelec D (2012) Measuring the prevalence of questionable research practices with incentives for truth telling. Psychol Sci 23:524–532

    Article  Google Scholar 

  • Joint Committee for Guides in Metrology (2006) International Vocabulary of Metrology: Basic and General Concepts and Associated Terms, 3rd ed. Joint Committee for Guides in Metrology/Working Group 2. Available online at: https://www.nist.gov/sites/default/files/documents/pml/div688/grp40/International-Vocabulary-of-Metrology.pdf. Accessed 10 Aug 2018

  • Kronick DA (1976) History of scientific and technical periodicals, 2nd edn. Scarecrow Press, Metuchen

    Google Scholar 

  • Le Novère N, Finney A, Hucka M, Bhalla US, Campagne F, Collado-Vides J, Crampin EJ, Halstead M, Klipp E, Mendes P, Nielsen P, Sauro H, Shapiro B, Snoep JL, Spence HD, Wanner BL (2005) Minimum information requested in the annotation of biochemical models (MIRIAM). Nat Biotechnol 23:1509–1515

    Article  Google Scholar 

  • Li C, Donizelli M, Rodriguez N, Dharuri H, Endler L, Chelliah V, Li L, He E, Henry A, Stefan MI, Snoep JL, Hucka M, Le Novère N, Laibe C (2004) BioModels database: an enhanced, curated and annotated resource for published quantitative kinetic models. BMC Syst Biol 4:92

    Article  Google Scholar 

  • Lloyd CM, Halstead MD, Nielsen PF (2004) CellML: its future, present and past. Progr Biophys Mol Biol 85:433–450

    Article  Google Scholar 

  • Lloyd CM, Lawson JR, Hunter PJ, Nielsen PF (2008) The CellML model repository. Bioinformatics 24:2122–2123

    Article  Google Scholar 

  • Loew LM, Schaff JC (2001) The virtual cell: a software environment for computational cell biology. Trends Biotechnol 19:401–406

    Article  Google Scholar 

  • Macleod MR, Michie S, Roberts I, Dirnagl U, Chalmers I, Ioannidis JPA, Salman RAH, Chan AW, Glasziou P (2014) Biomedical research: increasing value, reducing waste. Lancet 383:101–104

    Article  Google Scholar 

  • McNutt M (2014) Journals unite for reproducibility. Science 346:6210

    Google Scholar 

  • Medley JK, Choi K, Konig M, Smith L, Gu S, Hellerstein J, Sealfon SC, Sauro HM (2018) Tellurium notebooks—an environment for reproducible dynamical modeling in systems biology. PLoS Comput Biol 14:e1006220

    Article  Google Scholar 

  • Munafò MR, Nosek BA, Bishop DVM, Button KS, Chambers CD, Percie du Sert N, Simonsohn U, Wagenmakers EJ, Ware JJ, Ioannidis JPA (2017) A manifesto for reproducible science. Nat Hum Behav 1:0021

    Article  Google Scholar 

  • National Institutes of Health (2017) Principles and guidelines for reporting preclinical research. Available online at: https://www.nih.gov/research-training/rigor-reproducibility/principles-guidelines-reporting-preclinical-research. Accessed 10 Aug 2018

  • National Institutes of Health (2018) Rigor and reproducibility. Available online at: https://grants.nih.gov/reproducibility/index.htm. Accessed 10 Aug 2018

  • Olivier BG, Snoep JL (2004) Web-based kinetic modelling using JWS online. Bioinformatics 20:2143–2144

    Article  Google Scholar 

  • Pashler H, Wagenmakers EJ (2012) Editors’ introduction to the special section on replicability in psychological science: A crisis of confidence? Perspect Psychol Sci 7:528–530

    Article  Google Scholar 

  • Pastrana E, Swaminathan S (2018) Nature research journals trial new tools to enhance code peer review and publication of scheme and memes. A community blog from nature.com. 01 Aug 2018, 15:05 BST. Available online at: http://blogs.nature.com/ofschemesandmemes/2018/08/01/nature-research-journals-trial-new-tools-to-enhance-code-peer-review-and-publication. Accessed 10 Aug 2018

  • Peng RD (2009) Reproducible research and biostatistics. Biostatistics 10:405–408

    Article  Google Scholar 

  • Peng RD (2011) Reproducible research in computational science. Science 334:1226–1227

    Article  Google Scholar 

  • Plesser HS (2018) Reproducibility versus replicability: a brief history of a confused terminology. Front Neuroinform 11:76

    Article  Google Scholar 

  • Prinz F, Schlange T, Asadullah K (2011) Believe it or not: How much can we rely on published data on potential drug targets? Nat Rev Drug Discov 10:712

    Article  Google Scholar 

  • Redish AD, Kummerfeld E, Morris RL, Love AC (2018) Reproducibility failures are essential to scientific inquiry. Proc Natl Acad Sci USA 115:5042–5046

    Article  Google Scholar 

  • Renear AH, Sacchi S, Wickett KM (2010) Definitions of dataset in the scientific and technical literature. Proc Am Soc Inf Sci Technol 47:1–4

    Article  Google Scholar 

  • Rougier NP, Hinsen K, Alexandre F, Arildsen T, Barba LA, Benureau FCY et al (2017) Sustainable computational science: the ReScience initiative. Available online at: https://arxiv.org/abs/1707.04393. Accessed 10 Aug 2018

  • Schnell S (2015) Ten simple rules for a computational biologist’s laboratory notebook. PLoS Comput Biol 11:e1004385

    Article  Google Scholar 

  • Shapin S, Schaffer S (1985) Leviathan and the air-pump: Hobbes, Boyle, and the experimental life. Princeton University Press, Princeton

    Google Scholar 

  • Shapiro MF, Charrow RP (1989) The role of data audits in detecting scientific misconduct. Results of the FDA program. JAMA 261:2505–2511

    Article  Google Scholar 

  • Snoep JL (2005) The silicon cell initiative: working towards a detailed kinetic description at the cellular level. Curr Opin Biotechnol 16:336–343

    Article  Google Scholar 

  • Sommers J (2018) The scientific paper is obsolete. The Atlantic, April 5, 2018. Available online at: https://www.theatlantic.com/science/archive/2018/04/the-scientific-paper-is-obsolete/556676/. Accessed 10 Aug 2018

  • Stark PB (2018) Before reproducibility must come preproducibility. Nature 557:613

    Article  Google Scholar 

  • Steneck NH (2006) Fostering integrity in research: definitions, current knowledge, and future directions. Sci Eng Ethics 12:53–74

    Article  Google Scholar 

  • Stodden V, Mitchell I, LeVeque R (2012) Reproducible research for scientific computing: tools and strategies for changing the culture. Comput Sci Eng 14:13–17

    Article  Google Scholar 

  • Stodden V, Guo P, Ma Z (2013) Toward reproducible computational research: an empirical analysis of data and code policy adoption by journals. PLoS ONE 8:e67111

    Article  Google Scholar 

  • Stodden V, McNutt M, Bailey DH, Deelman E, Gil Y, Hanson B, Heroux MA, Ioannidis JP, Taufer M (2016) Enhancing reproducibility for computational methods. Science 354:1240–1241

    Article  Google Scholar 

  • Stodden V, Seiler J, Ma Z (2018) An empirical analysis of journal policy effectiveness for computational reproducibility. Proc Natl Acad Sci USA 115:2584–2589

    Article  Google Scholar 

  • Swat M, Thomas GL, Belmonte JM, Shirinifard A, Hmeljak D, Glazier JA (2012) Computational methods in cell biology. Methods Cell Biol 110:325–366

    Article  Google Scholar 

  • Van Bavel (2016) Why do so many studies fail to replicate? New York Times, May 27, 2016, page SR10. Available online at: https://www.nytimes.com/2016/05/29/opinion/sunday/why-do-so-many-studies-fail-to-replicate.html. Accessed 10 Aug 2018

  • Van Bavel JJ, Mende-Siedlecki P, Brady WJ, Reinero DA (2016) Contextual sensitivity in scientific reproducibility. Proc Natl Acad Sci USA 113:6454–6459

    Article  Google Scholar 

  • Wicherts JM, Borsboom D, Kats J, Molenaar D (2006) The poor availability of psychological research data for reanalysis. Am Psychol 61:726–728

    Article  Google Scholar 

  • Wilson G, Bryan J, Cranston K, Kitzes J, Nederbragt L, Teal TK (2017) Good enough practices in scientific computing. PLoS Comput Biol 13:e1005510

    Article  Google Scholar 

  • Wood P, Randall D (2016) How bad is the government’s science? Wall Street Journal, April 16, 2018 5:56 ET. Available online at https://www.wsj.com/articles/how-bad-is-the-governments-science-1523915765. Accessed 10 Aug 2018

  • Yale RoundTable Participants (2010) Reproducible research: addressing the need for data and code sharing in computational science. Comput Sci Eng 12:5

    Google Scholar 

Download references

Acknowledgements

I am very grateful for the helpful insights provided by Rick Danheiser (Editor-in-Chief of Organic Syntheses and Massachusetts Institute of Technology, USA), Edmund Crampin (University of Melbourne, Australia) and Wylie Stroberg (University of Michigan). This work was partially supported through the educational programs funded by NIGMS (T32 GM008322) and NIDDK (R25 DK088752).

Author information

Authors and Affiliations

  1. Department of Molecular and Integrative Physiology, University of Michigan Medical School, Ann Arbor, MI, 48109, USA

    Santiago Schnell

  2. Department of Computational Medicine and Bioinformatics, University of Michigan Medical School, Ann Arbor, MI, 48109, USA

    Santiago Schnell

Authors
  1. Santiago Schnell
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Santiago Schnell.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Schnell, S. “Reproducible” Research in Mathematical Sciences Requires Changes in our Peer Review Culture and Modernization of our Current Publication Approach. Bull Math Biol 80, 3095–3105 (2018). https://doi.org/10.1007/s11538-018-0500-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11538-018-0500-9

Keywords

Search

Navigation