Bulletin of Mathematical Biology

, Volume 80, Issue 12, pp 3095–3105 | Cite as

“Reproducible” Research in Mathematical Sciences Requires Changes in our Peer Review Culture and Modernization of our Current Publication Approach

  • Santiago SchnellEmail author
Perspectives Article
Part of the following topical collections:
  1. Reproducibility in Computational Biology


The nature of scientific research in mathematical and computational biology allows editors and reviewers to evaluate the findings of a scientific paper. Replication of a research study should be the minimum standard for judging its scientific claims and considering it for publication. This requires changes in the current peer review practice and a strict adoption of a replication policy similar to those adopted in experimental fields such as organic synthesis. In the future, the culture of replication can be easily adopted by publishing papers through dynamic computational notebooks combining formatted text, equations, computer algebra and computer code.


Reproducibility Repeatability Replicability Editorial policies Academic publishing 



I am very grateful for the helpful insights provided by Rick Danheiser (Editor-in-Chief of Organic Syntheses and Massachusetts Institute of Technology, USA), Edmund Crampin (University of Melbourne, Australia) and Wylie Stroberg (University of Michigan). This work was partially supported through the educational programs funded by NIGMS (T32 GM008322) and NIDDK (R25 DK088752).


  1. Alberts B, Stodden V, Young S, Choudhury S (2013) Testimony on scientific integrity and transparency. Available online at Accessed 10 Aug 2018
  2. Association for Computing Machinery (2016) Artifact review and badging. Available online at: Accessed 10 Aug 2018
  3. Baker M (2016) 1500 scientists lift the lid on reproducibility. Nature 533:452–454CrossRefGoogle Scholar
  4. Bakker M, Wicherts JM (2011) The (mis)reporting of statistical results in psychology journals. Behav Res Methods 43:666–678CrossRefGoogle Scholar
  5. Begley CG, Ellis LM (2012) Drug development: raise standards for preclinical cancer research. Nature 483:531–533CrossRefGoogle Scholar
  6. Bergman RG, Danheiser DL (2016) Reproducibility in chemical reaction. Angew Chem Int Ed 55:12548–12549CrossRefGoogle Scholar
  7. Björnmalm M, Faria M, Caruso F (2016) Increasing the impact of materials in and beyond bio-nano science. J Am Chem Soc 138:13449–13456CrossRefGoogle Scholar
  8. Bollen K, Cacioppo JT, Kaplan R, Krosnick J, Olds JL (2015) Social, behavioral, and economic sciences perspectives on robust and reliable science. National Science Foundation, Arlington, Virginia. Available online at: Accessed 10 Aug 2018
  9. Button KS, Ioannidis JP, Mokrysz C, Nosek BA, Flint J, Robinson ES, Munafò MR (2013) Power failure: why small sample size undermines the reliability of neuroscience. Nat Rev Neurosci 14:365–376CrossRefGoogle Scholar
  10. Chelliah V, Juty N, Ajmera I, Ali R, Dumousseau M, Glont M, Hucka M, Jalowicki G, Keating S, Knight-Schrijver V, Lloret-Villas A, Natarajan KN, Pettit JB, Rodriguez N, Schubert M, Wimalaratne SM, Zhao Y, Hermjakob H, Le Novère N, Laibe C (2015) BioModels: 10-year anniversary. Nucleic Acids Res 43:D542–D548CrossRefGoogle Scholar
  11. Collins FS, Tabak LA (2014) Policy: NIH plans to enhance reproducibility. Nature 505:612–613CrossRefGoogle Scholar
  12. Crook S, Davison AP, Plesser HE (2013) Learning from the past: approaches for reproducibility in computational neuroscience. In: Bower JM (ed) 20 Years in computational neuroscience. Springer, New York, pp 73–102CrossRefGoogle Scholar
  13. Danheiser DL (2011) Organic syntheses: the “Gold Standard” in experimental synthetic organic chemistry. Org Synth 88:1–3CrossRefGoogle Scholar
  14. Donoho DL, Maleki A, Rahman IU, Shahram M, Stodden V (2009) 15 Years of reproducible research in computational harmonic analysis. Comput Sci Eng 11:8–18CrossRefGoogle Scholar
  15. Editorial N (2014) Journals unite for reproducibility. Nature 515:7Google Scholar
  16. Fanelli D (2009) How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PLoS ONE 4:e5738CrossRefGoogle Scholar
  17. Fanelli D (2010) “Positive” results increase down the hierarchy of the sciences. PLoS ONE 5:e10068CrossRefGoogle Scholar
  18. Fanelli D (2018) Opinion: Is science really facing a reproducibility crisis, and do we need it to? Proc Natl Acad Sci USA 115:2628–2631CrossRefGoogle Scholar
  19. Fanelli D, Costas R, Ioannidis JPA (2017) Meta-assessment of bias in science. Proc Natl Acad Sci USA 114:3714–3719CrossRefGoogle Scholar
  20. Goodman SN, Fanelli D, Ioannidis JPA (2016) What does research reproducibility mean? Sci Transl Med 8:341ps12CrossRefGoogle Scholar
  21. Heroux M (2015) Editorial: ACM TOMS replicated computational results initiative. ACM Trans Math Softw 41:art13CrossRefGoogle Scholar
  22. Hirschhorn JN, Lohmueller K, Byrne E, Hirschhorn K (2002) A comprehensive review of genetic association studies. Genet Med 4:45–61CrossRefGoogle Scholar
  23. Hucka M, Finney A, Sauro HM, Bolouri H, Doyle JC, Kitano H, Arkin AP, Bornstein BJ, Bray D, Cornish-Bowden A, Cuellar AA, Dronov S, Gilles ED, Ginkel M, Gor V, Goryanin II, Hedley WJ, Hodgman TC, Hofmeyr JH, Hunter PJ, Juty NS, Kasberger JL, Kremling A, Kummer U, Le Novère N, Loew LM, Lucio D, Mendes P, Minch E, Mjolsness ED, Nakayama Y, Nelson MR, Nielsen PF, Sakurada T, Schaff JC, Shapiro BE, Shimizu TS, Spence HD, Stelling J, Takahashi K, Tomita M, Wagner J, Wang J (2003) The systems biology markup language (SBML): a medium for representation and exchange of biochemical network models. Bioinformatics 19:524–531CrossRefGoogle Scholar
  24. International Organization for Standardization (1994) Applications of statistical methods. Technical Committee ISO/TC 69, Subcommittee SC 6, Measurement methods and results. Available online at: Accessed 10 Aug 2018
  25. International Union of Pure and Applied Chemistry (1997) IUPAC Compendium of Chemical Terminology. In: McNaught AD, Wilkinson A (eds), 2end. Blackwell Scientific Publications, Oxford. Available online at: Accessed 10 Aug 2018
  26. Ioannidis JPA (2005) Why most published research findings are false. PLoS Med 2:e124CrossRefGoogle Scholar
  27. Ioannidis JPA (2014) How to make more published research true. PLoS Med 11:e1001747CrossRefGoogle Scholar
  28. Ioannidis JPA, Stanley TD, Doucouliagos H (2017) The power bias in economics research. Econ J 127:F236–F265CrossRefGoogle Scholar
  29. John LK, Loewenstein G, Prelec D (2012) Measuring the prevalence of questionable research practices with incentives for truth telling. Psychol Sci 23:524–532CrossRefGoogle Scholar
  30. Joint Committee for Guides in Metrology (2006) International Vocabulary of Metrology: Basic and General Concepts and Associated Terms, 3rd ed. Joint Committee for Guides in Metrology/Working Group 2. Available online at: Accessed 10 Aug 2018
  31. Kronick DA (1976) History of scientific and technical periodicals, 2nd edn. Scarecrow Press, MetuchenGoogle Scholar
  32. Le Novère N, Finney A, Hucka M, Bhalla US, Campagne F, Collado-Vides J, Crampin EJ, Halstead M, Klipp E, Mendes P, Nielsen P, Sauro H, Shapiro B, Snoep JL, Spence HD, Wanner BL (2005) Minimum information requested in the annotation of biochemical models (MIRIAM). Nat Biotechnol 23:1509–1515CrossRefGoogle Scholar
  33. Li C, Donizelli M, Rodriguez N, Dharuri H, Endler L, Chelliah V, Li L, He E, Henry A, Stefan MI, Snoep JL, Hucka M, Le Novère N, Laibe C (2004) BioModels database: an enhanced, curated and annotated resource for published quantitative kinetic models. BMC Syst Biol 4:92CrossRefGoogle Scholar
  34. Lloyd CM, Halstead MD, Nielsen PF (2004) CellML: its future, present and past. Progr Biophys Mol Biol 85:433–450CrossRefGoogle Scholar
  35. Lloyd CM, Lawson JR, Hunter PJ, Nielsen PF (2008) The CellML model repository. Bioinformatics 24:2122–2123CrossRefGoogle Scholar
  36. Loew LM, Schaff JC (2001) The virtual cell: a software environment for computational cell biology. Trends Biotechnol 19:401–406CrossRefGoogle Scholar
  37. Macleod MR, Michie S, Roberts I, Dirnagl U, Chalmers I, Ioannidis JPA, Salman RAH, Chan AW, Glasziou P (2014) Biomedical research: increasing value, reducing waste. Lancet 383:101–104CrossRefGoogle Scholar
  38. McNutt M (2014) Journals unite for reproducibility. Science 346:6210Google Scholar
  39. Medley JK, Choi K, Konig M, Smith L, Gu S, Hellerstein J, Sealfon SC, Sauro HM (2018) Tellurium notebooks—an environment for reproducible dynamical modeling in systems biology. PLoS Comput Biol 14:e1006220CrossRefGoogle Scholar
  40. Munafò MR, Nosek BA, Bishop DVM, Button KS, Chambers CD, Percie du Sert N, Simonsohn U, Wagenmakers EJ, Ware JJ, Ioannidis JPA (2017) A manifesto for reproducible science. Nat Hum Behav 1:0021CrossRefGoogle Scholar
  41. National Institutes of Health (2017) Principles and guidelines for reporting preclinical research. Available online at: Accessed 10 Aug 2018
  42. National Institutes of Health (2018) Rigor and reproducibility. Available online at: Accessed 10 Aug 2018
  43. Olivier BG, Snoep JL (2004) Web-based kinetic modelling using JWS online. Bioinformatics 20:2143–2144CrossRefGoogle Scholar
  44. Pashler H, Wagenmakers EJ (2012) Editors’ introduction to the special section on replicability in psychological science: A crisis of confidence? Perspect Psychol Sci 7:528–530CrossRefGoogle Scholar
  45. Pastrana E, Swaminathan S (2018) Nature research journals trial new tools to enhance code peer review and publication of scheme and memes. A community blog from 01 Aug 2018, 15:05 BST. Available online at: Accessed 10 Aug 2018
  46. Peng RD (2009) Reproducible research and biostatistics. Biostatistics 10:405–408CrossRefGoogle Scholar
  47. Peng RD (2011) Reproducible research in computational science. Science 334:1226–1227CrossRefGoogle Scholar
  48. Plesser HS (2018) Reproducibility versus replicability: a brief history of a confused terminology. Front Neuroinform 11:76CrossRefGoogle Scholar
  49. Prinz F, Schlange T, Asadullah K (2011) Believe it or not: How much can we rely on published data on potential drug targets? Nat Rev Drug Discov 10:712CrossRefGoogle Scholar
  50. Redish AD, Kummerfeld E, Morris RL, Love AC (2018) Reproducibility failures are essential to scientific inquiry. Proc Natl Acad Sci USA 115:5042–5046CrossRefGoogle Scholar
  51. Renear AH, Sacchi S, Wickett KM (2010) Definitions of dataset in the scientific and technical literature. Proc Am Soc Inf Sci Technol 47:1–4CrossRefGoogle Scholar
  52. Rougier NP, Hinsen K, Alexandre F, Arildsen T, Barba LA, Benureau FCY et al (2017) Sustainable computational science: the ReScience initiative. Available online at: Accessed 10 Aug 2018
  53. Schnell S (2015) Ten simple rules for a computational biologist’s laboratory notebook. PLoS Comput Biol 11:e1004385CrossRefGoogle Scholar
  54. Shapin S, Schaffer S (1985) Leviathan and the air-pump: Hobbes, Boyle, and the experimental life. Princeton University Press, PrincetonGoogle Scholar
  55. Shapiro MF, Charrow RP (1989) The role of data audits in detecting scientific misconduct. Results of the FDA program. JAMA 261:2505–2511CrossRefGoogle Scholar
  56. Snoep JL (2005) The silicon cell initiative: working towards a detailed kinetic description at the cellular level. Curr Opin Biotechnol 16:336–343CrossRefGoogle Scholar
  57. Sommers J (2018) The scientific paper is obsolete. The Atlantic, April 5, 2018. Available online at: Accessed 10 Aug 2018
  58. Stark PB (2018) Before reproducibility must come preproducibility. Nature 557:613CrossRefGoogle Scholar
  59. Steneck NH (2006) Fostering integrity in research: definitions, current knowledge, and future directions. Sci Eng Ethics 12:53–74CrossRefGoogle Scholar
  60. Stodden V, Mitchell I, LeVeque R (2012) Reproducible research for scientific computing: tools and strategies for changing the culture. Comput Sci Eng 14:13–17CrossRefGoogle Scholar
  61. Stodden V, Guo P, Ma Z (2013) Toward reproducible computational research: an empirical analysis of data and code policy adoption by journals. PLoS ONE 8:e67111CrossRefGoogle Scholar
  62. Stodden V, McNutt M, Bailey DH, Deelman E, Gil Y, Hanson B, Heroux MA, Ioannidis JP, Taufer M (2016) Enhancing reproducibility for computational methods. Science 354:1240–1241CrossRefGoogle Scholar
  63. Stodden V, Seiler J, Ma Z (2018) An empirical analysis of journal policy effectiveness for computational reproducibility. Proc Natl Acad Sci USA 115:2584–2589CrossRefGoogle Scholar
  64. Swat M, Thomas GL, Belmonte JM, Shirinifard A, Hmeljak D, Glazier JA (2012) Computational methods in cell biology. Methods Cell Biol 110:325–366CrossRefGoogle Scholar
  65. Van Bavel (2016) Why do so many studies fail to replicate? New York Times, May 27, 2016, page SR10. Available online at: Accessed 10 Aug 2018
  66. Van Bavel JJ, Mende-Siedlecki P, Brady WJ, Reinero DA (2016) Contextual sensitivity in scientific reproducibility. Proc Natl Acad Sci USA 113:6454–6459CrossRefGoogle Scholar
  67. Wicherts JM, Borsboom D, Kats J, Molenaar D (2006) The poor availability of psychological research data for reanalysis. Am Psychol 61:726–728CrossRefGoogle Scholar
  68. Wilson G, Bryan J, Cranston K, Kitzes J, Nederbragt L, Teal TK (2017) Good enough practices in scientific computing. PLoS Comput Biol 13:e1005510CrossRefGoogle Scholar
  69. Wood P, Randall D (2016) How bad is the government’s science? Wall Street Journal, April 16, 2018 5:56 ET. Available online at Accessed 10 Aug 2018
  70. Yale RoundTable Participants (2010) Reproducible research: addressing the need for data and code sharing in computational science. Comput Sci Eng 12:5Google Scholar

Copyright information

© Society for Mathematical Biology 2018

Authors and Affiliations

  1. 1.Department of Molecular and Integrative PhysiologyUniversity of Michigan Medical SchoolAnn ArborUSA
  2. 2.Department of Computational Medicine and BioinformaticsUniversity of Michigan Medical SchoolAnn ArborUSA

Personalised recommendations