Bulletin of Mathematical Biology

, Volume 80, Issue 12, pp 3081–3087 | Cite as

Reproducible Research Using Biomodels

  • Pedro MendesEmail author
Perspectives Article
Part of the following topical collections:
  1. Reproducibility in Computational Biology


Like other types of computational research, modeling and simulation of biological processes (biomodels) is still largely communicated without sufficient detail to allow independent reproduction of results. But reproducibility in this area of research could easily be achieved by making use of existing resources, such as supplying models in standard formats and depositing code, models, and results in public repositories.


Reproducibility Biomodels Standards 



Funding was provided by National Institute of General Medical Sciences (Grant No. GM080219).


  1. Baker M (2016) 1,500 scientists lift the lid on reproducibility. Nature 533(7604):452–454. CrossRefGoogle Scholar
  2. Chance B, Garfinkel D, Higgins J, Hess B (1960) Metabolic control mechanisms. V. A solution for the equations representing interaction between glycolysis and respiration in ascites tumor cells. J Biol Chem 235(8):2426–2439Google Scholar
  3. Chelliah V, Juty N, Ajmera I, Ali R, Dumousseau M, Glont M, Hucka M, Jalowicki G, Keating S, Knight-Schrijver V, Lloret-Villas A, Natarajan KN, Pettit JB, Rodriguez N, Schubert M, Wimalaratne SM, Zhao Y, Hermjakob H, Le Novère N, Laibe C (2015) BioModels: ten-year anniversary. Nucleic Acids Res 43:D542–548. CrossRefGoogle Scholar
  4. Claerbout JF, Karrenbach M (1992) Electronic documents give reproducible research a new meaning. Soc Explor Geophys.
  5. Dada JO, Spasić I, Paton NW, Mendes P (2010) SBRML: a markup language for associating systems biology data with models. Bioinformatics 26(7):932–938. CrossRefGoogle Scholar
  6. Gillespie DT (1976) A general method for numerically simulating the stochastic time evolution of coupled chemical reactions. J Comput Phys 22:403–434MathSciNetCrossRefGoogle Scholar
  7. Gleeson P, Crook S, Cannon RC, Hines ML, Billings GO, Farinella M, Morse TM, Davison AP, Ray S, Bhalla US, Barnes SR, Dimitrova YD, Silver RA (2010) NeuroML: a language for describing data driven models of neurons and networks with a high degree of biological detail. PLoS Comput Biol 6(6):e1000815. MathSciNetCrossRefGoogle Scholar
  8. Goodman SN, Fanelli D, Ioannidis JPA (2016) What does research reproducibility mean? Sci Transl Med 8(341):341ps12. CrossRefGoogle Scholar
  9. Greenbaum D, Rozowsky J, Stodden V, Gerstein M (2017) Structuring supplemental materials in support of reproducibility. Genome Biol 18:64. CrossRefGoogle Scholar
  10. Guerreiro M (2017) Forking software used in eLife papers to GitHub .
  11. Hoops S, Sahle S, Gauges R, Lee C, Pahle J, Simus N, Singhal M, Xu L, Mendes P, Kummer U (2006) COPASI: a COmplex PAthway SImulator. Bioinformatics 22(24):3067–3074. CrossRefGoogle Scholar
  12. Hothorn T, Held L, Friede T (2009) Biometrical journal and reproducible research. Biom J 51(4):553–555. MathSciNetCrossRefGoogle Scholar
  13. Hothorn T, Leisch F (2011) Case studies in reproducibility. Brief Bioinform 12(3):288–300. CrossRefGoogle Scholar
  14. Hübner K, Sahle S, Kummer U (2011) Applications and trends in systems biology in biochemistry: systems biology in biochemical research. FEBS J 278(16):2767–2857. CrossRefGoogle Scholar
  15. Hucka M, Finney A, Sauro HM, Bolouri H, Doyle JC, Kitano H, Arkin AP, Bornstein BJ, Bray D, Cornish-Bowden A, Cuellar AA, Dronov S, Gilles ED, Ginkel M, Gor V, Goryanin II, Hedley WJ, Hodgman TC, Hofmeyr JH, Hunter PJ, Juty NS, Kasberger JL, Kremling A, Kummer U, Le Novère N, Loew LM, Lucio D, Mendes P, Minch E, Mjolsness ED, Nakayama Y, Nelson MR, Nielsen PF, Sakurada T, Schaff JC, Shapiro BE, Shimizu TS, Spence HD, Stelling J, Takahashi K, Tomita M, Wagner J, Wang J (2003) The systems biology markup language (SBML): a medium for representation and exchange of biochemical network models. Bioinformatics 19(4):524–531. CrossRefGoogle Scholar
  16. Le Novère N, Bornstein B, Broicher A, Courtot M, Donizelli M, Dharuri H, Li L, Sauro H, Schilstra M, Shapiro B, Snoep JL, Hucka M (2006) BioModels database: a free, centralized database of curated, published, quantitative kinetic models of biochemical and cellular systems. Nucleic Acids Res 34(Database issue):D689–91 10.1093/nar/gkj092CrossRefGoogle Scholar
  17. Le Novère N, Finney A, Hucka M, Bhalla US, Campagne F, Collado-Vides J, Crampin EJ, Halstead M, Klipp E, Mendes P, Nielsen P, Sauro H, Shapiro B, Snoep JL, Spence HD, Wanner BL (2005) Minimum information requested in the annotation of biochemical models (MIRIAM). Natute Biotechnol 23(12):1509–15. CrossRefGoogle Scholar
  18. Le Novère N, Hucka M, Mi H, Moodie S, Schreiber F, Sorokin A, Demir E, Wegner K, Aladjem MI, Wimalaratne SM, Bergman FT, Gauges R, Ghazal P, Kawaji H, Li L, Matsuoka Y, Villéger A, Boyd SE, Calzone L, Courtot M, Dogrusoz U, Freeman TC, Funahashi A, Ghosh S, Jouraku A, Kim S, Kolpakov F, Luna A, Sahle S, Schmidt E, Watterson S, Wu G, Goryanin I, Kell DB, Sander C, Sauro H, Snoep JL, Kohn K, Kitano H (2009) The systems biology graphical notation. Nat Biotechnol 27(8):735–741. CrossRefGoogle Scholar
  19. Lloyd CM, Halstead MD, Nielsen PF (2004) CellML: its future, present and past. Prog Biophys Mol Biol 85(2–3):433–450. CrossRefGoogle Scholar
  20. Loew L, Beckett D, Egelman EH, Scarlata S (2015) Reproducibility of research in biophysics. Biophys J 108(7):E1. CrossRefGoogle Scholar
  21. Maddox J, Randi J, Stewart WW (1988) High-dilution experiments a delusion. Nature 334(6180):287–290. CrossRefGoogle Scholar
  22. Mesirov JP (2010) Accessible reproducible research. Science 327(5964):415–416. CrossRefGoogle Scholar
  23. Moraru I, Morgan F, Li Y, Loew L, Schaff J, Lakshminarayana A, Slepchenko B, Gao F, Blinov M (2008) Virtual cell modelling and simulation software environment. IET Syst Biol 2(5):352–362. CrossRefGoogle Scholar
  24. Pahle J, Challenger JD, Mendes P, McKane AJ (2012) Biochemical fluctuations, optimisation and the linear noise approximation. BMC Syst Biol 6(1):86. CrossRefGoogle Scholar
  25. Peng RD (2009) Reproducible research and biostatistics. Biostatistics 10(3):405–408. CrossRefGoogle Scholar
  26. Peng RD (2011) Reproducible research in computational science. Science 334(6060):1226–1227. CrossRefGoogle Scholar
  27. Piccolo SR, Frampton MB (2016) Tools and techniques for computational reproducibility. GigaScience. CrossRefGoogle Scholar
  28. Plesser HE (2018) Reproducibility vs. replicability: a brief history of a confused terminology. Front Neuroinform 11:76. CrossRefGoogle Scholar
  29. Popper K (1959) The logic of scientific discovery. Hutchinson, LondonzbMATHGoogle Scholar
  30. Sandve GK, Nekrutenko A, Taylor J, Hovig E (2013) Ten simple rules for reproducible computational research. PLoS Comput Biol 9(10):e1003285. CrossRefGoogle Scholar
  31. Stodden V, McNutt M, Bailey DH, Deelman E, Gil Y, Hanson B, Heroux MA, Ioannidis JPA, Taufer M (2016) Enhancing reproducibility for computational methods. Science 354(6317):1240–1241. CrossRefGoogle Scholar
  32. Stodden V, Seiler J, Ma Z (2018) An empirical analysis of journal policy effectiveness for computational reproducibility. Proc Natl Acad Sci 115(11):2584–2589. CrossRefGoogle Scholar
  33. Swat M, Moodie S, Wimalaratne S, Kristensen N, Lavielle M, Mari A, Magni P, Smith M, Bizzotto R, Pasotti L, Mezzalana E, Comets E, Sarr C, Terranova N, Blaudez E, Chan P, Chard J, Chatel K, Chenel M, Edwards D, Franklin C, Giorgino T, Glont M, Girard P, Grenon P, Harling K, Hooker A, Kaye R, Keizer R, Kloft C, Kok J, Kokash N, Laibe C, Laveille C, Lestini G, Mentré F, Munafo A, Nordgren R, Nyberg H, Parra-Guillen Z, Plan E, Ribba B, Smith G, Trocóniz I, Yvon F, Milligan P, Harnisch L, Karlsson M, Hermjakob H, Le Novère N (2015) Pharmacometrics markup language (PharmML): opening new perspectives for model exchange in drug development: PharmML - pharmacometrics markup language. CPT: Pharmacometr Syst Pharmacol 4(6):316–319. CrossRefGoogle Scholar
  34. Waltemath D, Adams R, Beard DA, Bergmann FT, Bhalla US, Britten R, Chelliah V, Cooling MT, Cooper J, Crampin EJ, Garny A, Hoops S, Hucka M, Hunter P, Klipp E, Laibe C, Miller AK, Moraru I, Nickerson D, Nielsen P, Nikolski M, Sahle S, Sauro HM, Schmidt H, Snoep JL, Tolle D, Wolkenhauer O, Le Novère N (2011) Minimum information about a simulation experiment (MIASE). PLoS Comput Biol 7(4):e1001122. CrossRefGoogle Scholar
  35. Waltemath D, Adams R, Bergmann FT, Hucka M, Kolpakov F, Miller AK, Moraru II, Nickerson D, Sahle S, Snoep JL, Le Novère N (2011) Reproducible computational biology experiments with SED-ML: the simulation experiment description markup language. BMC Syst Biol 5(1):198. CrossRefGoogle Scholar

Copyright information

© Society for Mathematical Biology 2018

Authors and Affiliations

  1. 1.Center for Quantitative Medicine and Department of Cell BiologyUniversity of Connecticut School of MedicineFarmingtonUSA

Personalised recommendations