Learning from the Past: Approaches for Reproducibility in Computational Neuroscience

  • Sharon M. Crook
  • Andrew P. Davison
  • Hans E. Plesser
Chapter
Part of the Springer Series in Computational Neuroscience book series (NEUROSCI, volume 9)

Abstract

Reproducible experiments are the cornerstone of science: only observations that can be independently confirmed enter the body of scientific knowledge. Computational science should excel in reproducibility, as simulations on digital computers avoid many of the small variations that are beyond the control of the experimental biologist or physicist. However, in reality, computational science has its own challenges for reproducibility: many computational scientists find it difficult to reproduce results published in the literature, and many authors have met problems replicating even the figures in their own papers. We present a distinction between different levels of replicability and reproducibility of findings in computational neuroscience. We also demonstrate that simulations of neural models can be highly sensitive to numerical details, and conclude that often it is futile to expect exact replicability of simulation results across simulator software packages. Thus, the computational neuroscience community needs to discuss how to define successful reproduction of simulation studies. Any investigation of failures to reproduce published results will benefit significantly from the ability to track the provenance of the original results. We present tools and best practices developed over the past 2 decades that facilitate provenance tracking and model sharing.

References

  1. Bower JM, Beeman D (1997) The book of GENESIS: exploring realistic neural models with the GEneral NEural SImulation System. Springer, New YorkGoogle Scholar
  2. Bhalla U, Bilitch DH, Bower J (1992) Rallpacks: a set of benchmarks for neural simulators. Trends Neurosci 15:453–458PubMedCrossRefGoogle Scholar
  3. Bray T, Paoli J, Sperberg-McQueen C (1998) Extensible markup language (XML) 1.0. http://www.w3.org/TR/REC-xml
  4. Breslow NE (2010) Commentary. Biostatistics 11(3):379–380. doi:10.1093/biostatistics/kxq025 PubMedCrossRefGoogle Scholar
  5. Brette R (2006) Exact simulation of integrate-and-fire models with synaptic conductances. Neural Comput 18:2004–2027PubMedCrossRefGoogle Scholar
  6. Brette R (2007) Exact simulation of integrate-and-fire models with exponential currents. Neural Comput 19:2604–2609PubMedCrossRefGoogle Scholar
  7. Brette R, Rudolph M, Carnevale T, Hines M, Beeman D, Bower JM, Diesmann M, Morrison A, Goodman PH, Harris FC Jr, Zirpe M, Natschläger T, Pecevski D, Ermentrout B, Djurfeldt M, Lansner A, Rochel O, Vieville T, Muller E, Davison AP, Boustani SE, Destexhe A (2007) Simulation of networks of spiking neurons: a review of tools and strategies. J Comput Neurosci 23:349–398PubMedCrossRefGoogle Scholar
  8. Brüderle D, Muller E, Davison A, Muller E, Schemmel J, Meier K (2009) Establishing a novel modeling tool: a Python-based interface for a neuromorphic hardware system. Front Neuroinform 3:17. doi:10.3389/neuro.11.017.2009 PubMedGoogle Scholar
  9. Brunel N (2000) Dynamics of sparsely connected networks of excitatory and inhibitory spiking neurons. J Comput Neurosci 8:183–208PubMedCrossRefGoogle Scholar
  10. Cannon R, Howell F, Goddard N, De Schutter E (2002) Non-curated distributed databases for experimental data and models in neuroscience. Network 13:415–428PubMedCrossRefGoogle Scholar
  11. Cannon RC, Gewaltig MO, Gleeson P, Bhalla US, Cornelis H, Hines ML, Howell FW, Muller E, Stiles JR, Wils S, Schutter ED (2007) Interoperability of neuroscience modeling software: current status and future directions. Neuroinformatics 5:127–138PubMedCrossRefGoogle Scholar
  12. Carnevale N, Hines M (2006) The NEURON book. Cambridge University Press, Cambridge, UKCrossRefGoogle Scholar
  13. Claerbout JF, Karrenbach M (1992) Electronic documents give reproducible research a new meaning. In: SEG expanded abstracts, Society of Exploration Geophysicists, vol 11, pp 601–604Google Scholar
  14. Cox DR, Donnelly C (2010) Commentary. Biostatistics 11(3):381–382. doi:10.1093/biostatistics/kxq026 PubMedCrossRefGoogle Scholar
  15. Craver CF (2007) Explaining the brain: mechanisms and the Mosaic Unity of Neuroscience. Oxford University Press, New YorkCrossRefGoogle Scholar
  16. Crook S, Howell F (2007) XML for data representation and model specification. In: Crasto C (ed)Methods in molecular biology book series: neuroinformatics. Humana Press, Totowa, NJ, pp 53–66Google Scholar
  17. Crook S, Gleeson P, Howell F, Svitak J, Silver R (2007) MorphML: Level 1 of the NeuroML standards for neuronal morphology data and model specification. Neuroinformatics 5:96–104PubMedCrossRefGoogle Scholar
  18. D’Haene M (2010) Efficient simulation strategies for spiking neural networks. PhD thesis, Universiteit GentGoogle Scholar
  19. Davison A, Morse T, Migliore M, Marenco L, Shepherd G, Hines M (2002) ModelDB: a resource for neuronal and network modeling. In: Kötter R (ed) Neuroscience databases: a practical guide. Kluwer Academic, Norwell, MA, pp 99–122Google Scholar
  20. Davison A, Brüderle D, Eppler J, Kremkow J, Muller E, Pecevski D, Perrinet L, Yger P (2009) PyNN: a common interface for neuronal network simulators. Front Neuroinform 2:11. doi:10.3389/neuro.11.011.2008 Google Scholar
  21. de Leeuw J (2001) Reproducible research: the bottom line. Tech. Rep., Department of Statistics, UCLA, UC Los Angeles. http://escholarship.org/uc/item/9050x4r4
  22. DeAngelis CD, Fontanarosa PB (2010) The importance of independent academic statistical analysis. Biostatistics 11(3):383–384. doi:10.1093/biostatistics/kxq027 PubMedCrossRefGoogle Scholar
  23. Djurfeldt M, Lansner A (2007) Workshop report: 1st INCF Workshop on Large-scale Modeling of the Nervous System. Available from Nature Precedings http://dx.doi.org/10.1038/npre.2007.262.1Google Scholar
  24. Donoho DL (2010) An invitation to reproducible computational research. Biostatistics 11(3):385–388. doi:10.1093/biostatistics/kxq028 PubMedCrossRefGoogle Scholar
  25. Donoho DL, Maleki A, Rahman IU, Shahram M, Stodden V (2009) 15 years of reproducible research in computational harmonic analysis. Comput Sci Eng 11:8–18. doi:10.1109/MCSE.2009.15 CrossRefGoogle Scholar
  26. Drummond C (2009) Replicability is not reproducibility: nor is it good science. In: Proceedings of the evaluation methods for machine learning workshop at the 26th ICML, Montreal, CAGoogle Scholar
  27. Eppler JM, Helias M, Muller E, Diesmann M, Gewaltig MO (2008) PyNEST: a convenient interface to the NEST simulator. Front Neuroinform 2:12. doi:10.3389/neuro.11.012.2008 PubMedCrossRefGoogle Scholar
  28. Ferrenberg AM, Landau DP, Wong YJ (1992) Monte Carlo simulations: hidden errors from “good” random number generators. Phys Rev Lett 69(23):3382–3384. doi:10.1103/PhysRevLett.69.3382 PubMedCrossRefGoogle Scholar
  29. Finney A, Hucka M, Bornstein B, Keating S, Shapiro B, Matthews J, Kovitz B, Schilstra M, Funahashi A, Doyle J, Kitano H (2006) Software infrastructure for effective communication and reuse of computational models. In: Szallasi Z, Stelling J, Periwal V (eds) Systems modeling in cell biology: from concepts to nuts and bolts. MIT Press, Boston, pp 369–378Google Scholar
  30. Funk W, Dammann V, Donnevert G (2006) Quality assurance in analytical chemistry, 2nd edn. Wiley-VCH, WeinheimCrossRefGoogle Scholar
  31. Galluppi F, Rast A, Davies S, Furber S (2010) A general-purpose model translation system for a universal neural chip. In: Wong K, Mendis B, Bouzerdoum A (eds) Neural information processing. Theory and algorithms, Lecture notes in computer science, vol 6443. Springer, Berlin, pp 58–65Google Scholar
  32. Gardner D, Knuth KH, Abato M, Erde SM, White T, DeBellis R, Gardner EP (2001) Common data model for neuroscience data and data model exchange. J Am Med Inform Assoc 8:17–33PubMedCrossRefGoogle Scholar
  33. Gardner D, Akil H, Ascoli G, Bowden D, Bug W, Donohue D, Goldberg D, Grafstein B, Grethe J, Gupta A, Halavi M, Kennedy D, Marenco L, Martone M, Miller P, Müller H, Robert A, Shepherd G, Sternberg P, Van Essen D, Williams R (2008) The neuroscience information framework: a data and knowledge environment for neuroscience. Neuroinformatics 6(3):149–160PubMedCrossRefGoogle Scholar
  34. Gewaltig MO, Diesmann M (2007) NEST (NEural Simulation Tool). Scholarpedia 2(4):1430CrossRefGoogle Scholar
  35. Gewaltig MO, Koerner EE (2008) Self-sustained dynamics of sparsely connected networks without external noise. In: 2008 Neuroscience meeting planner, Society for Neuroscience, Washington, DC, program No. 220.1Google Scholar
  36. Gleeson P, Steuber V, Silver RA (2007) neuroConstruct: a tool for modeling networks in 3D space. Neuron 54:219–235PubMedCrossRefGoogle Scholar
  37. Gleeson P, Crook S, Cannon RC, Hines ML, Billings GO, Farinella M, Morse TM, Davison AP, Ray S, Bhalla US, Barnes SR, Dimitrova YD, Silver RA (2010) NeuroML: a language for describing data driven models of neurons and networks with a high degree of biological detail. PLoS Comput Biol 6(6):e1000815. doi:10.1371/journal.pcbi.1000815 PubMedCrossRefGoogle Scholar
  38. Goddard N, Hucka M, Howell F, Cornelis H, Shankar K, Beeman D (2001) NeuroML: model description methods for collaborative modelling in neuroscience. Philos Trans R Soc Lond B Biol Sci 356:1209–1228PubMedCrossRefGoogle Scholar
  39. Goodman D (2010a) Code generation: a strategy for neural network simulators. Neuroinformatics 8:183–196. doi:10.1007/s12021-010-9082-x PubMedCrossRefGoogle Scholar
  40. Goodman SN (2010b) Commentary. Biostatistics 11(3):389–390. doi:10.1093/biostatistics/kxq030 PubMedCrossRefGoogle Scholar
  41. Goodman D, Brette R (2008) Brian: a simulator for spiking neural networks in Python. Front Neuroinform 2:5. doi:10.3389/neuro.11.005.2008 PubMedCrossRefGoogle Scholar
  42. Gorchetchnikov A, The INCF Multiscale Modeling Taskforce (2010) NineML—a description language for spiking neuron network modeling: the user layer. BMC Neurosci 11(suppl 1):P71CrossRefGoogle Scholar
  43. Grebogi C, Hammel SM, Yorke JA, Sauer T (1990) Shadowing of physical trajectories in chaotic dynamics: containment and refinement. Phys Rev Lett 65(13):1527–1530. doi:10.1103/PhysRevLett.65.1527 PubMedCrossRefGoogle Scholar
  44. Groves T (2010) The wider concept of data sharing: view from the BMJ. Biostatistics 11(3):391–392. doi:10.1093/biostatistics/kxq031 PubMedCrossRefGoogle Scholar
  45. Gruntz D, Waldvogel J (2004) Orbits in the planar three-body problem. In: Gander W, Hřebíček J (eds) Solving problems in scientific computing using Maple and MATLAB, 4th edn. Springer, Berlin, pp 51–72CrossRefGoogle Scholar
  46. Hansel D, Mato G, Meunier C, Neltner L (1998) On numerical simulations of integrate-and-fire neural networks. Neural Comput 10:467–483PubMedCrossRefGoogle Scholar
  47. Hanuschkin A, Kunkel S, Helias M, Morrison A, Diesmann M (2010) A general and efficient method for incorporating exact spike times in globally time-driven simulations. Front Neuroinform 4:113. doi:10.3389/fninf.2010.00113 PubMedCrossRefGoogle Scholar
  48. Hedley W, Nelson M, Nielsen P, Bullivant D, Hunter P (2000) XML languages for describing biological models. In: Proceedings of the Physiological Society of New Zealand, vol 19Google Scholar
  49. Heijmen T (2011) Soft errors from space to ground: historical overview, empirical evidence, and future trends (chap 1). In: Nicolaidis M (ed) Soft errors in modern electronic systems, Frontiers in electronic testing, vol 41. Springer, New York, pp 1–25Google Scholar
  50. Hines M (1989) A program for simulation of nerve equations with branching geometries. Int J Biomed Comput 24:55–68PubMedCrossRefGoogle Scholar
  51. Hines ML, Carnevale NT (1997) The NEURON simulation environment. Neural Comput 9(6):1179–1209PubMedCrossRefGoogle Scholar
  52. Hines ML, Carnevale NT (2000) Expanding NEURON’s repertoire of mechanisms with NMODL. Neural Comput 12:995–1007PubMedCrossRefGoogle Scholar
  53. Hines ML, Morse T, Migliore M, Carnevale NT, Shepherd GM (2004) ModelDB: a database to support computational neuroscience. J Comput Neurosci 17(1):7–11. doi:10.1023/B:JCNS.0000023869.22017.2e PubMedCrossRefGoogle Scholar
  54. Hucka M, Finney A, Sauro H, Bolouri H, Doyle J, Kitano H, Arkin A (2003) The systems biology markup language (SBML): a medium for representation and exchange of biochemical network models. Bioinformatics 19:524–531PubMedCrossRefGoogle Scholar
  55. Humphreys P (2004) Extending ourselves: computational science, empiricism, and scientific method. Oxford University Press, OxfordCrossRefGoogle Scholar
  56. Hund F (1996) Geschichte der physikalischen Begriffe. Spektrum Akademischer Verlag, HeidelbergGoogle Scholar
  57. Ioannidis JPA (2005) Why most published research findings are false. PLoS Med 2:e124. doi:10.1371/journal.pmed.0020124 PubMedCrossRefGoogle Scholar
  58. Jones E, Oliphant T, Peterson P et al (2001) SciPy: open source scientific tools for Python. http://www.scipy.org/
  59. Keiding N (2010a) Reproducible research and the substantive context. Biostatistics 11(3):376–378. doi:10.1093/biostatistics/kxq033 PubMedCrossRefGoogle Scholar
  60. Keiding N (2010b) Reproducible research and the substantive context: response to comments. Biostatistics 11(3):395–396. doi:10.1093/biostatistics/kxq034 PubMedCrossRefGoogle Scholar
  61. Kilkenny C, Browne WJ, Cuthill IC, Emerson M, Altman DG (2010) Improving bioscience research reporting: the ARRIVE guidelines for reporting animal research. PLoS Biol 8(6):e1000412. doi:10.1371/journal.pbio.1000412 PubMedCrossRefGoogle Scholar
  62. Köhn D, Le Novère N (2008) SED-ML—an XML format for the implementation of the MIASE guidelines. In: Heiner M, Uhrmacher A (eds) Computational methods in systems biology, Lecture notes in computer science, vol 5307. Springer, Berlin, pp 176–190Google Scholar
  63. Küppers G, Lenhard J (2005) Validation of simulation: patterns in the social and natural sciences. J Artif Soc Soc Simul 8(4):3Google Scholar
  64. Lapicque L (1907) Recherches quantitatives sur l’excitation électrique des nerfs traitée comme une polarization. J Physiol Pathol Gen 9:620–635Google Scholar
  65. Le Novere N, Bornstein B, Broicher A, Courtot M, Donizelli M, Dharuri H, Li L, Sauro H, Schilstra M, Shapiro B, Snoep J, Hucka M (2006) BioModels database: a free, centralized database of curated, published, quantitative kinetic models of biochemical and cellular systems. Nucleic Acids Res 34(Database issue):D689–D691PubMedCrossRefGoogle Scholar
  66. Lehrer J (2010) The truth wears off. The New Yorker, 13 Dec 2010:52Google Scholar
  67. Lloyd C, Halstead M, Nielsen P (2004) CellML: its future, present and past. Prog Biophys Mol Biol 85:433–450PubMedCrossRefGoogle Scholar
  68. Lloyd CM, Lawson JR, Hunter PJ, Nielsen PF (2008) The CellML model repository. Bioinformatics 24(18):2122–2123. doi:10.1093/bioinformatics/btn390 PubMedCrossRefGoogle Scholar
  69. Ludäscher B, Altintas I, Berkley C, Higgins D, Jaeger-Frank E, Jones M, Lee E, Tao J, Zhao Y (2006) Scientific workflow management and the Kepler system. Concurrency Comput Pract Exp 18(10):1039–1065CrossRefGoogle Scholar
  70. Marenco L, Wang R, Shepherd G, Miller P (2010) The NIF DISCO framework: facilitating automated integration of neuroscience content on the web. Neuroinformatics 8(2):101–112PubMedCrossRefGoogle Scholar
  71. Maunsell J (2010) Announcement regarding supplemental material. J Neurosci 30:10599–10600Google Scholar
  72. Mesirov JP (2010) Accessible reproducible research. Science 327(5964):415–416. doi:10.1126/science.1179653 PubMedCrossRefGoogle Scholar
  73. Migliore M, Morse TM, Davison AP, Marenco L, Shepherd GM, Hines ML (2003) ModelDB: making models publicly accessible to support computational neuroscience. Neuroinformatics 1(1):135–139PubMedCrossRefGoogle Scholar
  74. Miner R (2005) The importance of MathML to mathematical communication. Notices AMS 52:532–538Google Scholar
  75. Morrison A, Straube S, Plesser HE, Diesmann M (2007) Exact subthreshold integration with continuous spike times in discrete time neural network simulations. Neural Comput 19:47–79PubMedCrossRefGoogle Scholar
  76. Morse TM (2007) Model sharing in computational neuroscience. Scholarpedia 2(4):3036CrossRefGoogle Scholar
  77. Nordlie E, Plesser HE (2010) Visualizing neuronal network connectivity with connectivity pattern tables. Front Neuroinform 3:39. doi:10.3389/neuro.11.039.2009 PubMedGoogle Scholar
  78. Nordlie E, Gewaltig MO, Plesser HE (2009) Towards reproducible descriptions of neuronal network models. PLoS Comput Biol 5(8):e1000456. doi:10.1371/journal.pcbi.1000456 PubMedCrossRefGoogle Scholar
  79. Oinn T, Greenwood M, Addis M, Alpdemir M, Ferris J, Glover K, Goble C, Goderis A, Hull D, Marvin D, Li P, Lord P, Pocock MR, Senger M, Stevens R, Wipat A, Wroe C (2006) Taverna: lessons in creating a workflow environment for the life sciences. Concurr Comput Pract Exp 18(10):1067–1100. doi:10.1002/cpe.993 CrossRefGoogle Scholar
  80. Pecevski D, Natschläger T, Schuch K (2009) PCSIM: a parallel simulation environment for neural circuits fully integrated with Python. Front Neuroinform 3:11. doi:10.3389/neuro.11.011.2009 PubMedCrossRefGoogle Scholar
  81. Peng RD (2009) Reproducible research and biostatistics. Biostatistics 10(3):405–408. doi:10.1093/biostatistics/kxp014 PubMedCrossRefGoogle Scholar
  82. Peng RD (2010) Discussion of Keiding. Biostatistics 11(3):393–394. doi:10.1093/biostatistics/kxq032 PubMedCrossRefGoogle Scholar
  83. Peterson B, Healy M, Nadkarni P, Miller P, Shepherd G (1996) ModelDB: an environment for running and storing computational models and their results applied to neuroscience. J Am Med Inform Assoc 3:389–398PubMedCrossRefGoogle Scholar
  84. Petzold L (1983) Automatic selection of methods for solving stiff and nonstiff systems of ordinary differential equations. SIAM J Sci Comput 4:136–148. doi:10.1137/0904010 CrossRefGoogle Scholar
  85. Plesser HE, Austvoll K (2009) Specification and generation of structured neuronal network models with the NEST topology module. BMC Neurosci 10(suppl 1):P56. doi:10.1186/1471-2202-10-S1-P56 CrossRefGoogle Scholar
  86. Quirk JJ (2005) Computational science: “same old silence, same old mistakes; something more is needed”. In: Plewa T, Linde T, Weirs VG (eds) Adaptive mesh refinement—theory and applications. Springer, Berlin, pp 3–28CrossRefGoogle Scholar
  87. Raikov I, The INCF Multiscale Modeling Taskforce (2010) NineML—a description language for spiking neuron network modeling: the abstraction layer. BMC Neurosci 11(suppl 1):P66CrossRefGoogle Scholar
  88. Rotter S, Diesmann M (1999) Exact digital simulation of time-invariant linear systems with applications to neuronal modeling. Biol Cybern 81:381–402PubMedCrossRefGoogle Scholar
  89. Schrödinger E (1915) Zur Theorie der Fall-und Steigversuche an Teilchen mit Brownscher Bewegung. Phys Zeit 16:289–295Google Scholar
  90. Shelley MJ, Tao L (2001) Efficient and accurate time-stepping schemes for integrate-and-fire neuronal networks. J Comput Neurosci 11:111–119PubMedCrossRefGoogle Scholar
  91. Silva CT, Freire J, Callahan S (2007) Provenance for visualizations: reproducibility and beyond. Comput Sci Eng 9(5):82–90CrossRefGoogle Scholar
  92. Stodden V (2009a) Enabling reproducible research: open licensing for scientific innovation. Int J Commun Law Policy 13:2–25, http://www.ijclp.net/files/ijclp_web-doc_1-13-2009.pdf Google Scholar
  93. Stodden V (2009b) The legal framework for reproducible research in the sciences: licensing and copyright. IEEE Comput Sci Eng 11(1):35–40CrossRefGoogle Scholar
  94. Usui S (2003) Visiome: neuroinformatics research in vision project. Neural Netw 16:1293–1300PubMedCrossRefGoogle Scholar
  95. Vandewalle P, Kovačević J, Vetterli M (2009) Reproducible research in signal processing. IEEE Signal Proc Mag 26(3):37–47. doi:10.1109/MSP.2009.932122 CrossRefGoogle Scholar
  96. Wilson MA, Bhalla US, Uhley JD, Bower JM (1989) GENESIS: a system for simulating neural networks. In: Touretzky D (ed) Advances in neural information processing systems. Morgan Kaufmann, San Mateo, CA, pp 485–492Google Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  • Sharon M. Crook
    • 1
  • Andrew P. Davison
    • 2
  • Hans E. Plesser
    • 3
  1. 1.School of Mathematical and Statistical Sciences and School of Life Sciences, Center for Adaptive Neural SystemsArizona State UniversityTempeUSA
  2. 2.Unité de Neuroscience Information et Complexité (UNIC), CNRSGif-sur-YvetteFrance
  3. 3.Department of Mathematical Sciences and TechnologyNorwegian University of Life SciencesAasNorway

Personalised recommendations