The Importance of Standards for Sharing of Computational Models and Data

Abstract

The target article by Lee et al. (in review) highlights the ways in which ongoing concerns about research reproducibility extend to model-based approaches in cognitive science. Whereas Lee et al. focus primarily on the importance of research practices to improve model robustness, we propose that the transparent sharing of model specifications, including their inputs and outputs, is also essential to improving the reproducibility of model-based analyses. We outline an ongoing effort (within the context of the Brain Imaging Data Structure community) to develop standards for the sharing of the structure of computational models and their outputs.

This is a preview of subscription content, access via your institution.

References

  1. Anderson, J. R., Bothell, D., Byrne, M. D., Douglass, S., Lebiere, C., & Qin, Y. (2004). An integrated theory of the mind. Psychological Review, 111, 1036–1060.

    Article  Google Scholar 

  2. Baumgaertner, B., Devezer, B., Buzbas, E.O., and Nardin, L.G. (2018). A model-centric analysis of openness, replication, and reproducibility.

  3. Bekolay, T., Bergstra, J., Hunsberger, E., Dewolf, T., Stewart, T. C., Rasmussen, D., Choo, X., Voelker, A. R., & Eliasmith, C. (2014). Nengo: A python tool for building large-scale functional brain models. Frontiers in Neuroinformatics, 7, 48.

    Article  Google Scholar 

  4. Benureau, F. C. Y., & Rougier, N. P. (2017). Re-run, repeat, reproduce, reuse, replicate: Transforming code into scientific contributions. Frontiers in Neuroinformatics, 11, 69.

    Article  Google Scholar 

  5. Buckheit, J.B., and Donoho, D.L. (1995). WaveLab and reproducible research. In Wavelets and statistics, A. Antoniadis, and G. Oppenheim, eds. (New York, NY: Springer New York), pp. 55–81.

  6. Cannon, R. C., Gleeson, P., Crook, S., Ganapathy, G., Marin, B., Piasini, E., & Silver, R. A. (2014). LEMS: A language for expressing complex biological models in concise and hierarchical form and its use in underpinning NeuroML 2. Frontiers in Neuroinformatics, 8, 79.

    Article  Google Scholar 

  7. Carnevale, N.T., and Hines, M.L. (2006). The NEURON book (Cambridge University Press).

  8. Donkin, C., Brown, S., Heathcote, A., & Wagenmakers, E.-J. (2011). Diffusion versus linear ballistic accumulation: Different models but the same conclusions about psychological processes? Psychonomic Bulletin & Review, 18, 61–69.

    Article  Google Scholar 

  9. Eglen, S. J., Marwick, B., Halchenko, Y. O., Hanke, M., Sufi, S., Gleeson, P., Silver, R. A., Davison, A. P., Lanyon, L., Abrams, M., Wachtler, T., Willshaw, D. J., Pouzat, C., & Poline, J. B. (2017). Toward standard practices for sharing computer code and programs in neuroscience. Nature Neuroscience, 20, 770–773.

    Article  Google Scholar 

  10. Frank, M. J. (2015). Linking across levels of computation in model-based cognitive neuroscience. An introduction to model-based cognitive neuroscience (pp. 159–177).

    Google Scholar 

  11. Fum, D., Del Missier, F., and Stocco, A. (2007). The cognitive modeling of human behavior: Why a model is (sometimes) better than 10,000 words.

  12. Gleeson, P., Davison, A. P., Silver, R. A., & Ascoli, G. A. (2017). A commitment to open source in neuroscience. Neuron, 96, 964–965.

    Article  Google Scholar 

  13. Goldfarb, S., Leonard, N. E., Simen, P., Caicedo-Núñez, C. H., & Holmes, P. (2014). A comparative study of drift diffusion and linear ballistic accumulator models in a reward maximization perceptual choice task. Frontiers in Neuroscience, 8, 148.

    Article  Google Scholar 

  14. Gorgolewski, K. J., Auer, T., Calhoun, V. D., Craddock, R. C., Das, S., Duff, E. P., Flandin, G., Ghosh, S. S., Glatard, T., Halchenko, Y. O., Handwerker, D. A., Hanke, M., Keator, D., Li, X., Michael, Z., Maumet, C., Nichols, B. N., Nichols, T. E., Pellman, J., Poline, J. B., Rokem, A., Schaefer, G., Sochat, V., Triplett, W., Turner, J. A., Varoquaux, G., & Poldrack, R. A. (2016). The brain imaging data structure, a format for organizing and describing outputs of neuroimaging experiments. Scientific Data, 3, 160044.

    Article  Google Scholar 

  15. Navarro, D. J., & Fuss, I. G. (2009). Fast and accurate calculations for first-passage times in Wiener diffusion models. Journal of Mathematical Psychology, 53, 222–230.

    Article  Google Scholar 

  16. Ratcliff, R., & Childers, R. (2015). Individual differences and fitting methods for the two-choice diffusion model of decision making. Decision, 2, 237–279.

    Article  Google Scholar 

  17. Sanz Leon, P., Knock, S. A., Woodman, M. M., Domide, L., Mersmann, J., McIntosh, A. R., & Jirsa, V. (2013). The Virtual Brain: A simulator of primate brain network dynamics. Frontiers in Neuroinformatics, 7, 10.

    Article  Google Scholar 

  18. Turner, B. M., Forstmann, B. U., Love, B. C., Palmeri, T. J., & Van Maanen, L. (2017). Approaches to analysis in model-based cognitive neuroscience. Journal of Mathematical Psychology, 76, 65–79.

    Article  Google Scholar 

Download references

Funding

Development of this article was supported by the BRAIN Initiative and National Institute of Mental Health (1R24MH114705). P.R. funding sources: H2020 VirtualBrainCloud 826421, Human Brain Project 785907 and ERC 683049; German Research Foundation CRC 1315, CRC 936 and RI 2073/6-1; Berlin Institute of Health & Foundation Charité, Johanna Quandt Excellence Initiative.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Russell A Poldrack.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Poldrack, R.A., Feingold, F., Frank, M.J. et al. The Importance of Standards for Sharing of Computational Models and Data. Comput Brain Behav 2, 229–232 (2019). https://doi.org/10.1007/s42113-019-00062-x

Download citation

Keywords

  • Data sharing
  • Standards
  • Reproducibility
  • Computational models