Epistemic issues in computational reproducibility: software as the elephant in the room

Hocquet, Alexandre; Wieber, Frédéric

doi:10.1007/s13194-021-00362-9

Paper in Historical and Social Studies of Science
Published: 17 April 2021

Epistemic issues in computational reproducibility: software as the elephant in the room

European Journal for Philosophy of Science volume 11, Article number: 38 (2021) Cite this article

416 Accesses
3 Citations
11 Altmetric
Metrics details

Abstract

Computational reproducibility (i.e. issues of reproducibility stemming from the computer as a scientific tool) possesses its own dynamics and narratives of crisis. Alongside the difficulties of computing as an ubiquitous yet complex scientific activity, computational reproducibility suffers from a naive expectancy of total reproducibility and a moral imperative to embrace the principles of free software as a non-negotiable epistemic virtue. We argue that the epistemic issues at stake in actual practices of computational reproducibility are best unveiled by focusing on software as a pivotal concept, one that is surprisingly often overlooked in accounts of reproducibility issues. Software is not only about designing and coding but also about maintaining, supporting, distributing, licensing, and governance; it is not only about developers but also about users. We focus on openness debates among computational chemists involved in molecular modeling software packages as empirical grounding for our argument. We then identify and analyse four epistemic characteristics (transparency, consistency, sustainability and inclusivity) as key to the role of software in computational reproducibility.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Notes

As per the wording of an anonymous reviewer.
On computational templates and computational tractability, see Humphreys (2004).
We thank a anonymous reviewer for suggesting “documentation” as yet another layer. While we agree that a variety of epistemic concerns may arise from such a layer, we have chosen, based on our case study corpus, to account for the documentation layer within what we have named the “package” layer.

References

AlNoamany, Y., & Borghi, J. A. (2018). Towards computational reproducibility: Researcher perspectives on the use and sharing of software. PeerJ Computer Science, 4: e163. https://peerj.com/articles/cs-163. 31 Dec 2019.
Atmanspacher, H., & Maasen, S. (2016). Reproducibility: Principles, problems, practices, and prospects. Wiley-Blackwell.
Book Google Scholar
Baker, M. (2016). 1,500 Scientists lift the lid on reproducibility. Nature News, 533(7604): 452. http://www.nature.com/news/1-500-scientists-lift-the-lid-on-reproducibility-1.19970. 31 Dec 2019.
Bhandari Neupane, J., Neupane, R. P., Luo, Y., Yoshida, W. Y., Sun, R., & Williams, P. G. (2019). Characterization of leptazolines A-D, polar oxazolines from the cyanobacterium leptolyngbya sp., reveals a glitch with the “Willoughby–Hoye” scripts for calculating NMR chemical shifts. Organic Letters, 21(20), 8449–8453. https://doi.org/10.1021/acs.orglett.9b03216.
Article Google Scholar
Benureau, F. C. Y., & Rougier, N. P. (2018). Re-Run, Repeat, Reproduce, Reuse, Replicate: Transforming code into scientific contributions. Frontiers in Neuroinformatics, 11. https://www.frontiersin.org/articles/https://doi.org/10.3389/fninf.2017.00069/full. 31 Dec 2019.
Chue Hong, N., Hettrick S., Jones, A., & Katz, D. (2015). The price of open-source software – a joint response. Software Sustainability Institute blog. https://www.software.ac.uk/blog/2016-09-22-price-open-source-software-joint-response. 31 Dec 2019.
Ensmenger, N. L. (2010). The computer boys take over: computers, programmers, and the politics of technical expertise. The MIT Press.
Book Google Scholar
Geiger, S., Howard, D. R., Irani, L., Varoquaux, N., Paxton, A., & Holdgraf, C. (2019). “Who pays the costs of free and open-source scientific software?” 4S annual meeting. http://tinyurl.com/y2jqs4fb. 31 Dec 2019.
Gelfert, Axel. (2011). Scientific models, simulation, and the experimenter’s regress. In Paul Humphreys & Cyrille Imbert (Eds.), Models, Simulations, and Representations. (pp. 145–167). Routledge.
Google Scholar
Gezelter, J. D. (2015). Open source and open data should be standard practices. The Journal of Physical Chemistry Letters, 6(7), 1168–69. https://doi.org/10.1021/acs.jpclett.5b00285. 31 Dec 2019.
Article Google Scholar
Hatton, L., & van Genuchten, M. (2019). Computational reproducibility: The elephant in the room. IEEE Software, 36(2), 137–144. https://doi.org/10.1109/MS.2018.2883805.
Article Google Scholar
Hey, T., & Payne, M. C. (2015). Open science decoded. Nature Physics, 11(5): 367–69. https://www.nature.com/articles/nphys3313. 31 Dec 2019.
Hinsen, K. (2014). Computational science: shifting the focus from tools to models [version 2; peer review: 2 approved]. F1000Research 2014, 3, 101. https://doi.org/10.12688/f1000research.3978.2
Hinsen, K., & Rougier, N. (2019). Challenge to test reproducibility of old computer code. Nature, 574(7780): 634–634. https://www.nature.com/articles/d41586-019-03296-8. 31 Dec 2019.
Hocquet, A., & Wieber, F. (2017). ‘Only the initiates will have the secrets revealed’: Computational chemists and the openness of scientific software. IEEE Annals of the History of Computing, 39(4), 40–58. https://doi.org/10.1109/MAHC.2018.1221048.
Horner, J., & Symons, J. (2014). Reply to Angius and Primiero on software intensive science. Philosophy & Technology, 27(3), 491–494. https://doi.org/10.1007/s13347-014-0172-9.
Article Google Scholar
Humphreys, P. (2004). Extending ourselves: Computational science, empiricism, and scientific method. Oxford University Press.
Book Google Scholar
Jacob, C. R. (2016). How open is commercial scientific software? The Journal of Physical Chemistry Letters, 7(2), 351–53. https://doi.org/10.1021/acs.jpclett.5b02609. 31 Dec 2019.
Article Google Scholar
Kelty, Christopher M. (2008). Two bits: The cultural significance of free software. Duke University Press.
Book Google Scholar
Krylov, A. I., Herbert, J. M., Filipp, F., Head-Gordon, M., Knowles, P. J., Lindh, R., Manby, F. R., Pulay, P., Skylaris, C.-K., & Werner, H.-J. (2015). What is the price of open-source software? The Journal of Physical Chemistry Letters, 6(14), 2751–2754. https://doi.org/10.1021/acs.jpclett.5b01258. 31 Dec 2019.
Article Google Scholar
Lejaeghere, K., Bihlmayer, G., Björkman, T., Blaha, P., Blügel, S., Blum, V., Caliste, D., Castelli, I. E., Clark, S. J., Dal Corso, A., de Gironcoli, S., Deutsch, T., Dewhurst, J. K., Di Marco, I., Draxl, C., Dułak, M., Eriksson, O., Flores-Livas, J. A., Garrity, K. F., & Cottenier, S. (2016). Reproducibility in density functional theory calculations of solids. Science (New York, N.Y.), 351(6280), aad3000. https://doi.org/10.1126/science.aad3000.
Article Google Scholar
Lenhard, J., & Küster, U. (2019). Reproducibility and the concept of numerical solution. Minds and Machines, 29(1), 19–36. https://doi.org/10.1007/s11023-019-09492-9.
Article Google Scholar
Leonelli, S. (2019). Rethinking reproducibility as a criterion for research quality. In Including a Symposium on Mary Morgan: Curiosity, Imagination, and Surprise, Research in the History of Economic Thought and Methodology, Emerald Publishing Limited, 129–146. https://doi.org/10.1108/S0743-41542018000036B009. 31 Dec 2019.
Mahoney, M. S. (2008). What makes the history of software hard. IEEE Annals of the History of Computing, 30(3), 8–18. https://doi.org/10.1109/MAHC.2008.55.
Article Google Scholar
Miletić, V. (2015). “What is the price of open-source fear, uncertainty, and doubt?” nudged elastic band is my band name. https://nudgedelastic.band/2015/09/what-is-the-price-of-open-source-fear-uncertainty-and-doubt/. 31 Dec 2019.
Peng, R. D. (2011). Reproducible research in computational science. Science, 334(6060): 1226–27. https://science.sciencemag.org/content/334/6060/1226. 31 Dec 2019.
Reinhardt, C. (2001). Chemical sciences in the 20th century: Bridging boundaries. Weinheim; New York: Wiley-VCH
Reinhardt, C. (2006). A lead user of instruments in Science: John D. Roberts and the adaptation of nuclear magnetic resonance to organic Chemistry, 1955–1975. Isis, 97(2): 205–36. https://www.journals.uchicago.edu/doi/abs/https://doi.org/10.1086/504732. 29 Nov 2019.
Schappals, M., Mecklenfeld, A., Kröger, L., Botan, V., Köster, A., Stephan, S., García, E. J., Rutkai, G., Raabe, G., Klein, P., Leonhard, K., Glass, C. W., Lenhard, J., Vrabec, J., & Hasse, H. (2017). Round robin study: Molecular simulation of thermodynamic properties from models with internal degrees of freedom. Journal of Chemical Theory and Computation, 13(9), 4270–4280. https://doi.org/10.1021/acs.jctc.7b00489.
Article Google Scholar
Spencer, M. (2015). Brittleness and bureaucracy: Software as a material for Science. Perspectives on Science, 23(4), 466–84. https://doi.org/10.1162/POSC_a_00184. 31 Dec 2019.
Article Google Scholar
Stodden, V., McNutt, M., Bailey, D. H., Deelman, E., Gil, Y., Hanson, B., Heroux, M., A., Ioannidis, J. P. A., & Taufer, M. (2016). “Enhancing reproducibility for computational methods.” Science, 354(6317): 1240–41. https://science.sciencemag.org/content/354/6317/1240. 31 Dec 2019.
Symons, J., & Alvarado, R. (2019). Epistemic entitlements and the practice of computer simulation. Minds and Machines, 29(1), 37–60. https://doi.org/10.1007/s11023-018-9487-0.
Article Google Scholar
Wieber, F., & Hocquet, A. (2020). Models, parameterization, and software: Epistemic opacity in computational chemistry. Perspectives on Science, 28(5), 610–629. https://doi.org/10.1162/posc_a_00352.
Winsberg, E. (2010). Science in the Age of Computer Simulation. University of Chicago Press.
Book Google Scholar
Winsberg, E. (2019). “Computer Simulations in Science”, The Stanford Encyclopedia of Philosophy (Winter 2019 Edition), Edward N. Zalta (ed.). https://plato.stanford.edu/archives/win2019/entries/simulations-science/.

Download references

Acknowledgements

The authors gratefully acknowledge the Science History Institute (Philadelphia) for a fellowship during which some of this research had been pursued. This research project is being supported by a grant from the MSH Lorraine, France.

We are indebted to both anonymous reviewers for their valuable suggestions, which have clearly improved, in our view, this paper.

Author information

Authors and Affiliations

Archives Poincaré, UMR 7117 CNRS & Université de Lorraine, 91 avenue de la Libération, 54001, Nancy, France
Alexandre Hocquet & Frédéric Wieber

Authors

Alexandre Hocquet
View author publications
You can also search for this author in PubMed Google Scholar
Frédéric Wieber
View author publications
You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alexandre Hocquet.

Ethics declarations

Ethical approval

For this type of study formal consent is not required.

Informed consent

For this type of study formal consent is not required.

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This article belongs to the Topical Collection: Philosophical Perspectives on the Replicability Crisis, Guest Editors: Mattia Andreoletti, Jan Sprenger

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Hocquet, A., Wieber, F. Epistemic issues in computational reproducibility: software as the elephant in the room. Euro Jnl Phil Sci 11, 38 (2021). https://doi.org/10.1007/s13194-021-00362-9

Download citation

Received: 08 January 2020
Accepted: 19 March 2021
Published: 17 April 2021
DOI: https://doi.org/10.1007/s13194-021-00362-9

Keywords

Computational reproducibility
Software
Computational chemistry
Transparency
Consistency
Sustainability
Inclusivity