Truth, Proof, and Reproducibility: There’s No Counter-Attack for the Codeless

  • Charles T. GrayEmail author
  • Ben Marwick
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 1150)


Current concerns about reproducibility in many research communities can be traced back to a high value placed on empirical reproducibility of the physical details of scientific experiments and observations. For example, the detailed descriptions by 17th century scientist Robert Boyle of his vacuum pump experiments are often held to be the ideal of reproducibility as a cornerstone of scientific practice. Victoria Stodden has claimed that the computer is an analog for Boyle’s pump – another kind of scientific instrument that needs detailed descriptions of how it generates results. In the place of Boyle’s hand-written notes, we now expect code in open source programming languages to be available to enable others to reproduce and extend computational experiments. In this paper we show that there is another genealogy for reproducibility, starting at least from Euclid, in the production of proofs in mathematics. Proofs have a distinctive quality of being necessarily reproducible, and are the cornerstone of mathematical science. However, the task of the modern mathematical scientist has drifted from that of blackboard rhetorician, where the craft of proof reigned, to a scientific workflow that now more closely resembles that of an experimental scientist. So, what is proof in modern mathematics? And, if proof is unattainable in other fields, what is due scientific diligence in a computational experimental environment? How do we measure truth in the context of uncertainty? Adopting a manner of Lakatosian conversant conjecture between two mathematicians, we examine how proof informs our practice of computational statistical inquiry. We propose that a reorientation of mathematical science is necessary so that its reproducibility can be readily assessed.


Metaresearch Reproducibility Mathematics 


  1. 1.
    Amrhein, V., Greenland, S., McShane, B.: Scientists rise up against statistical significance. Nature 567(7748), 305 (2019).
  2. 2.
    Auburn, D.: Proof: A Play. Farrar, Straus and Giroux, New York (2001). Google-Books-ID: 6AUtQVhrY90CCrossRefGoogle Scholar
  3. 3.
    Bertot, Y.: A short presentation of Coq. In: Mohamed, O.A., Muñoz, C., Tahar, S. (eds.) TPHOLs 2008. LNCS, vol. 5170, pp. 12–16. Springer, Heidelberg (2008). Scholar
  4. 4.
    Brown, S.: Partial unpacking and indirect proofs: a study of students’ productive use of the symbolic proof scheme. In: Proceedings of the 16th Annual Conference on Research in Undergraduate Mathematics Education, vol. 2, pp. 47–54 (2013)Google Scholar
  5. 5.
    Bryan, J.: Excuse me, do you have a moment to talk about version control? Am. Stat. 72(1), 20–27 (2018).
  6. 6.
    Camerer, C.F., et al.: Evaluating replicability of laboratory experiments in economics. Science 351(6280), 1433–1436 (2016).
  7. 7.
    Carroll, L.: The Annotated Alice: The, Definitive Edition, updated, subsequent edn. W. W. Norton & Company, New York (1999)Google Scholar
  8. 8.
    Davey, B.A., Priestley, H.A.: Introduction to Lattices and Order. Cambridge University Press, Cambridge (2002). Google-Books-ID: vVVTxeuiyvQCCrossRefGoogle Scholar
  9. 9.
    Davey, B.A.: When is a Proof?, 2nd edn. La Trobe University, Bundoora (2009)Google Scholar
  10. 10.
    Davey, B.A., Gray, C.T., Pitkethly, J.G.: The homomorphism lattice induced by a finite algebra. Order 35(2), 193–214 (2018).
  11. 11.
    Donoho, D.L.: An invitation to reproducible computational research. Biostatistics 11(3), 385–388 (2010).
  12. 12.
    Fidler, F., Wilcox, J.: Reproducibility of scientific results. In: Zalta, E.N. (ed.) The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, winter 2018 edn. (2018)Google Scholar
  13. 13.
    Fraser, H., Parker, T., Nakagawa, S., Barnett, A., Fidler, F.: Questionable research practices in ecology and evolution. PLOS One 13(7), e0200303 (2018).
  14. 14.
    Haack, S.: Defending Science - within Reason: Between Scientism and Cynicism. Prometheus Books, Buffalo (2011). Google-Books-ID: RhXxaPTc\(\_\)EYCGoogle Scholar
  15. 15.
    Wickham, H.: Tidy data. J. Stat. Softw. 59(1), 1–23 (2014).
  16. 16.
    Hayes, A.: testing statistical software - aleatoric, July 2019.
  17. 17.
    Head, M.L., Holman, L., Lanfear, R., Kahn, A.T., Jennions, M.D.: The extent and consequences of p-hacking in science. PLOS Biol. 13(3), e1002106 (2015).
  18. 18.
    Hester, J.: covr: Bringing test coverage to R, January 2016.
  19. 19.
    Hester, J.: covr: Test Coverage for Packages (2018).
  20. 20.
    Katz, D.S., McHenry, K.: Super RSEs: Combining research and service in three dimensions of Research Software Engineering, July 2019.
  21. 21.
    Lakatos, I.: Proofs and Refutations: The Logic of Mathematical Discovery, reissue edn. Cambridge University Press, Cambridge (2015)Google Scholar
  22. 22.
    LeVeque, R.J., Mitchell, I.M., Stodden, V.: Reproducible research for scientific computing: tools and strategies for changing the culture. Comput. Sci. Eng. 14 (2012).
  23. 23.
    Martin-Löf, P.: Constructive mathematics and computer programming. In: Cohen, L.J., Łoś, J., Pfeiffer, H., Podewski, K.P. (eds.) Studies in Logic and the Foundations of Mathematics, Logic, Methodology and Philosophy of Science VI, vol. 104, pp. 153–175. Elsevier (1982).
  24. 24.
    Marwick, B.: rrtools: Creates a reproducible research compendium (2018).
  25. 25.
    Marwick, B., Boettiger, C., Mullen, L.: Packaging data analytical work reproducibly using R (and friends). Technical report e3192v2, PeerJ Inc., March 2018.,
  26. 26.
    Merton, R.K.: On Social Structure and Science. University of Chicago Press, Chicago (1996). Google-Books-ID: j94XiVDwAZECGoogle Scholar
  27. 27.
    Murray, C.: How to accuse the other guy of lying with statistics. Stat. Sci. 20(3), 239–241 (2005). Scholar
  28. 28.
    Nowogrodzki, A.: How to support open-source software and stay sane. Nature 571, 133 (2019).
  29. 29.
    Parker, H.: Opinionated analysis development. preprint (2017).
  30. 30.
    Peng, R.D.: Reproducible research in computational science. Science 334(6060), 1226–1227 (2011). Scholar
  31. 31.
    Pickering, A.: The Mangle of Practice: Time, Agency, and Science. University of Chicago Press, Chicago (2010)zbMATHGoogle Scholar
  32. 32.
    Robinson, D., Hayes, A.: broom: Convert Statistical Analysis Objects into Tidy Tibbles (2019).
  33. 33.
    Rodriguez-Sanchez, F., Pérez-Luque, A.J., Bartomeus, I., Varela, S.: Ciencia reproducible: qué, por qué, cómo. Revista Ecosistemas 25(2), 83–92-92 (2016).,
  34. 34.
    Shapin, S., Schaffer, S.: Leviathan and the Air-Pump: Hobbes, Boyle, and the Experimental Life (New in paper), vol. 32. Princeton University Press, Princeton (2011)CrossRefGoogle Scholar
  35. 35.
    Stodden, V.: What scientific idea is ready for retirement? (2014).
  36. 36.
    Stodden, V., Borwein, J., Bailey, D.H.: “Setting the default to reproducible” in computational science research. SIAM News 46(5), 4–6 (2013)Google Scholar
  37. 37.
    Sørensen, M.H., Urzyczyn, P.: Lectures on the Curry-Howard Isomorphism, vol. 149. Elsevier, Amsterdam (2006) zbMATHGoogle Scholar
  38. 38.
    Wallach, J.D., Boyack, K.W., Ioannidis, J.P.A.: Reproducible research practices, transparency, and open access data in the biomedical literature, 2015–2017. PLOS Biol. 16(11), e2006930 (2018).
  39. 39.
    Westgate, M., et al.: metaverse: Workflows for evidence synthesis projects (2019)., r package version 0.0.1
  40. 40.
    Wickham, H.: R Packages: Organize, Test, Document, and Share Your Code. O’Reilly Media, Sebastopol (2015).
  41. 41.
    Wickham, H.: testthat: Get Started with Testing (2011)Google Scholar
  42. 42.
    Wickham, H.: tidyverse: Easily Install and Load the ‘Tidyverse’ (2017).
  43. 43.
    Wilson, G., et al.: Best practices for scientific computing. PLoS Biol. 12(1), e1001745 (2014).
  44. 44.
    Wilson, G., Bryan, J., Cranston, K., Kitzes, J., Nederbragt, L., Teal, T.K.: Good enough practices in scientific computing. PLOS Comput. Biol. 13(6), e1005510 (2017).
  45. 45.
    Wyatt, C.: Research Software Engineers Association (2019).
  46. 46.
    Zeileis, A.: CRAN task views. R News 5(1), 39–40 (2005). Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2019

Authors and Affiliations

  1. 1.La Trobe UniversityMelbourneAustralia
  2. 2.University of WashingtonSeattleUSA

Personalised recommendations