Reproducible Pattern Recognition Research: The Case of Optimistic SSL

  • Jesse H. Krijthe
  • Marco Loog
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10214)


In this paper, we discuss the approaches we took and trade-offs involved in making a paper on a conceptual topic in pattern recognition research fully reproducible. We discuss our definition of reproducibility, the tools used, how the analysis was set up, show some examples of alternative analyses the code enables and discuss our views on reproducibility.


Reproducibility Pattern recognition Semi-supervised learning 



This work was funded by project P23 of the Dutch public/private research network COMMIT.


  1. 1.
    Donoho, D.L., Maleki, A., Rahman, I.U., Shahram, M., Stodden, V.: Reproducible research in computational harmonic analysis. Comput. Sci. Eng. 11(1), 8–18 (2009)CrossRefGoogle Scholar
  2. 2.
    Drummond, C.: Replicability is not reproducibility: nor is it good science. In: Proceedings of the Evaluation Methods for Machine Learning Workshop at the 26th ICML (2009)Google Scholar
  3. 3.
    Goodman, S.N., Fanelli, D., Ioannidis, J.P.A.: What does research reproducibility mean? Sci. Transl. Med. 8(341), 341ps12 (2016)CrossRefGoogle Scholar
  4. 4.
    Healy, M., Westmacott, M.: Missing values in experiments analysed on automatic computers. J. Roy. Stat. Soc. 5(3), 203–206 (1956)Google Scholar
  5. 5.
    Herndon, T., Ash, M., Pollin, R.: Does high public debt consistently stifle economic growth? A critique of reinhart and rogoff. Cambridge J. Econ. 38(2), 257–279 (2014)CrossRefGoogle Scholar
  6. 6.
    Krijthe, J.H.: RSSL: semi-supervised Learning in R (2016).
  7. 7.
    Krijthe, J.H., Loog, M.: Optimistic semi-supervised least squares classification. In: Proceedings of the 23rd International Conference on Pattern Recognition (2016)Google Scholar
  8. 8.
    Open Science Collaboration: Estimating the reproducibility of psychological science. Science 349(6251) (2015).
  9. 9.
    Patil, P., Peng, R.D., Leek, J.T.: A statistical definition for reproducibility and replicability. Biorxiv (2016).
  10. 10.
    R Core Team: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria (2016).
  11. 11.
    Wickham, H.: R Packages. O’Reilly Media, Sebastopol (2015)Google Scholar
  12. 12.
    Xie, Y.: knitr: a comprehensive tool for reproducible research in R. In: Stodden, V., Leisch, F., Peng, R.D. (eds.) Implementing Reproducible Research. CRC Press, Boca Raton (2014)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Pattern Recognition LaboratoryDelft University of TechnologyDelftNetherlands
  2. 2.Department of Molecular EpidemiologyLeiden University Medical CenterLeidenNetherlands
  3. 3.The Image SectionUniversity of CopenhagenCopenhagenDenmark

Personalised recommendations