Stepping Stones to Reproducible Research: A Study of Current Practices in Parallel Computing

  • Alexandra Carpen-Amarie
  • Antoine Rougier
  • Felix D. Lübbe
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8805)


Experimental research plays an important role in parallel computing, as in this field scientific discovery often relies on experimental findings, which complement and validate theoretical models. However, parallel hardware and applications have become extremely complex to study, due to their diversity and rapid evolution. Furthermore, applications are designed to run on thousands of nodes, often spanning across several programming models and generating large amounts of data. In this context, reproducibility is essential to foster reliable scientific results. In this paper we aim at studying the requirements and pitfalls of each stage of experimental research, from data acquisition to data analysis, with respect to achieving reproducible results. We investigate state-of-the-art experimental practices in parallel computing by conducting a survey on the papers published in EuroMPI 2013, a major conference targeting the MPI community. Our findings show that while there is a clear concern for reproducibility in the parallel computing community, a better understanding of the criteria for achieving it is necessary.


Source Code Parallel Computing Data Analysis Phase Reproducible Experiment Reproducible Research 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bonnet, P., Manegold, S., Bjørling, M., et al.: Repeatability and workability evaluation of SIGMOD 2011. SIGMOD Record 40(45), 48 (2011)Google Scholar
  2. 2.
    Collberg, C., Proebsting, T., et al.: Measuring Reproducibility in Computer Systems Research (2014),
  3. 3.
    Freire, J., Bonnet, P., Shasha, D.: Computational reproducibility: State-of-the-art, challenges, and database research opportunities. In: Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data, SIGMOD 2012, pp. 593–596. ACM, New York (2012)CrossRefGoogle Scholar
  4. 4.
    Hunold, S., Träff, J.L.: On the state and importance of reproducible experimental research in parallel computing. CoRR abs/1308.3648 (2013)Google Scholar
  5. 5.
    Manolescu, I., Afanasiev, L., Arion, A., et al.: The repeatability experiment of SIGMOD 2008. SIGMOD Record 37(39), 45 (2008)Google Scholar
  6. 6.
    Peng, R.D.: Reproducible research in computational science. Science 334(6060), 1226–1227 (2011)CrossRefGoogle Scholar
  7. 7.
    Peng, R.D., Eckel, S.P.: Distributed reproducible research using cached computations. Computing in Science and Engineering 11(1), 28–34 (2009)CrossRefGoogle Scholar
  8. 8.
    Sandve, G.K., Nekrutenko, A., J., Taylor, O.: Ten simple rules for reproducible computational research. PLoS Computational Biology 9(10), e1003285 (2013)Google Scholar
  9. 9.
    Vandewalle, P., Kovacevic, J., Vetterli, M.: Reproducible research in signal processing. IEEE Signal Processing Magazine 26(3), 37–47 (2009)CrossRefGoogle Scholar
  10. 10.
    Vitek, J., Kalibera, T.: R3: Repeatability, reproducibility and rigor. SIGPLAN Notices 47(4a), 30–36 (2012)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Alexandra Carpen-Amarie
    • 1
  • Antoine Rougier
    • 1
  • Felix D. Lübbe
    • 1
  1. 1.Faculty of Informatics, Institute of Information Systems, Research Group Parallel ComputingVienna University of Technology, AustriaViennaAustria

Personalised recommendations