Abstract
Experimental research plays an important role in parallel computing, as in this field scientific discovery often relies on experimental findings, which complement and validate theoretical models. However, parallel hardware and applications have become extremely complex to study, due to their diversity and rapid evolution. Furthermore, applications are designed to run on thousands of nodes, often spanning across several programming models and generating large amounts of data. In this context, reproducibility is essential to foster reliable scientific results. In this paper we aim at studying the requirements and pitfalls of each stage of experimental research, from data acquisition to data analysis, with respect to achieving reproducible results. We investigate state-of-the-art experimental practices in parallel computing by conducting a survey on the papers published in EuroMPI 2013, a major conference targeting the MPI community. Our findings show that while there is a clear concern for reproducibility in the parallel computing community, a better understanding of the criteria for achieving it is necessary.
Chapter PDF
Similar content being viewed by others
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
Bonnet, P., Manegold, S., Bjørling, M., et al.: Repeatability and workability evaluation of SIGMOD 2011. SIGMOD Record 40(45), 48 (2011)
Collberg, C., Proebsting, T., et al.: Measuring Reproducibility in Computer Systems Research (2014), http://reproducibility.cs.arizona.edu/tr.pdf
Freire, J., Bonnet, P., Shasha, D.: Computational reproducibility: State-of-the-art, challenges, and database research opportunities. In: Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data, SIGMOD 2012, pp. 593–596. ACM, New York (2012)
Hunold, S., Träff, J.L.: On the state and importance of reproducible experimental research in parallel computing. CoRR abs/1308.3648 (2013)
Manolescu, I., Afanasiev, L., Arion, A., et al.: The repeatability experiment of SIGMOD 2008. SIGMOD Record 37(39), 45 (2008)
Peng, R.D.: Reproducible research in computational science. Science 334(6060), 1226–1227 (2011)
Peng, R.D., Eckel, S.P.: Distributed reproducible research using cached computations. Computing in Science and Engineering 11(1), 28–34 (2009)
Sandve, G.K., Nekrutenko, A., J., Taylor, O.: Ten simple rules for reproducible computational research. PLoS Computational Biology 9(10), e1003285 (2013)
Vandewalle, P., Kovacevic, J., Vetterli, M.: Reproducible research in signal processing. IEEE Signal Processing Magazine 26(3), 37–47 (2009)
Vitek, J., Kalibera, T.: R3: Repeatability, reproducibility and rigor. SIGPLAN Notices 47(4a), 30–36 (2012)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Carpen-Amarie, A., Rougier, A., Lübbe, F.D. (2014). Stepping Stones to Reproducible Research: A Study of Current Practices in Parallel Computing. In: Lopes, L., et al. Euro-Par 2014: Parallel Processing Workshops. Euro-Par 2014. Lecture Notes in Computer Science, vol 8805. Springer, Cham. https://doi.org/10.1007/978-3-319-14325-5_43
Download citation
DOI: https://doi.org/10.1007/978-3-319-14325-5_43
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-14324-8
Online ISBN: 978-3-319-14325-5
eBook Packages: Computer ScienceComputer Science (R0)