Stepping Stones to Reproducible Research: A Study of Current Practices in Parallel Computing
Experimental research plays an important role in parallel computing, as in this field scientific discovery often relies on experimental findings, which complement and validate theoretical models. However, parallel hardware and applications have become extremely complex to study, due to their diversity and rapid evolution. Furthermore, applications are designed to run on thousands of nodes, often spanning across several programming models and generating large amounts of data. In this context, reproducibility is essential to foster reliable scientific results. In this paper we aim at studying the requirements and pitfalls of each stage of experimental research, from data acquisition to data analysis, with respect to achieving reproducible results. We investigate state-of-the-art experimental practices in parallel computing by conducting a survey on the papers published in EuroMPI 2013, a major conference targeting the MPI community. Our findings show that while there is a clear concern for reproducibility in the parallel computing community, a better understanding of the criteria for achieving it is necessary.
KeywordsSource Code Parallel Computing Data Analysis Phase Reproducible Experiment Reproducible Research
Unable to display preview. Download preview PDF.
- 1.Bonnet, P., Manegold, S., Bjørling, M., et al.: Repeatability and workability evaluation of SIGMOD 2011. SIGMOD Record 40(45), 48 (2011)Google Scholar
- 2.Collberg, C., Proebsting, T., et al.: Measuring Reproducibility in Computer Systems Research (2014), http://reproducibility.cs.arizona.edu/tr.pdf
- 4.Hunold, S., Träff, J.L.: On the state and importance of reproducible experimental research in parallel computing. CoRR abs/1308.3648 (2013)Google Scholar
- 5.Manolescu, I., Afanasiev, L., Arion, A., et al.: The repeatability experiment of SIGMOD 2008. SIGMOD Record 37(39), 45 (2008)Google Scholar
- 8.Sandve, G.K., Nekrutenko, A., J., Taylor, O.: Ten simple rules for reproducible computational research. PLoS Computational Biology 9(10), e1003285 (2013)Google Scholar