Reproducible Experiments in Parallel Computing: Concepts and Stencil Compiler Benchmark Study
For decades, the majority of the experiments on parallel computers have been reported at conferences and in journals usually without the possibility to verify the results presented. Thus, one of the major principles of science, reproducible results as a kind of correctness proof, has been neglected in the field of experimental high-performance computing. While this is still the state-of-the-art, current research targets for solutions to this problem. We discuss early results regarding reproducibility from a benchmark case study we did. In our experiments we explore the class of stencil calculations that are part of many scientific kernels and compare the performance results of four stencil compilers. In order to make these experiments reproducible from remote, a first prototype of an replication engine has been developed that can be accessed via the internet.
Unable to display preview. Download preview PDF.
- 1.Victoria, S.: Trust your science? Open your data and code. Amstat News (2011)Google Scholar
- 4.Hunold, S., Träff, J.L.: On the state and importance of reproducible experimental research in parallel computing. CoRR abs/1308.3648 (2013)Google Scholar
- 5.Dolfi, M., Gukelberger, J., Hehn, A., Imriska, J., Pakrouski, K., Rønnow, T.F., Troyer, M., Zintchenko, I., Chirigati, F.S., Freire, J., Shasha, D.: A model project for reproducible papers: critical temperature for the Ising model on a square lattice. CoRR abs/1401.2000 (2014)Google Scholar
- 10.HPL - A portable implementation of the high-performance Linpack Benchmark for distributed-memory computers (2008), http://www.netlib.org/benchmark/hpl/
- 11.Danilo Guerrera and Antonio Maffia. Workflow for reproducibility (2014), https://github.com/sguera/workflow_repro
- 12.Asanovic, K., Bodik, R., Catanzaro, B.C., Gebis, J.J., Husbands, P., Keutzer, K., Patterson, D.A., Plishker, W.L., Shalf, J., Williams, S.W., Yelick, K.A.: The landscape of parallel computing research: A view from Berkeley. Technical Report UCB/EECS-2006-183, EECS Department, University of California, Berkeley (December 2006)Google Scholar
- 13.Frigo, M., Strumpen, V.: Cache oblivious stencil computations. In: Proceedings of the 19th Annual International Conference on Supercomputing, ICS 2005, pp. 361–366. ACM (2005)Google Scholar
- 14.Tang, Y., Chowdhury, R.A., Kuszmaul, B.C., Luk, C.-K., Leiserson, C.E.: The Pochoir stencil compiler. In: Proceedings of the Twenty-Third Annual ACM Symposium on Parallelism in Algorithms and Architectures, SPAA 2011, pp. 117–128. ACM, New York (2011)Google Scholar
- 17.Bondhugula, U., Ramanujam, J., Sadayappan, P.: PLuTo: A practical and fully automatic polyhedral parallelizer and locality optimizer. Technical Report OSU-CISRC-10/07-TR70, The Ohio State University (October 2007)Google Scholar
- 18.Christen, M., Schenk, O., Burkhart, H.: PATUS: A code generation and autotuning framework for parallel iterative stencil computations on modern microarchitectures. In: 2011 IEEE International Parallel Distributed Processing Symposium (IPDPS), pp. 676–687 (2011)Google Scholar
- 19.Gsponer, S.: Stencil compilers in practice: Workflow engine and code generation issues. Master’s thesis, University of Basel (2014)Google Scholar