Reproducible Experiments in Parallel Computing: Concepts and Stencil Compiler Benchmark Study

  • Danilo Guerrera
  • Helmar Burkhart
  • Antonio Maffia
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8805)


For decades, the majority of the experiments on parallel computers have been reported at conferences and in journals usually without the possibility to verify the results presented. Thus, one of the major principles of science, reproducible results as a kind of correctness proof, has been neglected in the field of experimental high-performance computing. While this is still the state-of-the-art, current research targets for solutions to this problem. We discuss early results regarding reproducibility from a benchmark case study we did. In our experiments we explore the class of stencil calculations that are part of many scientific kernels and compare the performance results of four stencil compilers. In order to make these experiments reproducible from remote, a first prototype of an replication engine has been developed that can be accessed via the internet.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Victoria, S.: Trust your science? Open your data and code. Amstat News (2011)Google Scholar
  2. 2.
    Freire, J., Silva, C.T.: Making computations and publications reproducible with VisTrails. Computing in Science Engineering 14(4), 18–25 (2012)CrossRefGoogle Scholar
  3. 3.
    Davison, A.P.: Automated capture of experiment context for easier reproducibility in computational research. Computing in Science Engineering 14(4), 48–56 (2012)CrossRefGoogle Scholar
  4. 4.
    Hunold, S., Träff, J.L.: On the state and importance of reproducible experimental research in parallel computing. CoRR abs/1308.3648 (2013)Google Scholar
  5. 5.
    Dolfi, M., Gukelberger, J., Hehn, A., Imriska, J., Pakrouski, K., Rønnow, T.F., Troyer, M., Zintchenko, I., Chirigati, F.S., Freire, J., Shasha, D.: A model project for reproducible papers: critical temperature for the Ising model on a square lattice. CoRR abs/1401.2000 (2014)Google Scholar
  6. 6.
    Missier, P., Soiland-Reyes, S., Owen, S., Tan, W., Nenadic, A., Dunlop, I., Williams, A., Oinn, T., Goble, C.: Taverna, Reloaded. In: Gertz, M., Ludäscher, B. (eds.) SSDBM 2010. LNCS, vol. 6187, pp. 471–481. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  7. 7.
    Scheidegger, C.E., Vo, H.T., Koop, D., Freire, J., Silva, C.T.: Querying and re-using workflows with VisTrails. In: Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data, SIGMOD 2008, pp. 1251–1254. ACM, New York (2008)CrossRefGoogle Scholar
  8. 8.
    Ludäscher, B., Altintas, I., Berkley, C., Higgins, D., Jaeger, E., Jones, M., Lee, E.A., Tao, J., Zhao, Y.: Scientific workflow management and the Kepler system. Concurrency and Computation: Practice and Experience 18(10), 1039–1065 (2006)CrossRefGoogle Scholar
  9. 9.
  10. 10.
    HPL - A portable implementation of the high-performance Linpack Benchmark for distributed-memory computers (2008),
  11. 11.
    Danilo Guerrera and Antonio Maffia. Workflow for reproducibility (2014),
  12. 12.
    Asanovic, K., Bodik, R., Catanzaro, B.C., Gebis, J.J., Husbands, P., Keutzer, K., Patterson, D.A., Plishker, W.L., Shalf, J., Williams, S.W., Yelick, K.A.: The landscape of parallel computing research: A view from Berkeley. Technical Report UCB/EECS-2006-183, EECS Department, University of California, Berkeley (December 2006)Google Scholar
  13. 13.
    Frigo, M., Strumpen, V.: Cache oblivious stencil computations. In: Proceedings of the 19th Annual International Conference on Supercomputing, ICS 2005, pp. 361–366. ACM (2005)Google Scholar
  14. 14.
    Tang, Y., Chowdhury, R.A., Kuszmaul, B.C., Luk, C.-K., Leiserson, C.E.: The Pochoir stencil compiler. In: Proceedings of the Twenty-Third Annual ACM Symposium on Parallelism in Algorithms and Architectures, SPAA 2011, pp. 117–128. ACM, New York (2011)Google Scholar
  15. 15.
    Kelley, J.R., Barnes, C., Adams, A., Paris, S., Durand, F., Amarasinghe, S.: Halide: a language and compiler for optimizing parallelism, locality, and recomputation in image processing pipelines. SIGPLAN Not. 48(6), 519–530 (2013)CrossRefGoogle Scholar
  16. 16.
    Benabderrahmane, M.-W., Pouchet, L.-N., Cohen, A., Bastoul, C.: The Polyhedral Model Is More Widely Applicable Than You Think. In: Gupta, R. (ed.) CC 2010. LNCS, vol. 6011, pp. 283–303. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  17. 17.
    Bondhugula, U., Ramanujam, J., Sadayappan, P.: PLuTo: A practical and fully automatic polyhedral parallelizer and locality optimizer. Technical Report OSU-CISRC-10/07-TR70, The Ohio State University (October 2007)Google Scholar
  18. 18.
    Christen, M., Schenk, O., Burkhart, H.: PATUS: A code generation and autotuning framework for parallel iterative stencil computations on modern microarchitectures. In: 2011 IEEE International Parallel Distributed Processing Symposium (IPDPS), pp. 676–687 (2011)Google Scholar
  19. 19.
    Gsponer, S.: Stencil compilers in practice: Workflow engine and code generation issues. Master’s thesis, University of Basel (2014)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Danilo Guerrera
    • 1
  • Helmar Burkhart
    • 1
  • Antonio Maffia
    • 1
  1. 1.University of BaselSwitzerland

Personalised recommendations