Skip to main content
SpringerLink
Log in

Using Performance Analysis Tools for a Parallel-in-Time Integrator

Does My Time-Parallel Code Do What I Think It Does?

  • Conference paper
  • First Online:
Parallel-in-Time Integration Methods (PinT 2020)

Part of the book series: Springer Proceedings in Mathematics & Statistics ((PROMS,volume 356))

Included in the following conference series:

Abstract

While many ideas and proofs of concept for parallel-in-time integration methods exists, the number of large-scale, accessible time-parallel codes is rather small. This is often due to the apparent or subtle complexity of the algorithms and the many pitfalls awaiting developers of parallel numerical software. One example of such a time-parallel code is pySDC, which implements, among others, the parallel full approximation scheme in space and time (PFASST). Inspired by nonlinear multigrid ideas, PFASST allows to integrate multiple time steps simultaneously using a space-time hierarchy of spectral deferred corrections. In this paper, we demonstrate the application of performance analysis tools to the PFASST implementation pySDC. We trace the path we took for this work, show examples of how the tools can be applied, and explain the sometimes surprising findings we encountered. Although focusing only on a single implementation of a particular parallel-in-time integrator, we hope that our results and in particular the way we obtained them are a blueprint for other time-parallel codes.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 149.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 199.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 199.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://www.parallel-in-time.org.

  2. 2.

    https://www.parallel-in-time.org/pySDC.

  3. 3.

    https://www.par-tec.com/products/parastation-mpi.html.

  4. 4.

    https://pop-coe.eu/node/69.

  5. 5.

    https://www.vi-hps.org/training/tws/tuning-workshop-series.html.

References

  1. Adhianto, L., Banerjee, S., Fagan, M., Krentel, M., Marin, G., Mellor-Crummey, J., Tallent, N.R.: HPCToolkit: Tools for performance analysis of optimized parallel programs. Concurrency and Computation: Practice and Experience 22(6), 685–701 (2010)

    Google Scholar 

  2. Böhme, D., Wolf, F., de Supinski, B.R., Schulz, M., Geimer, M.: Scalable critical-path based performance analysis. In: 2012 IEEE 26th International Parallel and Distributed Processing Symposium, pp. 1330–1340. IEEE (2012)

    Google Scholar 

  3. Bolten, M., Moser, D., Speck, R.: A multigrid perspective on the parallel full approximation scheme in space and time. Numerical Linear Algebra with Applications 24(6), e2110–n/a (2017). DOI https://doi.org/10.1002/nla.2110. E2110 nla.2110

  4. Bolten, M., Moser, D., Speck, R.: Asymptotic convergence of the parallel full approximation scheme in space and time for linear problems. Numerical linear algebra with applications 25(6), e2208 – (2018). DOI https://doi.org/10.1002/nla.2208. URL https://juser.fz-juelich.de/record/857114

  5. Bradley, T.: GPU Performance Analysis and Optimisation. NVIDIA Corporation (2012)

    Google Scholar 

  6. Center, B.S.: Website for POP CoE (2019). URL https://pop-coe.eu/. [Online; accessed August 13, 2019]

  7. Dalcin, L.D., Paz, R.R., Kler, P.A., Cosimo, A.: Parallel distributed computing using python. Advances in Water Resources 34(9), 1124–1139 (2011). DOI https://doi.org/10.1016/j.advwatres.2011.04.013. URL http://www.sciencedirect.com/science/article/pii/S0309170811000777. New Computational Methods and Software Tools

  8. Dalcin, Lisandro and Mortensen, Mikael and Keyes, David E: Fast parallel multidimensional FFT using advanced MPI. Journal of Parallel and Distributed Computing (2019). DOI https://doi.org/10.1016/j.jpdc.2019.02.006

  9. Dutt, A., Greengard, L., Rokhlin, V.: Spectral deferred correction methods for ordinary differential equations. BIT Numerical Mathematics 40(2), 241–266 (2000). DOI https://doi.org/10.1023/A:1022338906936

  10. Emmett, M., Minion, M.L.: Toward an efficient parallel in time method for partial differential equations. Communications in Applied Mathematics and Computational Science 7, 105–132 (2012). DOI https://doi.org/10.2140/camcos.2012.7.105

  11. Emmett, M., Minion, M.L.: Efficient implementation of a multi-level parallel in time algorithm. In: Domain Decomposition Methods in Science and Engineering XXI, Lecture Notes in Computational Science and Engineering, vol. 98, pp. 359–366. Springer International Publishing (2014). DOI https://doi.org/10.1007/978-3-319-05789-7_33

  12. Eschweiler, D., Wagner, M., Geimer, M., Knüpfer, A., Nagel, W.E., Wolf, F.: Open Trace Format 2 - The next generation of scalable trace formats and support libraries. In: Proc. of the Intl. Conference on Parallel Computing (ParCo), Ghent, Belgium, August 30–September 2 2011, Advances in Parallel Computing, vol. 22, pp. 481–490. IOS Press (2012). DOI https://doi.org/10.3233/978-1-61499-041-3-481

  13. Feld, C., Convent, S., Hermanns, M.A., Protze, J., Geimer, M., Mohr, B.: Score-p and ompt: Navigating the perils of callback-driven parallel runtime introspection. In: X. Fan, B.R. de Supinski, O. Sinnen, N. Giacaman (eds.) OpenMP: Conquering the Full Hardware Spectrum, pp. 21–35. Springer International Publishing, Cham (2019)

    Chapter  Google Scholar 

  14. Gander, M.J.: 50 years of Time Parallel Time Integration. In: Multiple Shooting and Time Domain Decomposition. Springer (2015). DOI https://doi.org/10.1007/978-3-319-23321-5_3

  15. Geimer, M., Wolf, F., Wylie, B.J.N., Ábrahám, E., Becker, D., Mohr, B.: The SCALASCA performance toolset architecture. In: International Workshop on Scalable Tools for High-End Computing (STHEC), Kos, Greece, pp. 51–65 (2008)

    Google Scholar 

  16. Geimer, M., Saviankou, P., Strube, A., Szebenyi, Z., Wolf, F., Wylie, B.J.N.: Further improving the scalability of the scalasca toolset. In: K. Jónasson (ed.) Applied Parallel and Scientific Computing, pp. 463–473. Springer Berlin Heidelberg, Berlin, Heidelberg (2012)

    Chapter  Google Scholar 

  17. Gocht, A., Schöne, R., Frenzel, J.: Advanced Python Performance Monitoring with Score-P. In: Tools for High Performance Computing 2019, p. to appear. Springer International Publishing (2019)

    Google Scholar 

  18. Harlacher, M., Calotoiu, A., Dennis, J., Wolf, F.: Analysing the Scalability of Climate Codes Using New Features of Scalasca. In: K. Binder, M. Müller, M. Kremer, A. Schnurpfeil (eds.) Proc. of the John von Neumann Institute for Computing (NIC) Symposium 2016, Juelich, Germany, NIC Series, vol. 48, pp. 343–352. Forschungszentrum Jülich, John von Neumann-Institut for Computing (2016)

    Google Scholar 

  19. Hermanns, M.A., Geimer, M., Mohr, B., Wolf, F.: Trace-based detection of lock contention in MPI one-sided communication. In: C. Niethammer, J. Gracia, T. Hilbrich, A. Knüpfer, M.M. Resch, W.E. Nagel (eds.) Tools for High Performance Computing 2016, Proc. of the 10th Parallel Tools Workshop, Stuttgart, Germany, October 2016, pp. 97–114. Springer (2017). DOI https://doi.org/10.1007/978-3-319-56702-0_6. URL http://juser.fz-juelich.de/record/830159

  20. Huang, J., Jia, J., Minion, M.: Accelerating the convergence of spectral deferred correction methods. Journal of Computational Physics 214(2), 633–656 (2006)

    Google Scholar 

  21. Jülich Supercomputing Centre: JURECA: General-purpose supercomputer at Jülich Supercomputing Centre. Journal of large-scale research facilities 2(A62) (2016). DOI https://doi.org/10.17815/jlsrf-2-121

  22. Knobloch, M., Mohr, B.: Tools for GPU Computing – Debugging and Performance Analysis of Heterogenous HPC Applications. Supercomputing Frontiers and Innovations 7(1) (2020). URL https://superfri.org/superfri/article/view/311

  23. Knobloch, M., Saviankou, P., Schlütter, M., Visser, A., Mohr, B.: A picture is worth a thousand numbers – Enhancing Cube’s analysis capabilities with plugins. In: Tools for High Performance Computing 2019 (tbp)

    Google Scholar 

  24. Knüpfer, A., Brunst, H., Doleschal, J., Jurenz, M., Lieber, M., Mickler, H., Müller, M.S., Nagel, W.E.: The Vampir Performance Analysis Tool-Set. In: M. Resch, R. Keller, V. Himmler, B. Krammer, A. Schulz (eds.) Tools for High Performance Computing, pp. 139–155. Springer Berlin / Heidelberg (2008). DOI https://doi.org/10.1007/978-3-540-68564-7_9

  25. Knüpfer, A., Rössel, C., an Mey, D., Biersdorff, S., Diethelm, K., Eschweiler, D., Geimer, M., Gerndt, M., Lorenz, D., Malony, A.D., Nagel, W.E., Oleynik, Y., Philippen, P., Saviankou, P., Schmidl, D., Shende, S.S., Tschüter, R., Wagner, M., Wesarg, B., Wolf, F.: Score-P – A joint performance measurement run-time infrastructure for Periscope, Scalasca, TAU, and Vampir. In: Proc. of the 5th Int’l Workshop on Parallel Tools for High Performance Computing, September 2011, Dresden, pp. 79–91. Springer (2012). DOI https://doi.org/10.1007/978-3-642-31476-6_7

  26. LLNL: Website for XBraid (2018). URL https://www.llnl.gov/casc/xbraid. [Online; accessed July 30, 2018]

  27. Lührs, S., Rohe, D., Schnurpfeil, A., Thust, K., Frings, W.: Flexible and Generic Workflow Management. In: Parallel Computing: On the Road to Exascale, Advances in parallel computing, vol. 27, pp. 431–438. International Conference on Parallel Computing 2015, Edinburgh (United Kingdom), 1 Sep 2015–4 Sep 2015, IOS Press, Amsterdam (2016). DOI https://doi.org/10.3233/978-1-61499-621-7-431. URL http://juser.fz-juelich.de/record/808798

  28. Minion, M., Emmett, M.: Website for libpfasst (2019). URL https://github.com/libpfasst/LibPFASST. [Online; accessed August 13, 2019]

  29. Minion, M.L.: A hybrid parareal spectral deferred corrections method. Communications in Applied Mathematics and Computational Science 5(2), 265–301 (2010). DOI https://doi.org/10.2140/camcos.2010.5.265

  30. Mix, H., Herold, C., Weber, M.: Visualization of Multi-layer I/O Performance in Vampir. In: Parallel and Distributed Processing Symposium Workshop (IPDPSW), 2018 IEEE International (2018)

    Google Scholar 

  31. Ong, B.W., Schroder, J.B.: Applications of time parallelization. Computing and Visualization in Science 23(1), 1–15 (2020)

    MathSciNet  Google Scholar 

  32. Ong, B.W., Haynes, R.D., Ladd, K.: Algorithm 965: RIDC Methods: A Family of Parallel Time Integrators. ACM Trans. Math. Softw. 43(1), 8:1–8:13 (2016). DOI https://doi.org/10.1145/2964377

  33. Pillet, V., Labarta, J., Cortes, T., Girona, S.: Paraver: A tool to visualize and analyze parallel code. In: Proceedings of WoTUG-18: transputer and occam developments, vol. 44, pp. 17–31. Citeseer (1995)

    Google Scholar 

  34. Reinders, J.: Vtune performance analyzer essentials. Intel Press (2005)

    Google Scholar 

  35. Ruprecht, D., Speck, R.: Spectral deferred corrections with fast-wave slow-wave splitting. SIAM Journal on Scientific Computing 38(4), A2535–A2557 (2016)

    Article  MathSciNet  Google Scholar 

  36. Saviankou, P., Knobloch, M., Visser, A., Mohr, B.: Cube v4: From performance report explorer to performance analysis tool. In: Proceedings of the International Conference on Computational Science, ICCS 2015, Computational Science at the Gates of Nature, Reykjavík, Iceland, 1–3 June, 2015, pp. 1343–1352 (2015). DOI https://doi.org/10.1016/j.procs.2015.05.320

  37. Saviankou, P., Knobloch, M., Visser, A., Mohr, B.: Cube v4: From performance report explorer to performance analysis tool. Procedia Computer Science 51, 1343–1352 (2015)

    Article  Google Scholar 

  38. Sharples, W., Zhukov, I., Geimer, M., Goergen, K., Luehrs, S., Breuer, T., Naz, B., Kulkarni, K., Brdar, S., Kollet, S.: A run control framework to streamline profiling, porting, and tuning simulation runs and provenance tracking of geoscientific applications. Geoscientific Model Development 11(7), 2875–2895 (2018). DOI https://doi.org/10.5194/gmd-11-2875-2018

  39. Shende, S.S., Malony, A.D.: The TAU parallel performance system. The International Journal of High Performance Computing Applications 20(2), 287–311 (2006)

    Article  Google Scholar 

  40. Speck, R.: Algorithm 997: pySDC - Prototyping Spectral Deferred Corrections. ACM Transactions on Mathematical Software 45(3) (2019). DOI https://doi.org/10.1145/3310410

  41. Speck, R.: Parallel-in-time/pysdc: The performance release (2019). DOI https://doi.org/10.5281/zenodo.3407254

  42. Speck, R.: Website for pySDC (2019). URL https://parallel-in-time.org/pySDC/. [Online; accessed August 13, 2019]

  43. Terpstra, D., Jagode, H., You, H., Dongarra, J.: Collecting performance data with papi-c. In: Tools for High Performance Computing 2009, pp. 157–173. Springer (2010)

    Google Scholar 

  44. Treibig, J., Hager, G., Wellein, G.: Likwid: A lightweight performance-oriented tool suite for x86 multicore environments. In: 2010 39th International Conference on Parallel Processing Workshops, pp. 207–216. IEEE (2010)

    Google Scholar 

  45. Weiser, M.: Faster SDC convergence on non-equidistant grids by DIRK sweeps. BIT Numerical Mathematics 55(4), 1219–1241 (2014)

    Article  MathSciNet  Google Scholar 

  46. Zhang, J., Du, Q.: Numerical Studies of Discrete Approximations to the Allen-Cahn Equation in the Sharp Interface Limit. SIAM Journal on Scientific Computing 31(4), 3042–3063 (2009). DOI https://doi.org/10.1137/080738398

  47. Zhukov, I., Feld, C., Geimer, M., Knobloch, M., Mohr, B., Saviankou, P.: Scalasca v2: Back to the future. In: Proc. of Tools for High Performance Computing 2014, pp. 1–24. Springer (2015). DOI https://doi.org/10.1007/978-3-319-16012-2_1

Download references

Acknowledgements

Parts of this work have received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreements No 676553 and 824080. RS thankfully acknowledges the financial support by the German Federal Ministry of Education and Research through the ParaPhase project within the framework “IKT 2020 - Forschung für Innovationen” (project number 01IH15005A).

Author information

Authors and Affiliations

  1. Forschungszentrum Jülich GmbH, Jülich Supercomputing Centre, 52425, Jülich, Germany

    Robert Speck, Michael Knobloch & Sebastian Lührs

  2. Center of Information Services and High Performance Computing, Zellescher Weg 12, 01069, Dresden, Germany

    Andreas Gocht

Authors
  1. Robert Speck
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Michael Knobloch
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sebastian Lührs
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Andreas Gocht
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Robert Speck .

Editor information

Editors and Affiliations

  1. Department of Mathematical Sciences, Michigan Technological University, Houghton, MI, USA

    Benjamin Ong

  2. Department of Mathematics and Statistics, University of New Mexico, Albuquerque, NM, USA

    Jacob Schroder

  3. Department of Mathematics, University of Exeter, Exeter, UK

    Jemma Shipton

  4. Department of Mathematics, University of Wuppertal, Wuppertal, Nordrhein-Westfalen, Germany

    Stephanie Friedhoff

Rights and permissions

Reprints and permissions

Copyright information

© 2021 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Speck, R., Knobloch, M., Lührs, S., Gocht, A. (2021). Using Performance Analysis Tools for a Parallel-in-Time Integrator. In: Ong, B., Schroder, J., Shipton, J., Friedhoff, S. (eds) Parallel-in-Time Integration Methods. PinT 2020. Springer Proceedings in Mathematics & Statistics, vol 356. Springer, Cham. https://doi.org/10.1007/978-3-030-75933-9_3

Download citation

Publish with us

Policies and ethics

Search

Navigation