Skip to main content

Parallel Reverse Mode Automatic Differentiation for OpenMP Programs with ADOL-C

  • Conference paper

Part of the Lecture Notes in Computational Science and Engineering book series (LNCSE,volume 64)

Summary

Shared-memory multicore computing platforms are becoming commonplace, and loop parallelization with OpenMP offers an easy way for the user to harness their power. As a result, tools for automatic differentiation (AD) should be able to deal with such codes in a fashion that preserves their parallel nature also for the derivative evaluation. In this paper, we explore this issue using a plasma simulation code. Its structure, which in essence is a time stepping loop with several parallelizable inner loops, is representative of many other computations. Using this code as an example, we develop a strategy for the efficient implementation of the reverse mode of AD with trace-based AD-tools and implement it with the ADOL-C tool. The strategy combines checkpointing at the outer level with parallel trace generation and evaluation at the inner level. We discuss the extensions necessary for ADOL-C to work in a multithreaded environment and the setup necessary for the user code and present performance results on a shared-memory multiprocessor.

Keywords

  • Parallelism
  • OpenMP
  • reverse mode
  • checkpointing
  • ADOL-C

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (Canada)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   139.00
Price excludes VAT (Canada)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   179.99
Price excludes VAT (Canada)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Asanovic, K., Bodik, R., Catanzaro, B.C., Gebis, J.J., Husbands, P., Keutzer, K., Patterson, D.A., Plishker, W.L., Shalf, J., Williams, S.W., Yelick, K.A.: The landscape of parallel computing research: A view from Berkeley. Tech. Rep. UCB/EECS-2006-183, EECS Department, University of California, Berkeley (2006)

    Google Scholar 

  2. Benary, J.: Parallelism in the reverse mode. In: M. Berz, C. Bischof, G. Corliss, A. Griewank (eds.) Computational Differentiation: Techniques, Applications, and Tools, pp. 137–147. SIAM, Philadelphia, PA (1996)

    Google Scholar 

  3. Berz, M., Bischof, C., Corliss, G., Griewank, A. (eds.): Computational Differentiation: Techniques, Applications and Tools. SIAM, Philadelphia, PA (1996)

    MATH  Google Scholar 

  4. Bischof, C., Green, L., Haigler, K., Knauff, T.: Parallel calculation of sensitivity derivatives for aircraft design using automatic differentiation. In: Proc. 5th AIAA/NASA/USAF/ISSMO Symposium on Multidisciplinary Analysis and Optimization, AIAA 94-4261, pp. 73–84 (1994)

    Google Scholar 

  5. Bischof, C., Griewank, A., Juedes, D.: Exploiting parallelism in automatic differentiation. In: E. Houstis, Y. Muraoka (eds.) Proc. 1991 Int. Conf. on Supercomputing, pp. 146–153. ACM Press, Baltimore, Md. (1991)

    CrossRef  Google Scholar 

  6. Bischof, C.H.: Issues in Parallel Automatic Differentiation. In: A. Griewank, G.F. Corliss (eds.) Automatic Differentiation of Algorithms: Theory, Implementation, and Application, pp. 100–113. SIAM, Philadelphia, PA (1991)

    Google Scholar 

  7. Bischof, C.H., Bücker, H.M., Wu, P.: Time-parallel computation of pseudo-adjoints for a leapfrog scheme. International Journal of High Speed Computing 12(1), 1–27 (2004)

    CrossRef  MATH  Google Scholar 

  8. Bischof, C.H., Hovland, P.D.: Automatic differentiation: Parallel computation. In: C.A. Floudas, P.M. Pardalos (eds.) Encyclopedia of Optimization, vol. I, pp. 102–108. Kluwer Academic Publishers, Dordrecht, The Netherlands (2001)

    CrossRef  Google Scholar 

  9. Bischof, C.H., an Mey, D., Terboven, C., Sarholz, S.: Parallel computers everywhere. In: Proc. 16th Conf. on the Computation of Electromagnetic Fields (COMPUMAG2007)), pp. 693–700. Informationstechnische Gesellschaft im VDE (2007)

    Google Scholar 

  10. Bischof, C.H., an Mey, D., Terboven, C., Sarholz, S.: Petaflops basics – performance from SMP building blocks. In: D. Bader (ed.) Petascale Computing – Algorithms and Applications, pp. 311–331. Chapmann & Hall/CRC (2007)

    Google Scholar 

  11. Brunst, H., Nagel, W.E.: Scalable Performance Analysis of Parallel Systems: Concepts and Experiences. In: G.R. Joubert, W.E. Nagel, F.J. Peters, W.V. Walter (eds.) PARALLEL COMPUTING: Software Technology, Algorithms, Architectures and Applications, Advances in Parallel Computing, vol. 13, pp. 737–744. Elsevier (2003)

    Google Scholar 

  12. Bücker, H.M., Corliss, G.F., Hovland, P.D., Naumann, U., Norris, B. (eds.): Automatic Differentiation: Applications, Theory, and Implementations, Lecture Notes in Computational Science and Engineering, vol. 50. Springer, New York, NY (2005)

    Google Scholar 

  13. Bücker, H.M., Lang, B., Rasch, A., Bischof, C.H.: Automatic parallelism in differentiation of Fourier transforms. In: Proc. 18th ACM Symp. on Applied Computing, Melbourne, Florida, USA, March 9–12, 2003, pp. 148–152. ACM Press, New York (2003)

    Google Scholar 

  14. Bücker, H.M., Lang, B., Rasch, A., Bischof, C.H., an Mey, D.: Explicit loop scheduling in OpenMP for parallel automatic differentiation. In: J.N. Almhana, V.C. Bhavsar (eds.) Proc. 16th Annual Int. Symp. High Performance Computing Systems and Applications, Moncton, NB, Canada, June 16–19, 2002, pp. 121–126. IEEE Comput. Soc. Press (2002)

    Google Scholar 

  15. Bücker, H.M., Rasch, A., Vehreschild, A.: Automatic generation of parallel code for Hessian computations. In: Proceedings of the International Workshop on OpenMP (IWOMP 2006), Reims, France, June 12–15, 2006, Lecture Notes in Computer Science, vol. 4315. Springer. To appear

    Google Scholar 

  16. Bücker, H.M., Rasch, A., Wolf, A.: A class of OpenMP applications involving nested parallelism. In: Proc. 19th ACM Symp. on Applied Computing, Nicosia, Cyprus, March 14–17, 2004, vol. 1, pp. 220–224. ACM Press (2004)

    Google Scholar 

  17. Carle, A., Fagan, M.: Automatically differentiating MPI-1 datatypes: The complete story. In: G. Corliss, C. Faure, A. Griewank, L. Hascoët, U. Naumann (eds.) Automatic Differentiation of Algorithms: From Simulation to Optimization, Computer and Information Science, chap. 25, pp. 215–222. Springer, New York, NY (2002)

    Google Scholar 

  18. Corliss, G., Faure, C., Griewank, A., Hascoët, L., Naumann, U. (eds.): Automatic Differentiation of Algorithms: From Simulation to Optimization, Computer and Information Science. Springer, New York, NY (2002)

    Google Scholar 

  19. Dagum, L., Menon, R.: OpenMP: An Industry-Standard API for Shared-Memory Programming. IEEE Computational Science and Engineering 05(1), 46–55 (1998)

    CrossRef  Google Scholar 

  20. Faure, C., Dutto, P.: Extension of Odyssee to the MPI Library – Direct Mode. Rapport de Recherche 3715, INRIA, Sophia-Antipolis (1999)

    Google Scholar 

  21. Faure, C., Dutto, P.: Extension of Odyssee to the MPI Library – Reverse Mode. Rapport de Recherche 3774, INRIA, Sophia-Antipolis (1999)

    Google Scholar 

  22. Gerndt, A., Sarholz, S., Wolter, M., an Mey, D., Bischof, C., Kuhlen, T.: Nested OpenMP for efficient computation of 3D critical points in multi-block CFD datasets. In: Proc. ACM/IEEE SC 2006 Conference (2006)

    Google Scholar 

  23. Giering, R., Kaminski, T., Todling, R., Errico, R., Gelaro, R., Winslow, N.: Tangent linear and adjoint versions of NASA/GMAO’s Fortran 90 global weather forecast model. In: H.M. Bücker, G.F. Corliss, P.D. Hovland, U. Naumann, B. Norris (eds.) Automatic Differentiation: Applications, Theory, and Implementations, pp. 275–284. Springer (2005)

    Google Scholar 

  24. Griewank, A.: Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation. No. 19 in Frontiers in Appl. Math. SIAM, Philadelphia, PA (2000)

    MATH  Google Scholar 

  25. Griewank, A.: A mathematical view of automatic differentiation. Acta Numerica 12, 1–78 (2003)

    CrossRef  MathSciNet  Google Scholar 

  26. Griewank, A., Corliss, G.F. (eds.): Automatic Differentiation of Algorithms: Theory, Implementation, and Application. SIAM, Philadelphia, PA (1991)

    Google Scholar 

  27. Griewank, A., Juedes, D., Utke, J.: Algorithm 755: ADOL-C: A Package for the Automatic Differentiation of Algorithms Written in C/C++. ACM Transactions on Mathematical Software 22(2), 131–167 (1996)

    CrossRef  MATH  Google Scholar 

  28. Guertler, N.: Parallel Automatic Differentiation of a Quantum Plasma Code (2007). Computer science diploma thesis, RWTH-Aachen University, available at http://www.sc. rwth-aachen.de/Diplom/Thesis-Guertler.pdf

  29. Heimbach, P., Hill, C., Giering, R.: Automatic generation of efficient adjoint code for a parallel Navier-Stokes solver. In: P.M.A. Sloot, C.J.K. Tan, J.J. Dongarra, A.G. Hoekstra (eds.) Computational Science – ICCS 2002, Proc. Int. Conf. on Computational Science, Amsterdam, The Netherlands, April 21–24, 2002. Part II, Lecture Notes in Computer Science, vol. 2330, pp. 1019–1028. Springer, Berlin (2002)

    Google Scholar 

  30. Hovland, P.D., Bischof, C.H.: Automatic differentiation of message-passing parallel programs. In: Proc. 1st Merged Int. Parallel Processing Symposium and Symposium on Parallel and Distributed Processing, pp. 98–104. IEEE Computer Society Press (1998)

    Google Scholar 

  31. Kowarz, A., Walther, A.: Optimal Checkpointing for Time-Stepping Procedures in ADOL-C. In: V.N. Alexandrov, G.D. van Albada, P.M.A. Sloot, J. Dongarra (eds.) Computational Science – ICCS 2006, Lecture Notes in Computer Science, vol. 3994, pp. 541–549. Springer, Heidelberg (2006)

    CrossRef  Google Scholar 

  32. Kowarz, A., Walther, A.: Efficient Calculation of Sensitivities for Optimization Problems. Discussiones Mathematicae. Differential Inclusions, Control and Optimization 27, 119–134 (2007). To appear

    Google Scholar 

  33. Kowarz, A., Walther, A.: Parallel Derivative Computation Using ADOL-C. Proceedings PASA 2008, Lecture Notes in Informatics, Vol. 124, pp. 83–92 (2008)

    Google Scholar 

  34. Luca, L.D., Musmanno, R.: A parallel automatic differentiation algorithm for simulation models. Simulation Practice and Theory 5(3), 235–252 (1997)

    CrossRef  Google Scholar 

  35. Mancini, M.: A parallel hierarchical approach for automatic differentiation. In: G. Corliss, C. Faure, A. Griewank, L. Hascoët, U. Naumann (eds.) Automatic Differentiation of Algorithms: From Simulation to Optimization, Computer and Information Science, chap. 27, pp. 231–236. Springer, New York, NY (2002)

    Google Scholar 

  36. P. Heimbach and C. Hill and R. Giering: An efficient exact adjoint of the parallel MIT general circulation model, generated via automatic differentiation. Future Generation Computer Systems 21(8), 1356–1371 (2005)

    CrossRef  Google Scholar 

  37. Rall, L.B.: Automatic Differentiation: Techniques and Applications, Lecture Notes in Computer Science, vol. 120. Springer, Berlin (1981)

    MATH  Google Scholar 

  38. Rasch, A., Bücker, H.M., Bischof, C.H.: Automatic Computation of Sensitivities for a Parallel Aerodynamic Simulation. In: C. Bischof, M. Bücker, P. Gibbon, G. Joubert, T. Lippert, B. Mohr, F. Peters (eds.) Parallel Computing: Architectures, Algorithms and Applications, Proceedings of the International Conference ParCo 2007, Advances in Parallel Computing, vol. 15, pp. 303–310. IOS Press, Amsterdam, The Netherlands (2008)

    Google Scholar 

  39. Spiegel, A., an Mey, D., Bischof, C.: Hybrid parallelization of CFD applications with dynamic thread balancing. In: Proc. PARA04 Workshop, Lyngby, Denmark, June 2004, Lecture Notes in Computer Science, vol. 3732, pp. 433–441. Springer Verlag (2006)

    Google Scholar 

  40. Terboven, C., Spiegel, A., an Mey, D., Gross, S., Reichelt, V.: Parallelization of the C++ navier-stokes solver DROPS with OpenMP. In: Parallel Computing (ParCo 2005): Current & Future Issues of High-End Computing, Malaga, Spain, September 2005, NIC Series, vol. 33, pp. 431–438 (2006)

    Google Scholar 

  41. Walther, A., Griewank, A.: Advantages of binomial checkpointing for memory-reduces adjoint calculations. In: M. Feistauer, V. Dolejsi, P. Knobloch, K. Najzar (eds.) Numerical mathematics and advanced applications, Proceedings ENUMATH 2003, pp. 834–843. Springer (2004)

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2008 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Bischof, C., Guertler, N., Kowarz, A., Walther, A. (2008). Parallel Reverse Mode Automatic Differentiation for OpenMP Programs with ADOL-C. In: Bischof, C.H., Bücker, H.M., Hovland, P., Naumann, U., Utke, J. (eds) Advances in Automatic Differentiation. Lecture Notes in Computational Science and Engineering, vol 64. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-68942-3_15

Download citation