Higher-Order and Symbolic Computation

, Volume 21, Issue 3, pp 311–331 | Cite as

On the implementation of automatic differentiation tools

  • Christian H. BischofEmail author
  • Paul D. Hovland
  • Boyana Norris


Automatic differentiation is a semantic transformation that applies the rules of differential calculus to source code. It thus transforms a computer program that computes a mathematical function into a program that computes the function and its derivatives. Derivatives play an important role in a wide variety of scientific computing applications, including numerical optimization, solution of nonlinear equations, sensitivity analysis, and nonlinear inverse problems. We describe the forward and reverse modes of automatic differentiation and provide a survey of implementation strategies. We describe some of the challenges in the implementation of automatic differentiation tools, with a focus on tools based on source transformation. We conclude with an overview of current research and future opportunities.


Semantic transformation Automatic differentiation 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Abate, J., Bischof, C., Carle, A., Roh, L.: Algorithms and design for a second-order automatic differentiation module. In: Proc. Int. Symposium on Symbolic and Algebraic Computing (ISSAC) ’97, New York, pp. 149–155. Association of Computing Machinery (1997) Google Scholar
  2. 2.
    Aho, A.V., Sethi, R., Ullman, J.D.: Compilers: Principles, Techniques and Tools. Addison–Wesley, Reading (1986) Google Scholar
  3. 3.
    Allen, R., Kennedy, K.: Optimizing Compilers for Modern Architectures: A Dependence-based Approach. Morgan Kaufmann, San Mateo (2002) Google Scholar
  4. 4.
    Amarasinghe, S.P., Anderson, J.M., Lam, M.S., Tseng, C.W.: The SUIF compiler for scalable parallel machine. In: Proceedings of the Seventh SIAM Conference on Parallel Processing for Scientific Computing (1995) Google Scholar
  5. 5.
    Aubert, P., Di Césaré, N.: Expression templates and forward mode automatic differentiation. In [26], Chap. 37, pp. 311–315 (2001) Google Scholar
  6. 6.
    Bagge, O.S., Kalleberg, K.T., Haveraaen, M., Visser, E.: Design of the CodeBoost transformation system for domain-specific optimisation of C++ programs. In: Binkley, D., Tonella, P. (eds.): Third International Workshop on Source Code Analysis and Manipulation (SCAM 2003). IEEE Computer Society Press, Amsterdam (2003, to appear) Google Scholar
  7. 7.
    Bartholomew-Biggs, M.: OPFAD—A users guide to the OPtima forward automatic differentiation tool. Technical report, Numerical Optimization Centre, University of Hertfordsshire (1995) Google Scholar
  8. 8.
    Bartholomew-Biggs, M.C., Brown, S., Christianson, B., Dixon, L.C.W.: The efficient calculation of gradients, Jacobians and Hessians. Technical Report NOC TR301, The Numerical Optimisation Center, University of Hertfordshire, Hatfield, U.K. (1995) Google Scholar
  9. 9.
    Barton, J.J., Nackman, L.R.: Automatic differentiation. C++ Rep. 8(2), 61–63 (1996) Google Scholar
  10. 10.
    Beda, L.M., Korolev, L.N., Sukkikh, N.V., Frolova, T.S.: Programs for automatic differentiation for the machine BESM. Technical Report, Institute for Precise Mechanics and Computation Techniques, Academy of Science, Moscow, USSR (1959) (In Russian) Google Scholar
  11. 11.
    Bell, B.M.: CppAD User Manual. Available at (2003)
  12. 12.
    Bendtsen, C., Stauning, O.: FADBAD, a flexible C++ package for automatic differentiation. Technical Report IMM-REP-1996-17, Department of Mathematical Modelling, Technical University of Denmark, Lyngby, Denmark (1996) Google Scholar
  13. 13.
    Berz, M., Bischof, C., Corliss, G., Griewank, A. (eds.): Computational Differentiation: Techniques, Applications, and Tools. SIAM, Philadelphia (1996) zbMATHGoogle Scholar
  14. 14.
    Binkley, D.W., Gallagher, K.B.: Program slicing. Adv. Comput. 43, 1–50 (1996) CrossRefGoogle Scholar
  15. 15.
    Bischof, C., Roh, L.: The automatic differentiation intermediate form (AIF). Unpublished Information (1996) Google Scholar
  16. 16.
    Bischof, C., Carle, A., Corliss, G., Griewank, A., Hovland, P.: ADIFOR: Generating derivative codes from Fortran programs. Sci. Program. 1(1), 11–29 (1992) Google Scholar
  17. 17.
    Bischof, C., Carle, A., Khademi, P., Mauer, A.: ADIFOR 2.0: Automatic differentiation of Fortran 77 programs. IEEE Comput. Sci. Eng. 3(3), 18–32 (1996) CrossRefGoogle Scholar
  18. 18.
    Bischof, C., Roh, L., Mauer, A.: ADIC—An extensible automatic differentiation tool for ANSI-C. Softw. Pract. Exp. 27(12), 1427–1456 (1997) CrossRefGoogle Scholar
  19. 19.
    Bischof, C.H., Bücker, H.M., Lang, B., Rasch, A.: An interactive environment for supporting the paradigm shift from simulation to optimization. In: 4th Workshop on Parallel/High-Performance OO Scientific Computing (POOSC’01) 14 October 2001, at the ACM Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA’01) 14–18 October, Tampa Bay, FL (2001, to appear) Google Scholar
  20. 20.
    Bischof, C.H., Bücker, H.M., Lang, B., Rasch, A., Vehreschild, A.: Combining source transformation and operator overloading techniques to compute derivatives for MATLAB programs. Preprint RWTH-CS-SC-02-04, Institute for Scientific Computing, Aachen University of Technology (2002) Google Scholar
  21. 21.
    Brooke, A., Kendrick, D., Meeraus, A.: GAMS: A User’s Guide. The Scientific Press, South San Francisco (1988) Google Scholar
  22. 22.
    Brown, S.: OPRAD—A users guide to the OPtima reverse automatic differentiation tool. Technical report, Numerical Optimization Centre, University of Hertfordsshire (1995) Google Scholar
  23. 23.
    Christianson, B., Dixon, L.C.W., Brown, S.: Sharing storage using dirty vectors. In [13], pp. 107–115 Google Scholar
  24. 24.
    Christianson, D.B.: Automatic Hessians by reverse accumulation in ada. IMA J. Numer. Anal. (1991) Presented at SIAM Workshop on Automatic Differentiation of Algorithms, Breckenridge, CO, January 1991 Google Scholar
  25. 25.
    Coleman, T.F., Verma, A.: ADMIT-1: Automatic differentiation and MATLAB interface toolbox. ACM Trans. Math. Softw. 26(1), 150–175 (2000) zbMATHCrossRefMathSciNetGoogle Scholar
  26. 26.
    Corliss, G., Faure, C., Griewank, A., Hascoët, L., Naumann, U. (eds.): Automatic Differentiation: From Simulation to Optimization. Computer and Information Science Springer, New York (2001) Google Scholar
  27. 27.
    Edison Design Group: EDG C++ Front End (2003).
  28. 28.
    Faure, C.: Adjoining strategies for multi-layered programs. Optim. Methods Softw. (2001, to appear). Also appeared as INRIA Rapport de recherche no. 3781, BP 105-78153 Le Chesnay Cedex, France (1999) Google Scholar
  29. 29.
    Faure, C., Naumann, U.: Minimizing the tape size. In [26], Chap. 34, pp. 293–298 (2001) Google Scholar
  30. 30.
    Forth, S.: An efficient implementation of AD in MATLAB. Presentation at Joint University of Hertfordshire/Cranfield University (RMCS Shrivenham) Automatic Differentiation Symposium (2001). Available at
  31. 31.
    Fourer, R., Gay, D.M., Kernighan, B.W.: AMPL: A Modeling Language for Mathematical Programming. The Scientific Press, South San Francisco (1993) Google Scholar
  32. 32.
    Frazier, Z.: PyAD User manual. Available at (2003)
  33. 33.
    Giering, R., Kaminski, T.: Recipes for adjoint code construction. ACM TOMS 24(4), 437–474 (1998) zbMATHCrossRefGoogle Scholar
  34. 34.
    Giering, R., Kaminski, T.: Applying TAF to generate efficient derivative code of Fortran 77-95 programs. In: Proceedings of GAMM 2002, Augsburg, Germany (2002) Google Scholar
  35. 35.
    Griewank, A.: On automatic differentiation. In: Mathematical Programming: Recent Developments and Applications, pp. 83–108. Kluwer Academic, Amsterdam (1989) Google Scholar
  36. 36.
    Griewank, A.: Personal communication (1998) Google Scholar
  37. 37.
    Griewank, A.: Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation. SIAM, Philadelphia (2000) zbMATHGoogle Scholar
  38. 38.
    Griewank, A.: Achieving logarithmic growth of temporal and spatial complexity in reverse automatic differentiation. Optim. Methods Softw. 1, 35–54 (1992) CrossRefGoogle Scholar
  39. 39.
    Griewank, A., Corliss, G.F.: Automatic Differentiation of Algorithms: Theory, Implementation, and Application. SIAM, Philadelphia (1991) zbMATHGoogle Scholar
  40. 40.
    Griewank, A., Reese, S.: On the calculation of Jacobian matrices by the Markowitz rule. In [39], pp. 126–135 (1991) Google Scholar
  41. 41.
    Griewank, A., Juedes, D., Utke, J.: ADOL-C, a package for the automatic differentiation of algorithms written in C/C++. ACM Trans. Math. Softw. 22(2), 131–167 (1996) zbMATHCrossRefGoogle Scholar
  42. 42.
    Grimm, J., Pottier, L., Rostaing-Schmidt, N.: Optimal time and minimum space-time product for reversing a certain class of programs. In [13], pp. 95–106 (1996) Google Scholar
  43. 43.
    Guyer, S.Z., Lin, C.: Optimizing the use of high performance software libraries. Lecture Notes in Computer Science, vol. 2017, pp. 227–243. Springer, Berlin (2001) Google Scholar
  44. 44.
    Hill, D.R., Rich, L.C.: Automatic differentiation in MATLAB. Appl. Numer. Math. 9, 33–43 (1992) zbMATHCrossRefGoogle Scholar
  45. 45.
    Hinkins, R.L.: Parallel computation of automatic differentiation applied to magnetic field calculations. Master’s thesis, University of California, Berkeley (1994) Google Scholar
  46. 46.
    Hinsen, K.: Scientific python collection. Module Scientific. Functions. Derivatives (2003). Available at
  47. 47.
    Hovland, P.D., Naumann, U., Norris, B.: An XML-based platform for semantic transformation of numerical programs. Preprint ANL/MCS-P950-0402, Mathematics and Computer Science Division, Argonne National Laboratory. Proceedings of Software Engineering and Applications (SEA 2002) (2002, to appear) Google Scholar
  48. 48.
    Huss, R.E.: An ADA library for automatic evaluation of derivatives. Appl. Math. Comput. 35(2), 103–123 (1990) zbMATHCrossRefMathSciNetGoogle Scholar
  49. 49.
    Jerrell, M.: Automatic differentiation using almost any language. ACM SIGNUM Newsl. pp. 2–9 (1989) Google Scholar
  50. 50.
    Juedes, D.W.: A taxonomy of automatic differentiation tools. In [39], pp. 315–329 (1991) Google Scholar
  51. 51.
    Kahrimanian, H.G.: Analytical differentiation by a digital computer. Master’s thesis, Temple University (1953) Google Scholar
  52. 52.
    Kalman, D., Lindell, R.: Automatic differentiation in astrodynamical modeling. In [39], pp. 228–243 (1991) Google Scholar
  53. 53.
    Karczmarczuk, J.: Functional differentiation of computer programs. J. HOSC 14, 35–57 (2001) zbMATHGoogle Scholar
  54. 54.
    Kennedy, K., Broom, B., Cooper, K., Dongarra, J., Fowler, R., Gannon, D., Johnsson, L., Mellor-Crummey, J., Torczon, L.: Telescoping languages: A strategy for automatic generation of scientific problem-solving systems from annotated libraries. J. Parallel Distrib. Comput. 61(12), 1803–1826 (2001) zbMATHCrossRefGoogle Scholar
  55. 55.
    Korelc, J.: Hybrid system for multi-language and multi-environment generation of numerical codes. In: Proceedings of the 2001 International Symposium on Symbolic and algebraic computation, pp. 209–216. ACM Press (2001) Google Scholar
  56. 56.
    Lawson, C.L.: Computing derivatives using W-arithmetic and U-arithmetic. Internal computing memorandum CM–286, Jet Propulsion Laboratory, Pasadena (1971) Google Scholar
  57. 57.
    Lesk, A.M.: Dynamic computation of derivatives. Commun. ACM 10(9), 571–572 (1967) zbMATHCrossRefGoogle Scholar
  58. 58.
    Maany, Z.: Ada automatic differentiation package for the optimization of functions of many variables. Technical Report NOC TR209, The Numerical Optimisation Center, Hatfield Polytechnic, Hatfield, UK (1989) Google Scholar
  59. 59.
    Martins, J.R.R.A., Kroo, I.M., Alonso, J.J.: An automated method for sensitivity analysis using complex variables. In: Proceedings of the 38th Aerospace Sciences Meeting, Reno, NV (2000) Google Scholar
  60. 60.
    Martins, J.R.R.A., Sturdza, P., Alonso, J.J.: The connection between the complex-step derivative approximation and algorithmic differentiation. In: Proceedings of the 39th Aerospace Sciences Meeting, Reno, NV (2001). Complexify.h and derivify.h available at
  61. 61.
    Michelotti, L.: MXYZPTLK: A C++ Hacker’s implementation of automatic differentiation. In [39], pp. 218–227 (1991). Software available at
  62. 62.
    Monagan, M., Rodoni, R.R.: An implementation of the forward and reverse mode of automatic differentiation in maple. In: Berz, M., Bischof, C., Corliss, G., Griewank, A. (eds.) Computational Differentiation: Techniques, Applications, and Tools, pp. 353–362. SIAM, Philadelphia (1996) Google Scholar
  63. 63.
    Muchnick, S.S.: Advanced Compiler Design and Implementation. Morgan Kaufmann, San Mateo (1997) Google Scholar
  64. 64.
    NAG-AD: Differentiation enabled Fortran compiler technology (2003).
  65. 65.
    Naumann, U.: Reducing the memory requirement in reverse mode automatic differentiation by solving TBR flow equations. In: Sloot, P.M.A., Tan, C.J.K., Dongarra, J.J. (eds.) Proceedings of the International Conference on Computational Science, Part II, Amsterdam, The Netherlands, April 21–24, 2002. Lecture Notes in Computer Science, vol. 2330, pp. 1039–1048. Springer, Berlin (2002) Google Scholar
  66. 66.
    Naumann, U.: Statement level optimality of tangent-linear and adjoint models. Preprint ANL-MCS/P1066-0603, Argonne National Laboratory (2003) Google Scholar
  67. 67.
    Naumann, U., Gottschling, P.: Simulated annealing for optimal pivot selection in Jacobian accumulation. In: Albrecht, A. (ed.): Stochastic Algorithms, Foundations and Applications—SAGA’03. Berlin, Springer (2003, to appear). See also
  68. 68.
    Neidinger, R.D.: Automatic differentiation and APL. Coll. Math. J. 20(3), 238–251 (1989) CrossRefGoogle Scholar
  69. 69.
    Nilsson, H.: Functional automatic differentiation with Dirac impulses. ACM SIGPLAN Not. 38(9), 153–164 (2003) CrossRefGoogle Scholar
  70. 70.
    Nolan, J.F.: Analytical differentiation on a digital computer. Master’s thesis, Massachusetts Institute of Technology (1953) Google Scholar
  71. 71.
    Pryce, J.D., Reid, J.K.: ADO1, a Fortran 90 code for automatic differentiation. Technical Report RAL-TR-1998-057, Rutherford Appleton Laboratory, Chilton, Didcot, Oxfordshire, OX11 OQX, England (1998) Google Scholar
  72. 72.
    Quinlan, D.: ROSE: Compiler support for object-oriented frameworks. Parallel Process. Lett. 10(2/3), 215 (2000) CrossRefGoogle Scholar
  73. 73.
    Reps, T., Rosay, G.: Precise interprocedural chopping. In: Kaiser, G.E. (ed.) SIGSOFT’95: Proceedings of the Third ACM SIGSOFT Symposium on the Foundations of Software Engineering, pp. 41–52. ACM Press (1995) Google Scholar
  74. 74.
    Restrepo, J.M., Leaf, G.K., Griewank, A.: Circumventing storage limitations in variational data assimilation. SIAM J. Sci. Comput. 19, 1586–1605 (1998) zbMATHCrossRefMathSciNetGoogle Scholar
  75. 75.
    Rostaing, N., Dalmas, S., Galligo, A.: Automatic differentiation in Odyssee. Tellus 45a(5), 558–568 (1993) Google Scholar
  76. 76.
    Rote, G.: Path problems in graphs. In: Tinhofer, G., Mayr, E., Noltemeier, H., M.M.S. in cooperation with Albrecht, R. (eds.) Computational Graphs Theory, Springer-Verlag Computing Supplementum 7. Springer (1990) Google Scholar
  77. 77.
    SGI: WHIRL intermediate language specification (1999). Available at
  78. 78.
    Stamatiadis, S., Prosmiti, R., Farantos, S.C.: auto_deriv: Tool for automatic differentiation of a fortran code. Comput. Phys. Commun. 127(2&3), 343–355 (2000). Catalog number: ADLS zbMATHCrossRefGoogle Scholar
  79. 79.
    Tadjouddine, M., Forth, S.A., Pryce, J.D.: Hierarchicalal automatic differentiation by vertex elimination and source transformation. In: Kumar, V., Gavrilova, M.L., Tan, C.J.K. (eds.) Proceedings of the International Conference on Computational Science and its Applications, Part II, Montreal, Canada, May 18–21, 2003. Lecture Notes in Computer Science, vol. 2668, pp. 95–104. Springer, Berlin (2003) Google Scholar
  80. 80.
    TAPENADE: TAPENADE tutorial (2002).
  81. 81.
    Tsukanov, I.M.H.: Data structure and algorithms for fast automatic differentiation. Int. J. Numer. Methods Eng. 56(13), 1949–1972 (2003) zbMATHCrossRefGoogle Scholar
  82. 82.
    Veldhuizen, T.: Expression templates. C++ Rep. 7(5), 26–31 (1995) Google Scholar
  83. 83.
    Wengert, R.E.: A simple automatic derivative evaluation program. Commun. ACM 7(8), 463–464 (1964) zbMATHCrossRefGoogle Scholar
  84. 84.
    Wilkins, R.D.: Investigation of a new analytic model for numerical derivative evaluation. Commun. ACM 7(8), 465–471 (1964) zbMATHCrossRefGoogle Scholar

Copyright information

© US Government 2008

Authors and Affiliations

  • Christian H. Bischof
    • 1
    Email author
  • Paul D. Hovland
    • 2
  • Boyana Norris
    • 2
  1. 1.Institute for Scientific ComputingAachen University of TechnologyAachenGermany
  2. 2.Mathematics and Computer Science DivisionArgonne National LaboratoryArgonneUSA

Personalised recommendations