Numerical Algorithms

, Volume 72, Issue 3, pp 775–811 | Cite as

A Hitchhiker’s Guide to Automatic Differentiation

  • Philipp H. W. Hoffmann
Original Paper


This article provides an overview of some of the mathematical principles of Automatic Differentiation (AD). In particular, we summarise different descriptions of the Forward Mode of AD, like the matrix-vector product based approach, the idea of lifting functions to the algebra of dual numbers, the method of Taylor series expansion on dual numbers and the application of the push-forward operator, and explain why they all reduce to the same actual chain of computations.We further give a short mathematical description of some methods of higher-order Forward AD and, at the end of this paper, briefly describe the Reverse Mode of Automatic Differentiation.


Automatic Differentiation Forward AD Reverse AD Dual numbers 

Mathematics Subject Classifications (2010)

65-02 65K99 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Berz, M.: Differential algebraic description of beam dynamics to very high orders. Particle Accelerators 24, 109–124 (1989)Google Scholar
  2. 2.
    Bischof, C.H.: On the Automatic Differentiation of Computer Programs and an Application to Multibody Systems. IUTAM Symp. Optim. Mech. Syst. Solid Mech. Appl. 43, 41–48 (1996)zbMATHGoogle Scholar
  3. 3.
    Clifford, W.K.: Preliminary Sketch of Bi-quaternions. Proc. London Math. Soc. 4, 381–395 (1873)MathSciNetzbMATHGoogle Scholar
  4. 4.
    Dixon, L.: Automatic differentiation: Calculation of the Hessian, 2nd edn. In: Encyclopedia of Optimization, pp. 133–137. Springer Science+Business Media, LLC. (2009)Google Scholar
  5. 5.
    Garczynski, V.: Remarks on differential algebraic approach to particle beam optics by M. Berz. Nucl. Inst. Methods Phys. Res. Sect. A 334(2–3), 294–298 (1993)CrossRefGoogle Scholar
  6. 6.
    Gower, R.M., Mello, M.P.: A new framework for the computation of Hessians. Optim. Methods Softw. 27(2), 251–273 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  7. 7.
    Griewank, A.: Achieving logarithmic growth of temporal and spatial complexity in reverse automatic differentiation. Optim. Methods Softw. 1, 35–54 (1992)CrossRefGoogle Scholar
  8. 8.
    Griewank, A.: A mathematical view of automatic differentiation. Acta Numerica 12, 321–398 (2003)MathSciNetCrossRefzbMATHGoogle Scholar
  9. 9.
    Griewank, A., Walther, A.: Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation, 2nd edn. SIAM, Philadelphia (2008)CrossRefzbMATHGoogle Scholar
  10. 10.
    Baydin, A.G., Pearlmutter, B.A.: DiffSharp: Automatic differentiation library, version of 17th June 2015.
  11. 11.
    Baydin, A.G., Pearlmutter, B.A., Radul, A.A., Siskind, J.M.: Automatic differentiation and machine learning: A survey. arXiv preprint. arXiv:1502.05767 (2015)
  12. 12.
    Hubbard, J.H., Lundell, B.E.: A first look at differential algebra. Amer. Math. Monthly 118(3), 245–261 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  13. 13.
    John, F.: Partial Differential Equations, 2nd edn. Springer, New York-Heidelberg-Berlin (1975)CrossRefzbMATHGoogle Scholar
  14. 14.
    Kalman, D.: Double recursive multivariate automatic differentiation. Math. Mag. 75(3), 187–202 (2002)MathSciNetCrossRefGoogle Scholar
  15. 15.
    Karczmarczuk, J.: Functional coding of differential forms. In: Proc First Scottish Workshop on Functional Programming. Stirling (1999)Google Scholar
  16. 16.
    Karczmarczuk, J.: Functional differentiation of computer programs. In: Proc of the III ACM SIGPLAN International Conference on Functional Programming, pp. 195–203. Baltimore (1998)Google Scholar
  17. 17.
    Karczmarczuk, J.: Functional differentiation of computer programs. Higher-Order Symb. Comput. 14(1), 35–57 (2001)CrossRefzbMATHGoogle Scholar
  18. 18.
    Lang, S.: Introduction to Differentiable Manifolds, 2nd edn. Springer-Verlag, New York-Heidelberg-Berlin (2002)zbMATHGoogle Scholar
  19. 19.
    Manzyuk, O.: A simply typed λ-calculus of forward automatic differentiation. Electron. Notes Theor. Comput. Sci. 286, 257–272 (2012)MathSciNetCrossRefGoogle Scholar
  20. 20.
    Pearlmutter, B.A., Siskind, J.M.: Lazy multivariate higher-order forward-mode AD. In: Proc of the 2007 Symposium on Principles of Programming Languages, pp. 155–160. Nice (2007)Google Scholar
  21. 21.
    Pearlmutter, B.A., Siskind, J.M.: Reverse-mode AD in a functional framework: Lambda the ultimate backpropagator. TOPLAS 30(2), 1–36 (2008)CrossRefGoogle Scholar
  22. 22.
    Rall, L.B.: Differentiation and generation of Taylor coefficients in Pascal-SC. In: A New Approach to Scientific Computation, pp 291–309. Academic Press, New York (1983)Google Scholar
  23. 23.
    Rall, L.B.: The arithmetic of differentiation. Math. Mag. 59, 275–282 (1986)MathSciNetCrossRefzbMATHGoogle Scholar
  24. 24.
    Ritt, J.: Differential Algebra, American Mathematical Society Colloquium Publications, vol. XXXIII. American Mathematical Society, New York (1950)Google Scholar
  25. 25.
    Siskind, J.M., Pearlmutter, B.A.: Nesting forward-mode AD in a functional framework. Higher-Order Symb. Comput. 21(4), 361–376 (2008)CrossRefzbMATHGoogle Scholar
  26. 26.
    Wengert, R.: A simple automatic derivative evaluation program. Commun. ACM 7(8), 463–464 (1964)CrossRefzbMATHGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  1. 1.Department of Computer ScienceNational University of IrelandMaynoothIreland

Personalised recommendations