Using Programming Language Theory to Make Automatic Differentiation Sound and Efficient

  • Barak A. Pearlmutter
  • Jeffrey Mark Siskind
Part of the Lecture Notes in Computational Science and Engineering book series (LNCSE, volume 64)


This paper discusses a new automatic differentiation (AD) system that correctly and automatically accepts nested and dynamic use of the AD operators, without any manual intervention. The system is based on a new formulation of AD as highly generalized first-class citizens in a λ-calculus, which is briefly described. Because the λ-calculus is the basis for modern programming-language implementation techniques, integration of AD into the λ-calculus allows AD to be integrated into an aggressive compiler. We exhibit a research compiler which does this integration. Using novel analysis techniques, it accepts source code involving free use of a first-class forward AD operator and generates object code which attains numerical performance comparable to, or better than, the most aggressive existing AD systems.


Nesting lambda calculus multiple transformation forward mode optimization 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Church, A.: The Calculi of Lambda Conversion. Princeton University Press, Princeton, NJ (1941)Google Scholar
  2. 2.
    Hascoët, L., Pascual, V.: TAPENADE 2.1 user’s guide. Rapport technique 300, INRIA, Sophia Antipolis (2004). URL
  3. 3.
    Hughes, J.: Why functional programming matters. The Computer Journal 32(2), 98–107 (1989). URL CrossRefGoogle Scholar
  4. 4.
    Johnsson, T.: Lambda lifting: Transforming programs to recursive equations. In: Functional Programming Languages and Computer Architecture. Springer-Verlag, Nancy, France (1985)Google Scholar
  5. 5.
    Nielson, F., Nielson, H.R., Hankin, C.: Principles of Program Analysis. Springer-Verlag, New York (1999)zbMATHGoogle Scholar
  6. 6.
    Pearlmutter, B.A., Siskind, J.M.: Reverse-mode AD in a functional framework: Lambda the ultimate backpropagator. ACM Trans. on Programming Languages and Systems (2008). In pressGoogle Scholar
  7. 7.
    Siskind, J.M., Pearlmutter, B.A.: First-class nonstandard interpretations by opening closures. In: Proceedings of the 2007 Symposium on Principles of Programming Languages, pp. 71–6. Nice, France (2007)Google Scholar
  8. 8.
    Siskind, J.M., Pearlmutter, B.A.: Nesting forward-mode AD in a functional framework. Higher-Order and Symbolic Computation (2008). To appearGoogle Scholar
  9. 9.
    Siskind, J.M., Pearlmutter, B.A.: Putting the automatic back into AD: Part I, What’s wrong. Tech. Rep. TR-ECE-08-02, School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN, USA (2008). URL
  10. 10.
    Siskind, J.M., Pearlmutter, B.A.: Using polyvariant union-free flow analysis to compile a higher-order functional-programming language with a first-class derivative operator to efficient Fortran-like code. Tech. Rep. TR-ECE-08-01, School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN, USA (2008). URL
  11. 11.
    Wadler, P.L.: Comprehending monads. In: Proceedings of the 1990 ACM Conference on Lisp and Functional Programming, pp. 61–78. Nice, France (1990)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Barak A. Pearlmutter
    • 1
  • Jeffrey Mark Siskind
    • 2
  1. 1.Hamilton InstituteNational University of Ireland Maynooth, Co.KildareIreland
  2. 2.School of Electrical and Computer EngineeringPurdue UniversityWest LafayetteUSA

Personalised recommendations