Using Programming Language Theory to Make Automatic Differentiation Sound and Efficient
This paper discusses a new automatic differentiation (AD) system that correctly and automatically accepts nested and dynamic use of the AD operators, without any manual intervention. The system is based on a new formulation of AD as highly generalized first-class citizens in a λ-calculus, which is briefly described. Because the λ-calculus is the basis for modern programming-language implementation techniques, integration of AD into the λ-calculus allows AD to be integrated into an aggressive compiler. We exhibit a research compiler which does this integration. Using novel analysis techniques, it accepts source code involving free use of a first-class forward AD operator and generates object code which attains numerical performance comparable to, or better than, the most aggressive existing AD systems.
KeywordsNesting lambda calculus multiple transformation forward mode optimization
Unable to display preview. Download preview PDF.
- 1.Church, A.: The Calculi of Lambda Conversion. Princeton University Press, Princeton, NJ (1941)Google Scholar
- 2.Hascoët, L., Pascual, V.: TAPENADE 2.1 user’s guide. Rapport technique 300, INRIA, Sophia Antipolis (2004). URL http://www.inria.fr/rrrt/rt-0300.html
- 4.Johnsson, T.: Lambda lifting: Transforming programs to recursive equations. In: Functional Programming Languages and Computer Architecture. Springer-Verlag, Nancy, France (1985)Google Scholar
- 6.Pearlmutter, B.A., Siskind, J.M.: Reverse-mode AD in a functional framework: Lambda the ultimate backpropagator. ACM Trans. on Programming Languages and Systems (2008). In pressGoogle Scholar
- 7.Siskind, J.M., Pearlmutter, B.A.: First-class nonstandard interpretations by opening closures. In: Proceedings of the 2007 Symposium on Principles of Programming Languages, pp. 71–6. Nice, France (2007)Google Scholar
- 8.Siskind, J.M., Pearlmutter, B.A.: Nesting forward-mode AD in a functional framework. Higher-Order and Symbolic Computation (2008). To appearGoogle Scholar
- 9.Siskind, J.M., Pearlmutter, B.A.: Putting the automatic back into AD: Part I, What’s wrong. Tech. Rep. TR-ECE-08-02, School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN, USA (2008). URL ftp://ftp.ecn.purdue.edu/qobi/TR-ECE-08-02.pdf
- 10.Siskind, J.M., Pearlmutter, B.A.: Using polyvariant union-free flow analysis to compile a higher-order functional-programming language with a first-class derivative operator to efficient Fortran-like code. Tech. Rep. TR-ECE-08-01, School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN, USA (2008). URL http://docs.lib.purdue.edu/ecetr/367/
- 11.Wadler, P.L.: Comprehending monads. In: Proceedings of the 1990 ACM Conference on Lisp and Functional Programming, pp. 61–78. Nice, France (1990)Google Scholar