Abstract
Notwithstanding the superiority of the Leibniz notation for differential calculus, the dot-and-bar notation predominantly used by the Automatic Differentiation community is resolutely Newtonian. In this paper we extend the Leibniz notation to include the reverse (or adjoint) mode of Automatic Differentiation, and use it to demonstrate the stepwise numerical equivalence of the three approaches using the reverse mode to obtain second order derivatives, namely forward-over-reverse, reverse-over-forward, and reverse-over-reverse.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
Archimedes’ construction for the volume of a sphere probably entitles him to be considered the first to discover integral calculus.
- 2.
Sharaf al-Din al-Tusi already knew the derivative of a cubic in 1209 [1], but did not extend this result to more general functions.
- 3.
Although this fact did not become public knowledge until 1761, nearly 50 years later.
- 4.
The word Algorithm derives from the eponymous eighth century mathematician Al-Khwarizmi, known in Latin as Algoritmi. Prior to Leibniz, the term referred exclusively to mechanical arithmetical procedures, such as the process for extraction of square roots, applied (by a human) to numerical values rather than symbolic expressions. The italics are in the Latin original: “Ex cognito hoc velut Algorithmo, ut ita dicam, calculi hujus, quem voco differentialem.”
- 5.
The Analytical Society was founded by Babbage and some of his friends in 1812. So successful was their program of reform that 11 of the 16 original members subsequently became professors at Cambridge.
- 6.
Rouse Ball writes [4] “It would seem that the chief obstacle to the adoption of analytical methods and the notation of the differential calculus arose from the professorial body and the senior members of the senate, who regarded any attempt at innovation as a sin against the memory of Newton.”
- 7.
Since y ≡ f(w, x) we allow ourselves to write \(\frac{\partial f} {\partial x}\) interchangeably with \(\frac{\partial y} {\partial x}\).
- 8.
Actually the tradition of treating differentials as independent variables in their own right was begun by d’Alembert as a response to Berkeley’s criticisms of the infinitesimal approach [6], but significantly he made no changes to Leibniz’s original notation for them. Leibniz’s formulation allows for the possibility of non-negligible differential values, referring [19] to “the fact, until now not sufficiently explored, that dx, dy, dv, dw, dz can be taken proportional [my italics] to the momentary differences, that is, increments or decrements, of the corresponding x, y, v, w, z”, and Leibniz is careful to write \(d(xv) = xdv + vdx\), without the term dxdv.
- 9.
The familiarity comes in part from the fact that this is the very equation of which Hademard said [15] “que signifie ou que représente l’égalité? A mon avis, rien du tout.” [“What is meant, or represented, by this equality? In my opinion, nothing at all.”] It is good that the automatic differentiation community is now in a position to give Hadamard a clear answer: (y, dy, d 2 y) is the content of an active variable.
- 10.
The variables x and y may be vectors: in this case the corresponding differential dx and bariential by are respectively a column vector with components dx j and a row vector with components by i ; f′ is the matrix \({J}_{j}^{i} = {\partial }_{j}{f}^{i} = \partial {f}^{i}/\partial {x}^{j}\), and f′ is the mixed third order tensor \({K}_{jk}^{i} = {\partial }_{jk}^{2}{f}^{i} = {\partial }^{2}{f}^{i}/\partial {x}^{j}\partial {x}^{k}\).
- 11.
If x is a vector then pbx is a column vector.
- 12.
Recall that this term includes all combinations of differentials and barientials.
References
Al-Tusi, S.A.D.: Treatise on Equations. Manuscript, Baghdad (1209)
Anonymous: An account of the book entitled commercium epistolicum collinii et aliorum, de analysi promota; published by order of the Royal Society, in relation to the dispute between Mr. Leibnitz [sic] and Dr. Keill, about the right of invention of the method of fluxions, by some called the differential method. Philosophical Transaction of the Royal Society of London 342, 173–224 (January and February 1714/5)
Babbage, C.: Passages from the Life of a Philosopher. London (1864)
Ball, W.W.R.: A History of the Study of Mathematics at Cambridge. Cambridge (1889)
Barrow, I.: Lectiones Opticae et Geometricae. London (1669)
Berkeley, G.: The Analyst; or, A Discourse Addressed to an Infidel Mathematician, Wherein it is examined whether the Object, Principles, and Inferences of the modern Analysis are more distinctly conceived, or more evidently deduced, than Religious Mysteries and Points of Faith. London (1734)
Christianson, B.: Automatic Hessians by reverse accumulation. Technical Report NOC TR228, Numerical Optimisation Centre, Hatfield Polytechnic, Hatfield, United Kingdon (1990)
Christianson, B.: Reverse accumulation and accurate rounding error estimates for Taylor series coefficients. Optimization Methods and Software 1(1), 81–94 (1991). Also appeared as Tech. Report No. NOC TR239, The Numerical Optimisation Centre, University of Hertfordshire, U.K., July 1991
Christianson, B.: Automatic Hessians by reverse accumulation. IMA Journal of Numerical Analysis 12(2), 135–150 (1992)
Dixon, L.C.W.: Use of automatic differentiation for calculating Hessians and Newton steps. In: Griewank and Corliss [11], pp. 114–125
Griewank, A., Corliss, G.F. (eds.): Automatic Differentiation of Algorithms: Theory, Implementation, and Application. SIAM, Philadelphia, PA (1991)
Griewank, A., Juedes, D., Srinivasan, J., Tyner, C.: ADOL-C: A package for the automatic differentiation of algorithms written in C/C++. Preprint MCS-P180-1190, Mathematics and Computer Science Division, Argonne National Laboratory, Argonne, Illinois (1990)
Griewank, A., Juedes, D., Utke, J.: Algorithm 755: ADOL-C: A package for the automatic differentiation of algorithms written in C/C++. ACM Transactions on Mathematical Software 22(2), 131–167 (1996). URL http://doi.acm.org/10.1145/229473.229474
Griewank, A., Walther, A.: Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation, 2nd edn. No. 105 in Other Titles in Applied Mathematics. SIAM, Philadelphia, PA (2008). URL http://www.ec-securehost.com/SIAM/OT105.html
Hadamard, J.: La notion de différentiel dans l’enseignement. Mathematical Gazette XIX(236), 341–342 (1935)
Kubota, K.: PADRE2 Version 1 Users Manual. Research Memorandum RMI 90-01, Department of Mathematical Engineering and Information Physics, Faculty of Engineering, University of Tokyo, Tokyo, Japan (1990)
Kubota, K.: PADRE2, a Fortran precompiler yielding error estimates and second derivatives. In: Griewank and Corliss [11], pp. 251–262
Leibniz, G.W.: Machina arithmetica in qua non additio tantum et subtractio sedet multiplicatio nullo, division veropaene nullo animi labore peragantur. [An arithmetic machine which can be used to carry out not only addition and subtraction but also multiplication with no, and division with really almost no, intellectual exertion.]. Manuscript, Hannover (1685). A translation by Mark Kormes appears in ‘A Source Book in Mathematics’ by David Eugene Smith, Dover (1959)
Leibniz, G.W.: Nova methodvs pro maximis et minimis, itemque tangentibus, quae nec fractas, nec irrationales quantitates moratur, et singulare pro illis calculi genus. [A new method for maxima and minima as well as tangents, which is impeded neither by fractional nor irrational quantities, and a remarkable type of calculus for them.]. Acta Erutitorium (October 1684)
Menabrea, L.F.: Sketch of the analytical engine invented by Charles Babbage, with notes by the translator Augusta Ada King, Countess of Lovelace. Taylor’s Scientific Memoirs 3, 666–731 (1842)
Newton, I.: Philosophiae Naturalis Principia Mathematica. London (1687)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Christianson, B. (2012). A Leibniz Notation for Automatic Differentiation. In: Forth, S., Hovland, P., Phipps, E., Utke, J., Walther, A. (eds) Recent Advances in Algorithmic Differentiation. Lecture Notes in Computational Science and Engineering, vol 87. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-30023-3_1
Download citation
DOI: https://doi.org/10.1007/978-3-642-30023-3_1
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-30022-6
Online ISBN: 978-3-642-30023-3
eBook Packages: Mathematics and StatisticsMathematics and Statistics (R0)