Mathematical Programming

, Volume 155, Issue 1–2, pp 81–103 | Cite as

Higher-order reverse automatic differentiation with emphasis on the third-order

  • R. M. GowerEmail author
  • A. L. Gower
Full Length Paper Series A


It is commonly assumed that calculating third order information is too expensive for most applications. But we show that the directional derivative of the Hessian (\(D^3 f(x)\cdot d\)) can be calculated at a cost proportional to that of a state-of-the-art method for calculating the Hessian matrix. We do this by first presenting a simple procedure for designing high order reverse methods and applying it to deduce several methods including a reverse method that calculates \(D^3f(x)\cdot d\). We have implemented this method taking into account symmetry and sparsity, and successfully calculated this derivative for functions with a million variables. These results indicate that the use of third order information in a general nonlinear solver, such as Halley–Chebyshev methods, could be a practical alternative to Newton’s method. Furthermore, high-order sensitivity information is used in methods for robust aerodynamic design. An efficient high-order differentiation tool could facilitate the use of similar methods in the design of other mechanical structures.


Automatic differentiation High-order methods Tensors vector products  Hessian matrix Sensitivity analysis 

Mathematics Subject Classification

15A69 (Tensor products) 65D25 (Numerical differentiation) 65F50 (Sparse matrices) 49Q12 (Sensitivity analysis) 


  1. 1.
    Griewank, A., Walther, A.: Evaluating derivatives, 2nd edn. Society for Industrial and Applied Mathematics (SIAM), Philadelphia (2008)zbMATHCrossRefGoogle Scholar
  2. 2.
    Gebremedhin, A.H., Manne, F., Pothen, A.: What color is your Jacobian? Graph coloring for computing derivatives. SIAM Rev. 47(4), 629–705 (2005)zbMATHMathSciNetCrossRefGoogle Scholar
  3. 3.
    Griewank, A., Naumann, U.: Accumulating Jacobians as chained sparse matrix products. Math. Progr. 95(3), 555–571 (2003)zbMATHMathSciNetCrossRefGoogle Scholar
  4. 4.
    Gower, R.M., Mello, M.P.: A new framework for the computation of Hessians. Optim. Methods Softw. 27(2), 251–273 (2012)zbMATHMathSciNetCrossRefGoogle Scholar
  5. 5.
    Gebremedhin, A.H., Tarafdar, A., Pothen, A., Walther, A.: Efficient computation of sparse Hessians using coloring and automatic differentiation. INFORMS J. Comput. 21(2), 209–223 (2009)zbMATHMathSciNetCrossRefGoogle Scholar
  6. 6.
    Gutiérrez, J.M., Hernández, M.A.: A family of Chebyshev–Halley type methods in Banach spaces. Bull. Aust. Math. Soc. 55(01), 113–130 (1997)zbMATHCrossRefGoogle Scholar
  7. 7.
    Amat, S., Busquier, S.: Third-order iterative methods under Kantorovich conditions. J. Math. Anal. Appl. 336(1), 243–261 (2007)zbMATHMathSciNetCrossRefGoogle Scholar
  8. 8.
    Wang, X., Kou, J.: Semilocal convergence and R-order for modified Chebyshev–Halley methods. J. Numer. Algorithms 64(1),105–126 (2013)Google Scholar
  9. 9.
    Ezquerro, J.A., Hern, M.A.: New Kantorovich-type conditions for Halley’s method. Appl. Numer. Anal. Comput. Math. 77(1), 70–77 (2005)CrossRefGoogle Scholar
  10. 10.
    Xu, X., Ling, Y.: Semilocal convergence for Halley’s method under weak Lipschitz condition. Appl. Math. Comput. 215(8), 3057–3067 (2009)zbMATHMathSciNetCrossRefGoogle Scholar
  11. 11.
    Susanto, H., Karjanto, N.: Newtons methods basins of attraction revisited. Appl. Math. Comput. 215(3), 1084–1090 (2009)zbMATHMathSciNetCrossRefGoogle Scholar
  12. 12.
    Yau, L., Ben-Israel, A.: The Newton and Halley methods for complex roots. Am. Math. Mon. 105(9), 806–818 (1998)zbMATHMathSciNetCrossRefGoogle Scholar
  13. 13.
    Gundersen, G., Steihaug, T.: Sparsity in higher order methods for unconstrained optimization. Optim. Methods Softw. 27(2), 275–294 (2012)zbMATHMathSciNetCrossRefGoogle Scholar
  14. 14.
    Gundersen, G., Steihaug, T.: On diagonally structured problems in unconstrained optimization using an inexact super Halley method. J. Comput. Appl. Math. 236(15), 3685–3695 (2012)zbMATHMathSciNetCrossRefGoogle Scholar
  15. 15.
    Gundersen, G., Steihaug, T.: On large-scale unconstrained optimization problems and higher order methods. Optim. Methods Softw. 25(3), 337–358 (2010)zbMATHMathSciNetCrossRefGoogle Scholar
  16. 16.
    Papadimitriou, D.I., Giannakoglou, K.C.: Robust design in aerodynamics using third-order sensitivity analysis based on discrete adjoint. Application to quasi-1D flows. Int. J. Numer. Methods Fluids 69, 691–709 (2012)MathSciNetCrossRefGoogle Scholar
  17. 17.
    Papdimitriou, D.I., Giannakoglou, K.C.: Third-order sensitivity analysis for robust aerodynamic design using continuous adjoint. Int. J. Numer. Methods Fluids 71, 652–670 (2013)CrossRefGoogle Scholar
  18. 18.
    Ozaki, I., Kimura, F., Berz, M.: Higher-order sensitivity analysis of finite element method by automatic differentiation. Comput. Mech. 16(4), 223–234 (1995)zbMATHCrossRefGoogle Scholar
  19. 19.
    Ederington, L.H., Guan, W.: Higher order greeks. J. Deriv. 14(3), 7–34 (2007)CrossRefGoogle Scholar
  20. 20.
    Kariwala, V.: Automatic differentiation-based quadrature method of moments for solving population balance equations. AIChE J. 58(3), 842–854 (2012)CrossRefGoogle Scholar
  21. 21.
    Corliss, G.F.F., Griewank, A., Henneberger, P.: High-order stiff ODE solvers via automatic differentiation and rational prediction. In: Vulkov, L., Waśniewski, J., Yalamov, P. (eds.) Lecture notes in computer science, pp. 114–125. Springer, Berlin, Heidelberg (1997)Google Scholar
  22. 22.
    Guckenheimer, J., Meloon, B.: Computing periodic orbits and their bifurcations with automatic differentiation. SIAM J. Sci. Comput. 22(3), 951–985 (2000)zbMATHMathSciNetCrossRefGoogle Scholar
  23. 23.
    Griewank, A., Walther, A., Utke, J.: Evaluating higher derivative tensors by forward propagation of univariate Taylor series. Math. Comput. 69(231), 1117–1130 (2000)zbMATHMathSciNetCrossRefGoogle Scholar
  24. 24.
    Naumann, U.: The art of differentiating computer programs: An introduction to algorithmic differentiation. Number 24 in Software, Environments, and Tools. SIAM, Philadelphia (2012)Google Scholar
  25. 25.
    Neidinger, R.D.: An efficient method for the numerical evaluation of partial derivatives of arbitrary order. ACM Trans. Math. Softw. 18(2), 159–173 (1992)zbMATHMathSciNetCrossRefGoogle Scholar
  26. 26.
    Fraenkel, L.E.: Formulae for high derivatives of composite functions. Math. Proc. Camb. Philos. Soc. 83(02), 159 (1978)zbMATHMathSciNetCrossRefGoogle Scholar
  27. 27.
    Griewank, A., Juedes, D., Utke, J.: ADOL-C, a package for the automatic differentiation of algorithms written in C/C++. ACM Trans. Math. Softw. 22(2), 131–167 (1996)zbMATHCrossRefGoogle Scholar
  28. 28.
    Bongartz, I., Conn, A.R., Gould, N., Toint, P.L.: CUTE: constrained and unconstrained testing environment. ACM Trans. Math. Softw. 21(1), 123–160 (1995)Google Scholar
  29. 29.
    Hock, W., Schittkowski, K.: Test examples for nonlinear programming codes. J. Optim. Theory Appl. 30(1), 127–129 (1980)zbMATHCrossRefGoogle Scholar
  30. 30.
    Luksan, L., Vlcek, J.: Test problems for unconstrained optimization. Technical Report 897, Academy of Sciences of the Czech Republic (2003)Google Scholar
  31. 31.
    Abate, J., Bischof, C., Roh, L., Carle, A.: Algorithms and design for a second-order automatic differentiation module. In: Proceedings of the 1997 International Symposium on Symbolic and Agebraic Computation (ACM), pp. 149–155, New York (1997)Google Scholar
  32. 32.
    Walther, A., Griewank, A.: Advantages of binomial checkpointing for memory-reduced adjoint calculations. In: Feistauer, M., Dolejší, V., Knobloch, P., Najzar, K. (eds.) Numerical Mathematics and Advanced Applications, pp. 834–843. Springer, Berlin, Heidelberg (2004). ISBN: 978-3-642-62288-5. doi: 10.1007/978-3-642-18775-9_82
  33. 33.
    Sternberg, J., Griewank, A.: Reduction of storage requirement by checkpointing for time-dependent optimal control problems in ODEs. In: Norris, B., Bücker, M., Corliss, G., Hovland, P., Naumann, U. (eds.) Automatic Differentiation: Applications, Theory, and Implementations, 0, pp. 99–110. Springer, 1 edition (2006)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg and Mathematical Optimization Society 2014

Authors and Affiliations

  1. 1.Maxwell Institute for Mathematical Sciences, School of MathematicsUniversity of EdinburghEdinburghUK
  2. 2.School of Mathematics, Statistics and Applied MathematicsNational University of Ireland GalwayGalwayIreland

Personalised recommendations