Mathematical Programming Computation

, Volume 8, Issue 4, pp 393–433 | Cite as

Capitalizing on live variables: new algorithms for efficient Hessian computation via automatic differentiation

Full Length Paper


We revisit an algorithm [called Edge Pushing (EP)] for computing Hessians using Automatic Differentiation (AD) recently proposed by Gower and Mello (Optim Methods Softw 27(2): 233–249, 2012). Here we give a new, simpler derivation for the EP algorithm based on the notion of live variables from data-flow analysis in compiler theory and redesign the algorithm with close attention to general applicability and performance. We call this algorithm Livarh and develop an extension of Livarh that incorporates preaccumulation to further reduce execution time—the resulting algorithm is called Livarhacc. We engineer robust implementations for both algorithms Livarh and Livarhacc within ADOL-C, a widely-used operator overloading based AD software tool. Rigorous complexity analyses for the algorithms are provided, and the performance of the algorithms is evaluated using a mesh optimization application and several kinds of synthetic functions as testbeds. The results show that the new algorithms outperform state-of-the-art sparse methods (based on sparsity pattern detection, coloring, compressed matrix evaluation, and recovery) in some cases by orders of magnitude. We have made our implementation available online as open-source software and it will be included in a future release of ADOL-C.


Algorithmic differentiation Hessian computation Reverse mode AD Edge pushing ADOL-C Data-flow analysis 

Mathematics Subject Classification

90C30 (Nonlinear programming) 49M37 (Methods of nonlinear programming type) 65K05 (Mathematical programming methods) 


  1. 1.
    Griewank, A., Walther, A.: Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation, 2nd edn. Number 105. SIAM, Philadelphia (2008)Google Scholar
  2. 2.
    Naumann, U.: The Art of Differentiating Computer Programs: An Introduction to Algorithmic Differentiation. SIAM (2012)Google Scholar
  3. 3.
    Jackson, R.H.F., McCormick, G.P.: The polyadic structure of factorable function tensors with applications to higher-order minimization techniques. J. Optim. Theory Appl. 51, 63–94 (1986)MathSciNetCrossRefMATHGoogle Scholar
  4. 4.
    Christianson, B.: Automatic Hessians by reverse accumulation. IMA J. Numer. Anal 12(2), 135–150 (1992)MathSciNetCrossRefMATHGoogle Scholar
  5. 5.
    Dixon, L.C.W.: Use of automatic differentiation for calculating Hessians and Newton steps. In: Griewank, A., Corliss, G.F. (eds.) Automatic Differentiation of Algorithms: Theory, Implementation and Application, pp. 114–125. SIAM, Philadelphia (1991)Google Scholar
  6. 6.
    Bhowmick, S., Hovland, P.D.: A polynomial-time algorithm for detecting directed axial symmetry in Hessian computational graphs. In: Bischof, C.H., Bucker, H.M., Hovland, P.D., Naumann, U., Utke, J. (eds.) Advances in Automatic Differentiation, pp. 91–102. Springer, Berlin (2008)CrossRefGoogle Scholar
  7. 7.
    Coleman, T., Moré, J.: Estimation of sparse Hessian matrices and graph coloring problems. Math. Program. 28, 243–270 (1984)MathSciNetCrossRefMATHGoogle Scholar
  8. 8.
    Coleman, T., Cai, J.: The cyclic coloring problem and estimation of sparse Hessian matrices. SIAM J. Alg. Disc. Meth. 7(2), 221–235 (1986)MathSciNetCrossRefMATHGoogle Scholar
  9. 9.
    Gebremedhin, A.H., Tarafdar, A., Manne, F., Pothen, A.: New acyclic and star coloring algorithms with applications to Hessian computation. SIAM J. Sci. Comput. 29(3), 1042–1072 (2007)MathSciNetCrossRefMATHGoogle Scholar
  10. 10.
    Gebremedhin, A.H., Tarafdar, A., Pothen, A., Walther, A.: Efficient computation of sparse Hessians using coloring and automatic differentiation. INFORMS J. Comput. 1(2), 209–223 (2009)MathSciNetCrossRefMATHGoogle Scholar
  11. 11.
    Gower, R.M., Mello, M.P.: A new framework for Hessian automatic differentiation. Optim. Methods Softw. 27(2), 233–249 (2012)MathSciNetCrossRefGoogle Scholar
  12. 12.
    Gower, R.M., Mello, M.P.: Computing the sparsity pattern of Hessians using automatic differentiation. ACM Trans. Math. Softw. 40(2) (2014) (Article No. 10)Google Scholar
  13. 13.
    Gower, R.M., Gower, A.L.: Higher-order reverse automatic differentiation with emphasis on the third-order. Math. Program. 1–23 (2014) (ISSN: 0025–5610)Google Scholar
  14. 14.
    Giewank, A., Juedes, D., Utke, J.: ADOL-C: a package for the automatic differentiation of algorithms written in C/C++. ACM Trans. Math. Softw. 22, 131–167 (1996)CrossRefMATHGoogle Scholar
  15. 15.
    Walther, A., Griewank, A.: Getting started with ADOL-C. In: Combinatorial Scientific Computing, Chapman-Hall CRC Computational Science, Chapter 7, pp. 181–202 (2012)Google Scholar
  16. 16.
    Hascoët, L., Naumann, U., Pascual, V.: “To Be Recorded” analysis in reverse-mode automatic differentiation. Future Gener. Comput. Syst. 21, 1401–1417 (2005)CrossRefGoogle Scholar
  17. 17.
    Hascoët, L., Araya-Polo, M.: The adjoint data-flow analyses: formalization, properties, and applications. In: Bücker, M., Corliss, G., Hovland, P., Naumann, U., Norris, B. (eds.) Automatic Differentiation: Applications, Theory and Tools, Lecture Notes in Comput. Sci. Engr. 50, pp. 135–146. Springer, Berlin (2005)Google Scholar
  18. 18.
    Griewank, A.: Sequential Evaluation of Adjoints and Higher Derivative Vectors by Overloading and Reverse Accumulation. Research Report Konrad-Zuse-Zentrum for Informationstechnik Berlin, No 0 (1991)Google Scholar
  19. 19.
    Utke, J.: Flattening of Basic Blocks for Preaccumulation Automatic Differentiation: Applications, Theory, and Implementations, pp. 121–133. Springer, Berlin (2006)CrossRefMATHGoogle Scholar
  20. 20.
    Luksan, L., Matonoha, C., Vlcek, J.: Sparse Test Problems for Unconstrained Optimization. Tech. Rep. V-1064, ICS AS CR (2010)Google Scholar
  21. 21.
    Munson, T.S., Hovland, P.D.: The FeasNewt benchmark. In: Proceedings of the IEEE International Workload Characterization Symposium, pp. 150–154 (2005)Google Scholar
  22. 22.
    Gebremedhin, A.H., Nguyen, D., Patwary, M., Pothen, A.: ColPack: software for graph coloring and related problems in scientific computing. ACM Trans. Math. Sofw. 40(1), 1–31 (2013)MathSciNetCrossRefMATHGoogle Scholar
  23. 23.
    Walther, A.: Computing sparse Hessians with automatic differentiation. ACM Trans. Math. Softw. 34(1), 1–15 (2008)MathSciNetCrossRefMATHGoogle Scholar
  24. 24.
    Walther, A.: On the efficient computation of sparse patterns for Hessians. In: Forth, S. et al. (eds.) Recent Advances in Algorithmic Differentiation, Lecture Notes in Computational Science and Engineering 87, pp. 139–149 (2012)Google Scholar
  25. 25.
    Nicholas, N., Julian, S.: Valgrind: A framework for heavyweight dynamic binary instrumentation. In: Proceedings of ACM SIGPLAN 2007 Conference on Programming Language Design and Implementation (PLDI 2007), San Diego, California, USA (2007)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg and The Mathematical Programming Society 2016

Authors and Affiliations

  1. 1.Department of Computer SciencePurdue UniversityWest LafayetteUSA
  2. 2.School of Electrical Engineering and Computer ScienceWashington State UniversityPullmanUSA
  3. 3.Department of Computer SciencePurdue UniversityWest LafayetteUSA

Personalised recommendations