Optimization and Engineering

, Volume 15, Issue 3, pp 575–608 | Cite as

Inexact Hessian-vector products in reduced-space differential-equation constrained optimization

  • Jason E. HickenEmail author


Reduced-space inexact-Newton–Krylov (RSNK) algorithms provide a modular and scalable framework for solving differential-equation-constrained optimization problems; thus, this class of algorithms provides an attractive compromise between poorly scaling “black-box” methods and more intrusive, full-space optimization algorithms. One of the challenges of implementing RSNK methods is the efficient solution of the linear subproblems, which involve the reduced Hessian. This paper explores inexact Hessian-vector products to improve the efficiency of these subproblems. The reduced-Hessian-vector products, which are required by the Krylov solver in RSNK, can be determined by solving two second-order adjoint equations. To reduce computational cost, we consider the approximate (i.e. inexact) solution of the second-order adjoints. We present bounds for the second-order adjoint tolerances that ensure the Hessian-vector product remains sufficiently accurate for inexact-Krylov methods. The bounds involve the 2-norm of the state and first-order adjoint sensitivities, and we provide an inexpensive Lanczos algorithm to estimate these matrix norms. Using numerical experiments, we investigate the proposed RSNK algorithm and compare it to other optimization algorithms.


PDE-constrained optimization Inexact-Krylov Second-order adjoint 



The author thanks the anonymous referees for their careful reading of the manuscript and helpful remarks. The author gratefully acknowledges the financial support of Rensselaer Polytechnic Institute.


  1. Akçelik V, Biros G, Ghattas O (2002) Parallel multiscale Gauss–Newton–Krylov methods for inverse wave propagation. SC Conference, p. 41 doi: 10.1109/SC.2002.10002
  2. Akçelik V, Biros G, Ghattas O, Hill J, Keyes D, van Bloemen Waanders B (2006) Parallel algorithms for PDE-constrained optimization. In: Heroux MA, Raghavan P, Simon HD (eds) Parallel processing for scientific computing, chap. 16. Society for Industrial and Applied Mathematics, pp 291–322. doi: 10.1137/1.9780898718133.ch16
  3. Anderson JD (1991) Fundamentals of aerodynamics, 2nd edn. McGraw-Hill Inc., New YorkGoogle Scholar
  4. Arnold DN, Brezzi F, Cockburn B, Marini LD (2002) Unified analysis of discontinuous Galerkin methods for elliptic problems. SIAM J Numer Anal 39(5):1749–1779. doi: 10.1137/s0036142901384162 MathSciNetCrossRefzbMATHGoogle Scholar
  5. Biros G, Ghattas O (2005) Parallel Lagrange–Newton–Krylov–Schur methods for PDE-constrained optimization. Part I: The Krylov–Schur solver. SIAM J Sci Comput 27:687–713MathSciNetCrossRefzbMATHGoogle Scholar
  6. Biros G, Ghattas O (2005) Parallel Lagrange–Newton–Krylov–Schur methods for PDE-constrained optimization. Part II: The Lagrange–Newton solver and its application to optimal control of steady viscous flows. SIAM J Sci Comput 27:714–739MathSciNetCrossRefzbMATHGoogle Scholar
  7. Borzì A, Schulz V (2011) Computational optimization of systems governed by partial differential equations. Society for Industrial and Applied Mathematics, pp. 47–48 doi: 10.1137/1.9781611972054
  8. Borzì A, Schulz V (2011) Computational optimization of systems governed by partial differential equations. Society for Industrial and Applied Mathematics, pp 62–66. doi: 10.1137/1.9781611972054
  9. Bouras A, Frayssé V (2005) Inexact matrix-vector products in Krylov methods for solving linear systems: a relaxation strategy. SIAM J Matrix Anal Appl 26(3):660–678. doi: 10.1137/s0895479801384743 MathSciNetCrossRefzbMATHGoogle Scholar
  10. Braack M (2009) Optimal control in fluid mechanics by finite elements with symmetric stabilization. SIAM J Control Optim 48(2):672–687. doi: 10.1137/060653494 MathSciNetCrossRefzbMATHGoogle Scholar
  11. Byrd RH, Curtis FE, Nocedal J (2008) An inexact SQP method for equality constrained optimization. SIAM J Optim 19(1):351–369. doi: 10.1137/060674004 MathSciNetCrossRefzbMATHGoogle Scholar
  12. Byrd RH, Curtis FE, Nocedal J (2010) An inexact Newton method for nonconvex equality constrained optimization. Math Program 122(2):273–299. doi: 10.1007/s10107-008-0248-3 MathSciNetCrossRefzbMATHGoogle Scholar
  13. Carpenter MH, Gottlieb D, Abarbanel S (1994) Time-stable boundary conditions for finite-difference schemes solving hyperbolic systems: methodology and application to high-order compact schemes. J Comput Phys 111(2):220–236. doi: 10.1006/jcph.1994.1057 MathSciNetCrossRefzbMATHGoogle Scholar
  14. Conn AR, Gould NIM, Toint PL (2000) Trust region methods. Society for Industrial and Applied Mathematics doi: 10.1137/1.9780898719857
  15. Dembo RS, Eisenstat SC, Steihaug T (1982) Inexact Newton methods. SIAM J Numer Anal 19(2):400–408. doi: 10.1137/0719025 MathSciNetCrossRefzbMATHGoogle Scholar
  16. Du X, Sarkis M, Schaerer CE, Szyld DB (2013) Inexact and truncated parareal-in-time Krylov subspace methods for parabolic optimal control problems. Electron Trans Numer Anal 40:36–57MathSciNetzbMATHGoogle Scholar
  17. Farin GE (1997) Curves and surfaces for computed aided geometric design: a practical guide, 4th edn. Academic Press, Inc, LondonGoogle Scholar
  18. Feng D, Pulliam TH (1995) An all-at-once reduced hessian SQP scheme for aerodynamic design optimization. Tech. Rep. RIACS-TR-95.19, NASA Ames Research Center, Moffett Field, CaliforniaGoogle Scholar
  19. Fletcher R (2000) Practical methods of optimization, 2nd edn. A Wiley-Interscience Publication, Wiley, New YorkCrossRefGoogle Scholar
  20. Frank PD, Shubin GR (1992) A comparison of optimization-based approaches for a model computational aerodynamics design problem. J Comput Phys 98(1):74–89CrossRefzbMATHGoogle Scholar
  21. Funaro D, Gottlieb D (1988) A new method of imposing boundary conditions in pseudospectral approximations of hyperbolic equations. Math Comput 51(184):599–613MathSciNetCrossRefzbMATHGoogle Scholar
  22. Gatsis J, Zingg DW (2003) A fully-coupled Newton–Krylov algorithm for aerodynamic optimization. In: 16th AIAA Computational Fluid Dynamics Conference, AIAA-2003-3956. Orlando.Google Scholar
  23. Ghate D, Giles M (2007) Efficient Hessian calculation using automatic differentiation. In: 25th AIAA Applied Aerodynamics Conference, AIAA-2007-4059. Miami. doi: 10.2514/6.2007-4059
  24. Golub GH, Ye Q (1999) Inexact preconditioned conjugate gradient method with inner–outer iteration. SIAM J Sci Comput 21(4):1305–1320. doi: 10.1137/s1064827597323415 MathSciNetCrossRefGoogle Scholar
  25. Griewank A, Walther A (2008) Evaluating derivatives: principles and techniques of algorithmic differentiation. Society for Industrial and Applied Mathematics
  26. Griewank A, Walther A (2008) Evaluating derivatives: principles and techniques of algorithmic differentiation, chap. 5. Society for Industrial and Applied Mathematics, pp. 91–105.
  27. Gunzburger MD (2003) Perspectives in flow control and optimization. Society for Industrial and Applied Mathematics
  28. Haber E, Ascher UM (2001) Preconditioned all-at-once methods for large, sparse parameter estimation problems. Inverse Probl 17(6):1847–1864. doi: 10.1088/0266-5611/17/6/319 MathSciNetCrossRefzbMATHGoogle Scholar
  29. Hartmann R (2007) Adjoint consistency analysis of discontinuous Galerkin discretizations. SIAM J Numer Anal 45(6):2671–2696. doi: 10.1137/060665117 MathSciNetCrossRefzbMATHGoogle Scholar
  30. Heinkenschloss M, Vicente LN (1999) An interface optimization and application for the numerical solution of optimal control problems. ACM Trans Math Softw 25(2):157–190. doi: 10.1145/317275.317278 MathSciNetCrossRefzbMATHGoogle Scholar
  31. Hicken JE, Alonso JJ (2013) Comparison of reduced- and full-space algorithms for PDE-constrained optimization. In: 51st AIAA aerospace sciences meeting, AIAA-2013-1043. Grapevine. doi: 10.2514/6.2013-1043
  32. Hicken JE, Zingg DW (2011) Superconvergent functional estimates from summation-by-parts finite-difference discretizations. SIAM J Sci Comput 33(2):893–922. doi: 10.1137/100790987 MathSciNetCrossRefzbMATHGoogle Scholar
  33. Hicken JE, Zingg DW (2011) The role of dual consistency in functional accuracy: error estimation and superconvergence. In: 20th AIAA computational fluid dynamics conference, AIAA-2011-3855. Honolulu.Google Scholar
  34. Hicken JE, Zingg DW (2012) Dual consistency and functional accuracy: a finite-difference perspective. J Comput Phys 256:161–182MathSciNetCrossRefGoogle Scholar
  35. Hinze M, Pinnau R, Ulbrich M, Ulbrich S (2009) Optimization with PDE constraints. Springer, New York, pp 64–65.
  36. Iollo A, Salas MD, Ta’asan S (1993) Shape optimization governed by the Euler equations using an adjoint method. Tech. Rep. ICASE 93–78, NASA Langley Research Center, Hampton.Google Scholar
  37. Keyes DE (1995) Aerodynamic applications of Newton–Krylov–Schwarz solvers. In: Proceedings of the 14th international conference on numerical methods in fluid dynamics. Springer, New York, pp 1–20Google Scholar
  38. Knoll DA, Keyes DE (2004) Jacobian-free Newton–Krylov methods: a survey of approaches and applications. J Comput Phys 193(2):357–397. doi: 10.1016/ MathSciNetCrossRefzbMATHGoogle Scholar
  39. Kreiss HO, Scherer G (1974) Finite element and finite difference methods for hyperbolic partial differential equations. In: de Boor C (ed) Mathematical aspects of finite elements in partial differential equations. Mathematics Research Center, the University of Wisconsin Academic Press, MadisonGoogle Scholar
  40. Lanczos C (1950) An iteration method for the solution of the eigenvalue problem of linear differential and integral operators. J Res Natl Bur Stand 45(4):255–282MathSciNetCrossRefGoogle Scholar
  41. Liu DC, Nocedal J (1989) On the limited memory BFGS method for large scale optimization. Math Program 45:503–528. doi: 10.1007/BF01589116 MathSciNetCrossRefzbMATHGoogle Scholar
  42. Lu JC (2005) An a posteriori error control framework for adaptive precision optimization using discontinuous Galerkin finite element method. Ph.D. thesis, Massachusetts Institute of Technology, CambridgeGoogle Scholar
  43. Martins JRRA, Sturdza P, Alonso JJ (2003) The complex-step derivative approximation. ACM Trans Math Softw 29(3):245–262. doi: 10.1145/838250.838251 MathSciNetCrossRefzbMATHGoogle Scholar
  44. Mattsson K, Svärd M, Nordström J (2004) Stable and accurate artificial dissipation. J Sci Comput 21(1):57–79MathSciNetCrossRefzbMATHGoogle Scholar
  45. Melvin RG, Huffman WP, Young DP, Johnson FT, Hilmes CL (1999) Recent progress in aerodynamic design optimization. Int J Numer Methods Fluids 30(2):205–206. doi: 10.1002/(sici)1097-0363(19990530);2-u Google Scholar
  46. Morales JL, Nocedal J (2000) Automatic preconditioning by limited memory quasi-Newton updating. SIAM J Optim 10(4):1079–1096. doi: 10.1137/S1052623497327854 MathSciNetCrossRefzbMATHGoogle Scholar
  47. Nocedal J, Wright SJ (2006) Numerical optimization, 2nd edn. Springer, BerlinzbMATHGoogle Scholar
  48. Nocedal J, Wright SJ (2006) Numerical optimization, chap. 6, 2nd edn. Springer, BerlinGoogle Scholar
  49. Nocedal J, Wright SJ (2006) Numerical optimization, chap. 7, 2nd edn. Springer, BerlinGoogle Scholar
  50. Nocedal J, Wright SJ (2006) Numerical optimization, chap. 12, 2nd edn. Springer, BerlinGoogle Scholar
  51. Rumpfkeil MP, Mavriplis DJ (2010) Efficient Hessian calculations using automatic differentiation and the adjoint method with applications. AIAA J 48(10):2406–2417. doi: 10.2514/1.j050451 CrossRefGoogle Scholar
  52. Saad Y (1993) A flexible inner–outer preconditioned GMRES algorithm. SIAM J Sci Stat Comput 14(2):461–469MathSciNetCrossRefzbMATHGoogle Scholar
  53. Saad Y (2003) Iterative methods for sparse linear systems, chap. 6, 2nd edn. SIAM, PhiladelphiaCrossRefGoogle Scholar
  54. Saad Y (2011) Numerical methods for Large eigenvalue problems. Society for Industrial and Applied Mathematics. doi: 10.1137/1.9781611970739
  55. Saad Y, Schultz MH (1986) GMRES: a generalized minimal residual algorithm for solving nonsymmetric linear systems. SIAM J Sci Stat Comput 7(3):856–869MathSciNetCrossRefzbMATHGoogle Scholar
  56. Simoncini V, Szyld DB (2003) Theory of inexact Krylov subspace methods and applications to scientific computing. SIAM J Sci Comput 25(2):454–477. doi: 10.1137/s1064827502406415 MathSciNetCrossRefzbMATHGoogle Scholar
  57. Squire W, Trapp G (1998) Using complex variables to estimate derivatives of real functions. SIAM Rev 40(1):110–112MathSciNetCrossRefzbMATHGoogle Scholar
  58. Strand B (1994) Summation by parts for finite difference approximations for d/dx. J Comput Phys 110(1):47–67. doi: 10.1006/jcph.1994.1005 MathSciNetCrossRefzbMATHGoogle Scholar
  59. Ta’asan S, Kuruvila G, Salas MD (1992) Aerodynamic design and optimization in one shot. In: The 30th AIAA aerospace sciences meeting and exhibit, AIAA-1992-25Google Scholar
  60. Waltz RA, Nocedal J (2003), KNITRO user’s manual. Northwestern University, Evanston, Technical, Report OTC-2003/5Google Scholar
  61. Wang Z, Navon IM, Dimet FX, Zou X (1992) The second order adjoint analysis: theory and applications. Meteorol Atmos Phys 50:3–20. doi: 10.1007/BF01025501 CrossRefGoogle Scholar
  62. van den Eshof J, Sleijpen GLG (2004) Inexact Krylov subspace methods for linear systems. SIAM J Matrix Anal Appl 26(1):125–153. doi: 10.1137/s0895479802403459 MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  1. 1.Department of Mechanical, Aerospace, and Nuclear EngineeringRensselaer Polytechnic InstituteTroyUSA

Personalised recommendations