Computational Optimization and Applications

, Volume 10, Issue 3, pp 283–320 | Cite as

The Adjoint Newton Algorithm for Large-Scale Unconstrained Optimization in Meteorology Applications

  • Zhi Wang
  • K. Droegemeier
  • L. White


A new algorithm is presented for carrying out large-scale unconstrained optimization required in variational data assimilation using the Newton method. The algorithm is referred to as the adjoint Newton algorithm. The adjoint Newton algorithm is based on the first- and second-order adjoint techniques allowing us to obtain the Newton line search direction by integrating a tangent linear equations model backwards in time (starting from a final condition with negative time steps). The error present in approximating the Hessian (the matrix of second-order derivatives) of the cost function with respect to the control variables in the quasi-Newton type algorithm is thus completely eliminated, while the storage problem related to the Hessian no longer exists since the explicit Hessian is not required in this algorithm. The adjoint Newton algorithm is applied to three one-dimensional models and to a two-dimensional limited-area shallow water equations model with both model generated and First Global Geophysical Experiment data. We compare the performance of the adjoint Newton algorithm with that of truncated Newton, adjoint truncated Newton, and LBFGS methods. Our numerical tests indicate that the adjoint Newton algorithm is very efficient and could find the minima within three or four iterations for problems tested here. In the case of the two-dimensional shallow water equations model, the adjoint Newton algorithm improves upon the efficiencies of the truncated Newton and LBFGS methods by a factor of at least 14 in terms of the CPU time required to satisfy the same convergence criterion.

The Newton, truncated Newton and LBFGS methods are general purpose unconstrained minimization methods. The adjoint Newton algorithm is only useful for optimal control problems where the model equations serve as strong constraints and their corresponding tangent linear model may be integrated backwards in time. When the backwards integration of the tangent linear model is ill-posed in the sense of Hadamard, the adjoint Newton algorithm may not work. Thus, the adjoint Newton algorithm must be used with some caution. A possible solution to avoid the current weakness of the adjoint Newton algorithm is proposed.

adjoint Newton algorithm variational data assimilation LBFGS truncated Newton algorithm large-scale unconstrained minimization 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    A. Bennett, Inverse Problem in Physical Oceanography, Cambridge University Press, 1992, pp. 346.Google Scholar
  2. 2.
    M.S. Berger, Nonlinearity and Functional Analysis, Academic Press: New York, 1977, pp. 417.Google Scholar
  3. 3.
    S.R. Caradus, Operator Theory of the Pseudo-Inverse, A Queen’s Papers in Pure and Applied Mathematics, No. 38, Queen’s University, Kingston, Ontario, Canada, pp. 67, 1974.Google Scholar
  4. 4.
    G.F. Carey and J.T. Oden, Finite Elements Computational Aspects, Prentice-Hall Press, 1984, vol. 3, pp. 350.Google Scholar
  5. 5.
    W.C. Davidon, "Variable metric method for minimization," A.E.C. Research and Development Report, ANL-5990 (Rev.).Google Scholar
  6. 6.
    R.S. Dembo, S.C. Eisenstat, and T. Steihaug, "Inexact Newton methods," SIAM Journal of Numerical Analysis, vol. 19, pp. 400-408, 1982.Google Scholar
  7. 7.
    J. Dennis and Robert B. Schnabel, Numerical Methods for Unconstrained Optimization and Nonlinear Equations, Prentice-Hall: Englewood Cliffs, NJ, 1983, pp. 378.Google Scholar
  8. 8.
    J. Dieudonne, Foundations of Modern Analysis, Academic Press: New York, 1960, pp. 361.Google Scholar
  9. 9.
    John Fritz, Partial Differential Equations, 4th edition, Springer-Verlag: New York, 1986, pp. 247.Google Scholar
  10. 10.
    P.E. Gill and W. Murray, "Quasi-Newton methods for unconstrained optimization," J. Inst. Maths Applics, vol. 9, pp. 91-108, 1972.Google Scholar
  11. 11.
    P.E. Gill and W. Murray, Practical Optimization, Academic Press, 1981, pp. 401.Google Scholar
  12. 12.
    G.H. Golub and C.F. Van Loan, Matrix Computations, 2nd edition, The Johns Hopkins University Press: Baltimore and London, 1989, pp. 642.Google Scholar
  13. 13.
    A. Grammeltvedt, "A survey of finite-difference schemes for the primitive equations for a barotropic fluid," Mon. Wea. Rev., vol. 97, pp. 387-404, 1969.Google Scholar
  14. 14.
    R.N. Hoffmann, "SASS wind ambiguity removal by direct minimization," Mon. Wea. Rev., vol. 110, pp. 434-445, 1982.Google Scholar
  15. 15.
    R.N. Hoffmann, "SASS wind ambiguity removal by direct minimization Part II: Use of smoothness and dynamical constraints," Mon. Wea. Rev., vol. 112, pp. 1829-1852, 1984.Google Scholar
  16. 16.
    R.N. Hoffmann, "A four dimensional analysis exactly satisfying equations of motion," Mon. Wea. Rev., vol. 114, pp. 388-397, 1986.Google Scholar
  17. 17.
    J.F. Lacarra and O. Talagrand, Short-range evolution of small perturbations in a barotropic model," Tellus, vol. 40A, pp. 81-95, 1988.Google Scholar
  18. 18.
    F.X. Le Dimet and O. Talagrand, "Variational algorithms for analysis and assimilation of meteorological observations: Theoretical aspects," Tellus, vol. 38A, pp. 97-110, 1986.Google Scholar
  19. 19.
    J.L. Lions, Optimal control of systems governed by partial differential equations," Translated by S.K. Mitter, Springer-Verlag: Berlin-Heidelberg, 1971, pp. 404.Google Scholar
  20. 20.
    D.C. Liu and Jorge Nocedal, "On the limited memory BFGS method for large scale minimization," Mathematical Programming, vol. 45, pp. 503-528, 1989.Google Scholar
  21. 21.
    David G. Luenberger, Linear and Nonlinear Programming, 2nd edition, Addison-Wesley: Reading, MA, 1984, pp. 491.Google Scholar
  22. 22.
    S.G. Nash, "Truncated-Newton methods for large-scale function minimization,” in Applications of Nonlinear Programming to Optimization and Control, H.E. Rauch (Ed.), Pergamon Press: Oxford, 1984, pp. 91-100.Google Scholar
  23. 23.
    S.G. Nash, "Solving nonlinear programming problems using truncated Newton techniques," Numerical Optimization, P.T. Boggs, R.H. Byrd, and R.B. Schnabel (Eds.), SIAM: Philadelphia, 1984, pp. 119-136.Google Scholar
  24. 24.
    S.G. Nash, "Preconditioning of truncated-Newton methods," SIAM J. Sci. Stat. Comput., vol. 6, no.3, pp. 599-616, 1985.Google Scholar
  25. 25.
    S.G. Nash and Jorge Nocedal, "A numerical study of the limited memory BFGS method and the truncated-Newton method for large-scale optimization," Tech. Rep. NAM, 02, Department of Electrical Engineering and Computer Science, Northwestern University, 1989, p. 19.Google Scholar
  26. 26.
    I.M. Navon and D.M. Legler, "Conjugate gradient methods for large-scale minimization in meteorology," Mon. Wea. Rev., vol. 115, pp. 1479-1502, 1987.Google Scholar
  27. 27.
    I.M. Navon, X.L. Zou, J. Derber, and J. Sela, "Variational data assimilation with an adiabatic version of the NMC spectral model," Mon. Wea. Rev., vol. 122, pp. 1433-1446, 1992.Google Scholar
  28. 28.
    J. Nocedal, "Updating quasi-Newton matrices with limited storage," Mathematics of Computation, vol. 35, pp. 773-782, 1980.Google Scholar
  29. 29.
    D.P. O’Leary, "A discrete Newton algorithm for minimizing a function of many variables," Math. Prog., vol. 23, pp. 20-23, 1983.Google Scholar
  30. 30.
    Z. Pu, E. Kalnay, and J. Sela, "Sensitivity of forecast error to initial conditions with a quasi-inverse linear method," Mon. Wea. Rev., 1996, accepted for publication.Google Scholar
  31. 31.
    C.R. Rao and S.K. Mitra, Generalized Inverse of Matrices and its Applications to Statistics, John Wiley & and Sons, 1971, p. 240.Google Scholar
  32. 32.
    Fadil Santosa and William W. Symes, "Computation of the Hessian for least-squares solutions of inverse problems of reflection seismology," Inverse Problems, vol. 4, pp. 211-233, 1988.Google Scholar
  33. 33.
    Fadil Santosa and William W. Symes, "An analysis of least squares velocity inversion," Society of Exploration Geophysicists, Geophysical Monograph #4, Tulsa, 1989.Google Scholar
  34. 34.
    T. Schlick and A. Fogelson, "TNPACK-Atruncated Newton minimization package for large-scale problems: I. Algorithm and usage," ACMTOMS, vol. 18, no.1, pp. 46-70, 1992a.Google Scholar
  35. 35.
    T. Schlick and A. Fogelson, "TNPACK-Atruncated Newton minimization package for large-scale problems: II. Implementation examples," ACMTOMS, vol. 18, no.1, pp. 71-111, 1992b.Google Scholar
  36. 36.
    D.F. Shanno and K.H. Phua, "Remark on algorithm 500-A variable method subroutine for unconstrained nonlinear minimization," ACM Trans. on Mathematical Software, vol. 6, pp. 618-622, 1980.Google Scholar
  37. 37.
    J. Stoer and R. Bulirsch, Introduction to Numerical Analysis, 2nd edition, Springer-Verlag: New York, 1976, pp. 659.Google Scholar
  38. 38.
    William W. Symes, "A differential semblance algorithm for the inverse problem of reflection seismology," Computers Math. Applic., vol. 22, nos.4/5, pp. 147-178, 1991.Google Scholar
  39. 39.
    O. Talagrand and P. Courtier, "Variational assimilation of meteorological observations with the adjoint vorticity equation-Part 1. Theory," Q. J. R. Meteorol. Soc., vol. 113, pp. 1311-1328, 1987.Google Scholar
  40. 40.
    Zhi Wang, "Variational data assimilation with 2D shallow water equations and 3DFSU global spectral models," Tech. Rep. FSU-SCRI-93T-149, Florida State University, Tallahassee, Florida, 1993, p. 235.Google Scholar
  41. 41.
    Zhi Wang, I.M. Navon, F.X. Le Dimet, and X. Zou, "The second order adjoint analysis: Theory and application," Meteorol. and Atmos. Phy., vol. 50, pp. 3-20, 1992.Google Scholar
  42. 42.
    Zhi Wang, I.M. Navon, X. Zou, and F.X. Le Dimet, "A truncated Newton optimization algorithm in meteorology applications with analytic Hessian/vector products," Computational Optimization and Applications, vol. 4, no.3, pp. 241-262, 1995.Google Scholar
  43. 43.
    Zhi Wang, Kelvin K. Droegemeier, and L. White, "Application of a New Adjoint Newton Algorithm to the 3-D ARPS Storm Scale Model Using Simulated Data," Accepted for publication by Mon. Wea. Rev., 1997.Google Scholar
  44. 44.
    X.L. Zou, I.M. Navon, and F.X. Le Dimet, "Incomplete observations and control of gravity waves in variational data assimilation," Tellus, vol. 44A, pp. 273-296, 1992.Google Scholar

Copyright information

© Kluwer Academic Publishers 1998

Authors and Affiliations

  • Zhi Wang
    • 1
  • K. Droegemeier
    • 1
  • L. White
    • 2
  1. 1.Center for Analysis and Prediction of Storms and School of MeterologyUniversity of OklahomaNorman
  2. 2.Department of MathematicsUniversity of OklahomaUSA

Personalised recommendations