Abstract
The idea of hierarchical gradient methods for optimization is considered. It is shown that the proposed approach provides powerful means to cope with some global convergence problems characteristic of the classical gradient methods. Concerning global convergence problems, four topics are addressed: The “detour” effect, the problem of multiscale models, the problem of highly ill-conditioned objective functions, and the problem of local-minima traps related to ambiguous regions of attractions. The great potential of hierarchical gradient algorithms is revealed through a hierarchical Gauss-Newton algorithm for unconstrained nonlinear least-squares problems. The algorithm, while maintaining a superlinear convergence rate like the common conjugate gradient or quasi-Newton methods, requires the evaluation of partial derivatives with respect to only one variable on each iteration. This property enables economized consumption of CPU time in case the computer codes for the derivatives are intensive CPU consumers, e.g., when the gradient evaluations of ODE or PDE models are produced by numerical differentiation. The hierarchical Gauss-Newton algorithm is extended to handle interval constraints on the variables and its effectiveness demonstrated by computational results.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Bauer, F. L. (1965). Elimination with weighted row combinations for solving linear equations and least-squares problems,Numer. Math. 7, 338–352.
Brandt, A., Ron, D., and Amit, D. J. (1986). Multilevel approaches to discrete-state and stochastic problems, in Hackbush, W., and Trottenburg, V. (eds.),Multigrid Methods II, Springer-Verlag, Cologne, pp. 66–99.
Cornwell, L. W., Pegis, R. J., Rigler, A. K., and Vogl, T. P. (1973). Grey's method of nonlinear optimization,J. Opt. Soc. Am. 65(3).
Filip, F. G., Donciulescu, D. A., Muratcea, M., et al. (1983). Industrial applications of hierarchical optimization methods. In Straszak, A. (ed.),Large Scale Systems Theory and Applications, 1983 Proc. of the IFAC/IFORS symposium, Warsaw, Poland.
Fletcher, R. (1972). Minimizing general functions subject to linear constraints, in Lootsma, F. A. (ed.),Numerical Methods for Nonlinear Optimization, Academic Press, New York.
Golub, G. H. (1969). Matrix decompositions and statistical calculations. In Milton and Nelder (eds.),Statistical Computation, Academic Press, New York.
Grey, D. S. (1963). Aberration theories for semiautomatic lens design by electronic computers,J. Opt. Soc. Am. 53(6).
Grey, D. S. (1966). Boundary conditions in optimization problems, in Lavi and Vogl (eds.),Recent Advances in Optimization Techniques, J. Wiley, New York.
Papageorgio, M., and Schmidt, G. (1980). On the hierarchical solution of nonlinear optimal control problems,Large Scale Syst. 1, 265–271.
Roberts, P. D., Ellis, J. E., Li, C. W., etal. (1983). Algorithms for hierarchical steady state optimization of industrial processes, in Straszak, A. (ed.),Large Scale Systems Theory and Applications, 1983 Proc. of the IFAC/IFORS symposium, Warsaw, Poland.
Sultan, M. A., Hassan, M. F., and Calvet, J. L. (1988). Parameter estimation in large-scale systems using sequential decomposition,Int. J. Syst. Sci. 19(3), 487–96.
Tarvainan, K. (1983). Nonfeasible hierarchical multicriteria methods, In Straszak, A. (ed.),Large Scale Systems Theory and Applications, 1983 Proc. of the IFAC/IFORS symposium, Warsaw, Poland.
Waltz, F. M. (1967). An engineering approach: Hierarchical optimization criteria,IEEE Trans. Automatic Control AC-12, 179–180.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Berger, M.F. Hierarchical gradient methods for nonlinear LSQ problems. J Sci Comput 7, 197–228 (1992). https://doi.org/10.1007/BF01061328
Received:
Issue Date:
DOI: https://doi.org/10.1007/BF01061328