Abstract
In solving the nonlinear least-squares problem of minimizing ||f(x)|| 22 , difficulties arise with standard approaches, such as the Levenberg-Marquardt approach, when the Jacobian of f is rank-deficient or very ill-conditioned at the solution. To handle this difficulty, we study a special class of least-squares problems that are uniformly rank-deficient, i.e., the Jacobian of f has the same deficient rank in the neighborhood of a solution. For such problems, the solution is not locally unique. We present two solution tecniques: (i) finding a minimum-norm solution to the basic problem, (ii) using a Tikhonov regularization. Optimality conditions and algorithms are given for both of these strategies. Asymptotical convergence properties of the algorithms are derived and confirmed by numerical experiments. Extensions of the presented ideas make it possible to solve more general nonlinear least-squares problems in which the Jacobian of f at the solution is rank-deficient or ill-conditioned.
Similar content being viewed by others
References
J. Eriksson (1996) Optimization and Regularization of Nonlinear Least-Squares Problems, PhD Thesis Umeå University Umeå, Sweden
E.L. Allgower K. Georg (1991) Numerical Continuation Methods Springer Verlag Berlin, Germany
H. Walker (1990) Newton-Like Methods for Underdetermined Systems, Lectures in Applied Mathematics: Computational Solution of Nonlinear Systems of Equations E. Allgower K. Georg (Eds) American Mathematical Society. Providence Rhode Island 679–699
P. Lemmerling (1999) Structured Total Least Squares: Analysis, Algorithms, and Applications, PhD Thesis Katholieke Universiteit Lueven, Belgium
A. Pruessner D.P. O’Leary (2003) ArticleTitleBlind Deconvolution Using a Regularized Structured Total Least-Norm Algorithm SIAM Journal on Matrix Analysis and Applications 24 1018–1037 Occurrence Handle10.1137/S0895479801395446
Eriksson, J., and Gulliksson, M., Local Results for the Gauss-Newton Method on Constrained Exactly Rank-Deficient Nonlinear Least Squares, Technical Report UMINF 97.12, Department of Computing Science, Umeå University, Umeå, Sweden, 1997.
P.E. Gill W. Murray (1978) ArticleTitleAlgorithms for the Solution of the Nonlinear Least–Squares Problem SIAM Journal on Numerical Analysis 15 976–992 Occurrence Handle10.1137/0715063
P. Frank R.B. Schnabel (1984) ArticleTitleTensor Methods for Nonlinear Equations SIAM Journal on Numerical Analysis 21 815–843 Occurrence Handle10.1137/0721054
H. Engl M. Hanke A. Neubauer (1996) Regularization of Inverse Problems Kluwer Dordrecht, Netherlands
A.N. Tikhonov A. Goncharsky V.V. Stepanov A.G. Yagola (1995) Numerical Methods for the Solution of Ill-Posed Problems Kluwer Academic Publishers Dordrecht, Netherlands
L. Conlar (1993) Differential Manifolds: A First Course Birkhauser Advanced Texts Basel, Switzerland
Eriksson, J., Lindström, P., and Wedin, P. A., A New Regularization Method for Rank-Deficient Nonlinear Least Squares, Technical Report UMINF-95.01, Department of Computing Science, University of Umeå, Umeå, Sweden, 1995.
P.E. Gill W. Murray M.H. Wright (1982) Practical Optimization Academic Press London, United Kingdom
D.P. Bertsekas (1995) Nonlinear Programming Athena Scientific Belmont, Massachusetts
Moré, J. J., The Levenberg-Marquardt Algorithm: Implementation and Theory, Proceedings of the 1977 Dundee Conference on Numerical Analysis, Edited by G. A. Watson, Springer Verlag, Berlin, Germany, pp. 105–116, 1978.
Deuflhard, P., and Apostolescu, V., An Underrelaxed Gauss-Newton Method for Equality Constrained Nonlinear Least Squares, Proceedings of the IFIP Conference on Optimization Techniques, Edited by Balakrishnan and Thoma, Springer Verlag, Berlin, Germany, pp. 22–32, 1978.
L.C.W. Dixon D.J. Mills (1992) ArticleTitleNeural Network and Nonlinear Optimization, Part 1: The Representation of Continuous Functions Optimization Methods and Software 1 141–151
Lindström P., Wedin P.A., Methods and Software for Nonlinear Least-Squares Problems, Technical Report UMINF–133.87, Institution of Information Processing, University of Umeå, Umeå, Sweden, 1988.
Eriksson, J., and Wedin, P. A., Regularization Methods for Nonlinear Least Squares. Part 1: Exactly Rank-deficient Problems, Technical Report UMINF-96.03, Department of Computing Science, Umeå University, Umeå, Sweden, 1996.
M.E. Gulliksson P.A. Wedin (2000) ArticleTitleThe Use and Properties of Tikhonov Filter Matrices SIAM Journal on Matrix Analysis and Applications 22 276–281 Occurrence Handle10.1137/S0895479899355025
J.M. Ortega W.C. Rheinboldt (1970) Iterative Solution of Nonlinear Equations in Several Variables Academic Press New York, NY
H. Ramsin P.A. Wedin (1977) ArticleTitleA Comparison of Some Algorithms for the Nonlinear Least-Squares Problem, BIT 17 72–90
Author information
Authors and Affiliations
Additional information
The authors would like to thank Prof. Margaret H. Wright for greatly improving the readability of the manuscript.
Rights and permissions
About this article
Cite this article
Eriksson, J., Wedin, P.A., Gulliksson, M.E. et al. Regularization Methods for Uniformly Rank-Deficient Nonlinear Least-Squares Problems. J Optim Theory Appl 127, 1–26 (2005). https://doi.org/10.1007/s10957-005-6389-0
Issue Date:
DOI: https://doi.org/10.1007/s10957-005-6389-0