Hybrid method for nonlinear least-square problems without calculating derivatives

  • C. X. Xu
Contributed Papers


This paper presents a no-derivative modification of the hybrid Gauss-Newton-BFGS method for nonlinear least-square problems suggested initially by Al-Baali and Fletcher and modified later by Fletcher and Xu. The modification is made in such a way that, in a Gauss-Newton step, the Broyden's rank-one updating formula is used to obtain an approximate Jacobian and, in a BFGS step, the Jacobian is estimated using difference formulas. A set of numerical comparisons among the new hybrid method, the Gauss-Newton-Broyden method, and the finite-difference BFGS method is made and shows that the new hybrid method combines the better features of the Gauss-Newton-Broyden method and the finite-difference BFGS method. This paper also extends to the least-square problem the finite-termination property of the Broyden method, proved for a nonsingular system of equations by Gay and for the full-rank rectangular system of equations by Gerber and Luk.

Key Words

Nonlinear least squares hybrid method Gauss-Newton method BFGS method finite-termination property 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Powell, M. J. D.,Some Global Convergence Properties of a Variable Metric Algorithm for Minimization without Exact Line Searches, Nonlinear Programming, Edited by R. W. Cottle and C. E. Lemke, SIAM-AMS Proceedings, Vol. 9, Philadelphia, Pennsylvania, 1976.Google Scholar
  2. 2.
    Fletcher, R.,Practical Methods of Optimization, Vol. 1: Unconstrained Optimization, Wiley, Chichester, England, 1980.Google Scholar
  3. 3.
    Powell, M. J. D.,A Hybrid Method for Nonlinear Equations, Numerical Methods for Nonlinear Equations, Edited by P. Rabinowitz, Gordon and Breach, London, England, 1970.Google Scholar
  4. 4.
    Dennis, J. E., Jr., Gay, D. M., andWelsch, R. E.,An Adaptive Nonlinear Least-Squares Algorithm, TOMS, Vol. 7, pp. 348–368, 1981.Google Scholar
  5. 5.
    Dennis, J. E., Jr., Gay, D. M., andWelsch, R. E.,Algorithm 573, NL2SOL: An Adaptive Nonlinear Least-Square Algorithm], TOMS, Vol. 7, pp. 369–383, 1981.Google Scholar
  6. 6.
    Dennis, J. E., Jr., Martinez, H. J., andTapia, R. A.,A Convergence Theory for the Structured BFGS Secant Method with an Application to Nonlinear Least Squares, Rice University, Mathematical Sciences Department, Technical Report No. 87–15, 1987.Google Scholar
  7. 7.
    Al-Baali, M., andFletcher, R.,Variational Methods for Nonlinear Least Squares, Journal of the Operational Research Society, Vol. 36, pp. 405–421, 1985.Google Scholar
  8. 8.
    Fletcher, R., andXu, C.,Hybrid Methods for Nonlinear Least Squares, IMA Journal of Numerical Analysis, Vol. 7, pp. 371–389, 1987.Google Scholar
  9. 9.
    Dennis, J. E., Jr.,Algorithms for Nonlinear Problems Which Use Discrete Approximations to Derivatives, Proceedings of ACM 1971 National Conference, Chicago, Illinois, 1971.Google Scholar
  10. 10.
    Dennis, J. E., Jr.,Some Computational Techniques for Nonlinear Least-Square Problems, Numerical Solution of Systems of Nonlinear Equations, Edited by G. D. Byrne and C. A. Hall, Academic Press, New York, New York, 1973.Google Scholar
  11. 11.
    Dennis, J. E., Jr.,Nonlinear Least Squares and Equations, The State-of-the-Art in Numerical Analysis, Edited by D. Jacobs, Academic Press, New York, New York, 1977.Google Scholar
  12. 12.
    Boggs, P. T., andDennis, J. E., Jr.,A Stability Analysis for Perturbed Nonlinear Iterative Methods, Mathematics of Computation, Vol. 30, pp. 1–17, 1976.Google Scholar
  13. 13.
    Broyden, C. G.,A Class of Methods for Solving Nonlinear Simultaneous Equations, Mathematics of Computation, Vol. 19, pp. 577–593, 1965.Google Scholar
  14. 14.
    Barnes, J. G. P.,An Algorithm for Solving Nonlinear Equations Based on the Secant Method, Computer Journal, Vol. 8, pp. 66–72, 1965.Google Scholar
  15. 15.
    Gay, D. M.,Some Convergence Properties of Broyden's Method, SIAM Journal on Numerical Analysis, Vol. 16, pp. 623–630, 1979.Google Scholar
  16. 16.
    Gerber, R. R., andLuk, F. T.,A Generalized Broyden's Method for Solving Simultaneous Linear Equations, SIAM Journal on Numerical Analysis, Vol. 18, pp. 882–890, 1980.Google Scholar
  17. 17.
    Stewart, G. W.,A Modification of Davidon's Minimization Method to Accept Difference Approximation of Derivatives, Journal of the ACM, Vol. 14, pp. 72–83, 1967.Google Scholar
  18. 18.
    Gill, P. E., Murray, W., andPitfield, R. A.,The Implementation of Two Revised Quasi-Newton Algotithms for Unconstrained Optimization, National Physical Laboratory, Report No. NAC24, 1972.Google Scholar
  19. 19.
    Fletcher, R.,Fortran Subroutines for Minimization by Quasi-Newton Methods, AERE, Report No. R7125, 1972.Google Scholar
  20. 20.
    Brown, K. M., andDennis, J. E., Jr.,Derivative-Free Analogues of Levenberg-Marquardt and Gauss Algorithms for Nonlinear Least-Square Approximation, Numerische Mathematik, Vol. 18, pp. 289–297, 1972.Google Scholar
  21. 21.
    Gill, P. E., andMurray, W.,Quasi-Newton Methods for Unconstrained Optimization, Journal of Institute of Mathematics and Its Applications, Vol. 9, pp. 91–108, 1972.Google Scholar
  22. 22.
    McKeown, J. J.,Large Residual Nonlinear Least-Square Problems, Nonlinear Optimization: Theory and Algorithms, Edited by L. C. W. Dixon, E. Spedicato, and G. P. Szego, Birkhauser, Boston, Massachusetts, 1980.Google Scholar
  23. 23.
    Ortega, J. M., andRheinboldt, W. G.,Iterative Solution of Nonlinear Equations in Several Variables, Academic Press, New York, New York, 1970.Google Scholar
  24. 24.
    Ben-Israel, A.,On Iterative Methods for Solving Nonlinear Least-Square Problems over Convex Sets, Israel Journal of Mathematics, Vol. 5, pp. 211–224, 1967.Google Scholar
  25. 25.
    Fletcher, R.,Generalized Inverse Methods for the Best Least-Square Solution of Systems of Nonlinear Equations, Computer Journal, Vol. 9, pp. 392–399, 1967.Google Scholar
  26. 26.
    Boggs, P. T.,The Convergence of the Ben-Israel Iteration for Nonlinear Least-Square Problem, Mathematics of Computation, Vol. 30, pp. 199–215, 1976.Google Scholar
  27. 27.
    Rao, C. R., andMitra, S. K.,Generalized Inverse of Matrices and Its Applications, John Wiley, New York, New York, 1971.Google Scholar
  28. 28.
    Dennis, J. E., Jr., andMore, J. J.,Quasi-Newton Methods, Motivation and Theory, SIAM Review, Vol. 19, pp. 46–89, 1975.Google Scholar
  29. 29.
    Xu, C.,Hybrid Methods for Nonlinear Least Squares and Related Problems, University of Dundee, PhD Thesis, 1986.Google Scholar
  30. 30.
    Gill, P. E., andMurray, W.,Safeguarded Steplength Algorithms for Optimization Using Descent Methods, National Physical Laboratory, Report No. NAC37, 1974.Google Scholar
  31. 31.
    Al-Baali, M., andFletcher, R.,An Efficient Line Search for Nonlinear Least Squares, University of Dundee, Numerical Analysis Report No. NA78, 1984.Google Scholar

Copyright information

© Plenum Publishing Corporation 1990

Authors and Affiliations

  • C. X. Xu
    • 1
  1. 1.Department of MathematicsXian Jiaotong UniversityXianChina

Personalised recommendations