Computational Optimization and Applications

, Volume 54, Issue 3, pp 579–593 | Cite as

Variable projection for nonlinear least squares problems



The variable projection algorithm of Golub and Pereyra (SIAM J. Numer. Anal. 10:413–432, 1973) has proven to be quite valuable in the solution of nonlinear least squares problems in which a substantial number of the parameters are linear. Its advantages are efficiency and, more importantly, a better likelihood of finding a global minimizer rather than a local one. The purpose of our work is to provide a more robust implementation of this algorithm, include constraints on the parameters, more clearly identify key ingredients so that improvements can be made, compute the Jacobian matrix more accurately, and make future implementations in other languages easy.


Data fitting Model fitting Variable projection method Nonlinear least squares problems Jacobian approximation Least squares approximation Statistical software Mathematical software design and analysis 



We are grateful to Ronald F. Boisvert, Julianne Chung, David E. Gilsinn, Katharine M. Mullen, and the referee for helpful comments on the manuscript.

Supplementary material

10589_2012_9492_MOESM1_ESM.m (29 kb)
(M 29 kB)


  1. 1.
    Amemiya, T.: Regression analysis when the variance of the dependent variable is proportional to the square of its expectation. J. Am. Stat. Assoc. 68, 928–934 (1973) MathSciNetMATHCrossRefGoogle Scholar
  2. 2.
    Balda, M.: LMFnlsq, efficient and stable Levenberg-Marquard-Fletcher method for solving of nonlinear equations (15 Nov 2007 (Updated 27 Jan 2009)).
  3. 3.
    Balda, M.: LMFsolve.m: Levenberg-Marquardt-Fletcher algorithm for nonlinear least squares problems (23 Aug 2007 (Updated 11 Feb 2009)).
  4. 4.
    Belsley, D.A., Kuh, E., Welsch, R.E.: Regression Diagnostics: Identifying Influential Data and Sources of Collinearity. Wiley, New York (1980) MATHCrossRefGoogle Scholar
  5. 5.
    Bertsekas, D.P.: Nonlinear Programming, 2nd edn. Athena Scientific, Nashua (2004). ISBN 1-886529-00-0 Google Scholar
  6. 6.
    Björck, Å.: Numerical Methods for Least Squares Problems. SIAM, Philadelphia (1996) MATHCrossRefGoogle Scholar
  7. 7.
    Bolstad, J.: VARPRO computer program. Computer Science Department, Serra House, Stanford University (January 1977) Google Scholar
  8. 8.
    Borges, C.F.: A full-Newton approach to separable nonlinear least squares problems and its application to discrete least squares rational approximation. Electron. Trans. Numer. Anal. 35, 57–68 (2009) MathSciNetMATHGoogle Scholar
  9. 9.
    Chung, J.: Numerical approaches for large-scale ill-posed inverse problems. Ph.D. thesis, Mathematics and Computer Science Department, Emory University, Atlanta, Georgia (2009) Google Scholar
  10. 10.
    Gay, D.M., Kaufman, L.: Tradeoffs in algorithms for separable nonlinear least squares. In: Proceedings of the 13th World Congress on Computational and Applied Mathematics. Criterion Press, Dublin (1991) Google Scholar
  11. 11.
    Golub, G.H., Pereyra, V.: The differentiation of pseudoinverses and nonlinear least squares problems whose variables separate. Tech. Rep. STAN-CS-72-261, Computer Science Department, Stanford University, Stanford, CA (1972) Google Scholar
  12. 12.
    Golub, G.H., Pereyra, V.: The differentiation of pseudoinverses and nonlinear least squares problems whose variables separate. SIAM J. Numer. Anal. 10, 413–432 (1973) MathSciNetMATHCrossRefGoogle Scholar
  13. 13.
    Golub, G.H., Pereyra, V.: Separable nonlinear least squares: The variable projection method and its applications. Inverse Probl. 19(2), R1–R26 (2003) MathSciNetMATHCrossRefGoogle Scholar
  14. 14.
    Kaufman, L.: A variable projection method for solving separable nonlinear least squares problems. BIT Numer. Math. 15(1), 49–57 (1975) MathSciNetMATHCrossRefGoogle Scholar
  15. 15.
    Krogh, F.T.: Efficient implementation of a variable projection algorithm for nonlinear least squares problems. Commun. ACM 17(3), 167–169 (1974) MathSciNetMATHCrossRefGoogle Scholar
  16. 16.
    Lawton, W.H., Sylvestre, E.A.: Estimation of linear parameters in nonlinear regression. Technometrics 13(3), 461–467 (1971) MATHCrossRefGoogle Scholar
  17. 17.
    Mullen, K.M., van Stokkum, I.H.M.: TIMP: an R package for modeling multi-way spectroscopic measurements. J. Stat. Softw. 18(3), 1–46 (2007) Google Scholar
  18. 18.
    Mullen, K.M., Vengris, M., van Stokkum, I.H.M.: Algorithms for separable nonlinear least squares with application to modelling time-resolved spectra. J. Glob. Optim. 38, 201–213 (2007). doi: 10.1007/s10898-006-9071-7 MATHCrossRefGoogle Scholar
  19. 19.
    Paige, C.C., Saunders, M.A.: LSQR: an algorithm for sparse linear equations and sparse least squares. ACM Trans. Math. Softw. 8(1), 43–71 (1982). doi: MathSciNetMATHCrossRefGoogle Scholar
  20. 20.
    Pereyra, V., Scherer, G. (eds.): Exponential Data Fitting and Its Applications. Bentham Science Publishers, Oak Park (2010) Google Scholar
  21. 21.
    R Development Core Team: R: A Language and Environment for Statistical Computing, R Foundation for Statistical Computing, Vienna (2010). URL ISBN 3-900051-07-0 Google Scholar
  22. 22.
    Rall, J.E., Funderlic, R.E.: Interactive VARPRO (INVAR), a nonlinear least squares program. Tech. rep., Oak Ridge National Lab., TN, Report ORNL/CSD-55 (1980) Google Scholar
  23. 23.
    Rust, B.W., O’Leary, D.P., Mullen, K.M.: Modelling type 1a supernova light curves. In: Pereyra, V., Scherer, G. (eds.) Exponential Data Fitting and Its Applications, pp. 169–186. Bentham Science Publishers, Oak Park (2010) Google Scholar
  24. 24.
    Sima, D.M., Huffel, S.V.: Separable nonlinear least squares fitting with linear bound constraints and its application in magnetic resonance spectroscopy data quantification. J. Comput. Appl. Math. 203(1), 264–278 (2007) MathSciNetMATHCrossRefGoogle Scholar
  25. 25.
    Wolfe, C.M., Rust, B.W., Dunn, J.H., Brown, I.E.: An interactive nonlinear least squares program. Tech. rep., National Bureau of Standards (now NIST), Gaithersburg, MD, NBS Technical Note 1238 (1987) Google Scholar

Copyright information

© US National Institute of Standards and Technology 2012

Authors and Affiliations

  1. 1.National Institute of Standards and TechnologyGaithersburgUSA
  2. 2.Computer Science Department and Institute for Advanced Computer StudiesUniversity of MarylandCollege ParkUSA
  3. 3.Applied and Computational Mathematics DivisionNational Institute of Standards and TechnologyGaithersburgUSA

Personalised recommendations