Skip to main content
SpringerLink
Log in

Variable projection for nonlinear least squares problems

  • Published:
Computational Optimization and Applications Aims and scope Submit manuscript

Abstract

The variable projection algorithm of Golub and Pereyra (SIAM J. Numer. Anal. 10:413–432, 1973) has proven to be quite valuable in the solution of nonlinear least squares problems in which a substantial number of the parameters are linear. Its advantages are efficiency and, more importantly, a better likelihood of finding a global minimizer rather than a local one. The purpose of our work is to provide a more robust implementation of this algorithm, include constraints on the parameters, more clearly identify key ingredients so that improvements can be made, compute the Jacobian matrix more accurately, and make future implementations in other languages easy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

Notes

  1. More general matrices W can be used to simplify the correlation structure in the errors, but in our work we assume that the matrix is diagonal.

  2. Use of the normal equations gives a less reliable solution, and we do not consider it.

  3. ϵ mach is the gap between 1 and the next larger floating-point number.

  4. For example, in (4), \(\varPhi_{1}(t) = e^{\alpha_{1}t}\) depends only on α 1, and \(\varPhi_{2}(t) = e^{\alpha_{2}t}\) depends only on α 2.

  5. The decomposition diagnoses any redundancy in the basis functions, so the user need not be concerned about it.

  6. This solution is unique if rank(Φ w )=n; otherwise we specify the minimum-norm solution. Again, for a “good” model, Φ w should have rank n, but we allow for rank deficiencies.

  7. This definition differs from that used in [25].

  8. We believe that the true values of the linear parameters should be [7.4334e-08, 2.0094e-11, 8.0493e-07] rather than the [2.0094e-11, 7.4334e-08, 3.1559e-05] reported by Krogh.

  9. We show 2 digits in the table, but the output indicates that to 6-digits, no progress is made during the stalls.

  10. This weighting is widely used (but seldom justified) when the data arises from counts. It seems appropriate when the dominant source of error is conversion of observations to numbers with a fixed number of significant digits, since the standard deviation of the error is then proportional to the observation. Other applications of this weighting are given in [1].

  11. varpro.m and lsqnonlin.m stop if the change in the parameters is less than 10−6 or if the change in the weighted sum of squared residuals is less than 10−6 times (1 plus the old value).

References

  1. Amemiya, T.: Regression analysis when the variance of the dependent variable is proportional to the square of its expectation. J. Am. Stat. Assoc. 68, 928–934 (1973)

    Article  MathSciNet  MATH  Google Scholar 

  2. Balda, M.: LMFnlsq, efficient and stable Levenberg-Marquard-Fletcher method for solving of nonlinear equations (15 Nov 2007 (Updated 27 Jan 2009)). http://www.mathworks.com/matlabcentral/fileexchange/17534-lmfnlsqsolution-of-n

  3. Balda, M.: LMFsolve.m: Levenberg-Marquardt-Fletcher algorithm for nonlinear least squares problems (23 Aug 2007 (Updated 11 Feb 2009)). http://www.mathworks.com/matlabcentral/fileexchange/16063-lmfsolvem-levenberg-

  4. Belsley, D.A., Kuh, E., Welsch, R.E.: Regression Diagnostics: Identifying Influential Data and Sources of Collinearity. Wiley, New York (1980)

    Book  MATH  Google Scholar 

  5. Bertsekas, D.P.: Nonlinear Programming, 2nd edn. Athena Scientific, Nashua (2004). ISBN 1-886529-00-0

    Google Scholar 

  6. Björck, Å.: Numerical Methods for Least Squares Problems. SIAM, Philadelphia (1996)

    Book  MATH  Google Scholar 

  7. Bolstad, J.: VARPRO computer program. Computer Science Department, Serra House, Stanford University (January 1977)

  8. Borges, C.F.: A full-Newton approach to separable nonlinear least squares problems and its application to discrete least squares rational approximation. Electron. Trans. Numer. Anal. 35, 57–68 (2009)

    MathSciNet  MATH  Google Scholar 

  9. Chung, J.: Numerical approaches for large-scale ill-posed inverse problems. Ph.D. thesis, Mathematics and Computer Science Department, Emory University, Atlanta, Georgia (2009)

  10. Gay, D.M., Kaufman, L.: Tradeoffs in algorithms for separable nonlinear least squares. In: Proceedings of the 13th World Congress on Computational and Applied Mathematics. Criterion Press, Dublin (1991)

    Google Scholar 

  11. Golub, G.H., Pereyra, V.: The differentiation of pseudoinverses and nonlinear least squares problems whose variables separate. Tech. Rep. STAN-CS-72-261, Computer Science Department, Stanford University, Stanford, CA (1972)

  12. Golub, G.H., Pereyra, V.: The differentiation of pseudoinverses and nonlinear least squares problems whose variables separate. SIAM J. Numer. Anal. 10, 413–432 (1973)

    Article  MathSciNet  MATH  Google Scholar 

  13. Golub, G.H., Pereyra, V.: Separable nonlinear least squares: The variable projection method and its applications. Inverse Probl. 19(2), R1–R26 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  14. Kaufman, L.: A variable projection method for solving separable nonlinear least squares problems. BIT Numer. Math. 15(1), 49–57 (1975)

    Article  MathSciNet  MATH  Google Scholar 

  15. Krogh, F.T.: Efficient implementation of a variable projection algorithm for nonlinear least squares problems. Commun. ACM 17(3), 167–169 (1974)

    Article  MathSciNet  MATH  Google Scholar 

  16. Lawton, W.H., Sylvestre, E.A.: Estimation of linear parameters in nonlinear regression. Technometrics 13(3), 461–467 (1971)

    Article  MATH  Google Scholar 

  17. Mullen, K.M., van Stokkum, I.H.M.: TIMP: an R package for modeling multi-way spectroscopic measurements. J. Stat. Softw. 18(3), 1–46 (2007)

    Google Scholar 

  18. Mullen, K.M., Vengris, M., van Stokkum, I.H.M.: Algorithms for separable nonlinear least squares with application to modelling time-resolved spectra. J. Glob. Optim. 38, 201–213 (2007). doi:10.1007/s10898-006-9071-7

    Article  MATH  Google Scholar 

  19. Paige, C.C., Saunders, M.A.: LSQR: an algorithm for sparse linear equations and sparse least squares. ACM Trans. Math. Softw. 8(1), 43–71 (1982). doi:http://doi.acm.org/10.1145/355984.355989

    Article  MathSciNet  MATH  Google Scholar 

  20. Pereyra, V., Scherer, G. (eds.): Exponential Data Fitting and Its Applications. Bentham Science Publishers, Oak Park (2010)

    Google Scholar 

  21. R Development Core Team: R: A Language and Environment for Statistical Computing, R Foundation for Statistical Computing, Vienna (2010). URL http://www.R-project.org. ISBN 3-900051-07-0

    Google Scholar 

  22. Rall, J.E., Funderlic, R.E.: Interactive VARPRO (INVAR), a nonlinear least squares program. Tech. rep., Oak Ridge National Lab., TN, Report ORNL/CSD-55 (1980)

  23. Rust, B.W., O’Leary, D.P., Mullen, K.M.: Modelling type 1a supernova light curves. In: Pereyra, V., Scherer, G. (eds.) Exponential Data Fitting and Its Applications, pp. 169–186. Bentham Science Publishers, Oak Park (2010)

    Google Scholar 

  24. Sima, D.M., Huffel, S.V.: Separable nonlinear least squares fitting with linear bound constraints and its application in magnetic resonance spectroscopy data quantification. J. Comput. Appl. Math. 203(1), 264–278 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  25. Wolfe, C.M., Rust, B.W., Dunn, J.H., Brown, I.E.: An interactive nonlinear least squares program. Tech. rep., National Bureau of Standards (now NIST), Gaithersburg, MD, NBS Technical Note 1238 (1987)

Download references

Acknowledgements

We are grateful to Ronald F. Boisvert, Julianne Chung, David E. Gilsinn, Katharine M. Mullen, and the referee for helpful comments on the manuscript.

Author information

Authors and Affiliations

  1. National Institute of Standards and Technology, Gaithersburg, MD, USA

    Dianne P. O’Leary

  2. Computer Science Department and Institute for Advanced Computer Studies, University of Maryland, College Park, MD, 20742, USA

    Dianne P. O’Leary

  3. Applied and Computational Mathematics Division, National Institute of Standards and Technology, NIST Stop 8910, Gaithersburg, MD, 20899-8910, USA

    Bert W. Rust

Authors
  1. Dianne P. O’Leary
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bert W. Rust
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dianne P. O’Leary.

Additional information

Certain commercial software products are identified in this paper in order to adequately specify the computational procedures. Such identification does not imply recommendation or endorsement by the National Institute of Standards and Technology nor does it imply that the software products identified are necessarily the best available for the purpose. Part of this work was supported by the National Science Foundation under Grant NSF DMS 1016266.

Electronic Supplementary Material

Below is the link to the electronic supplementary material.

(M 29 kB)

Rights and permissions

Reprints and permissions

About this article

Cite this article

O’Leary, D.P., Rust, B.W. Variable projection for nonlinear least squares problems. Comput Optim Appl 54, 579–593 (2013). https://doi.org/10.1007/s10589-012-9492-9

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10589-012-9492-9

Keywords

Search

Navigation