Skip to main content
SpringerLink
Log in

Generalized and Sparse Least Squares Problems

  • Chapter
Algorithms for Continuous Optimization

Part of the book series: NATO ASI Series ((ASIC,volume 434))

  • 684 Accesses

Abstract

Least squares problems arise frequently in optimization, e.g., in interior point methods. This paper surveys methods for solving least squares problems of nonstandard form such as generalized and sparse problems. Algorithms for standard and banded problems are first studied. Methods for solving generalized least squares problems are then surveyed. The special case of weighted problems is treated in detail. Iterative refinement is discussed as a general technique for improving the accuracy of computed solutions. Least squares problems where the solution is constrained by linear equality constraints or quadratic constraints are also treated.

Graph theoretic methods for reordering rows and columns to reduce fill in when solving sparse least squares problems are surveyed. The numerical phase of sparse Cholesky and sparse QR factorization is then discussed. In particular the multifrontal method, which currently is the most efficient implementation, is described.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. M. Arioli, J. Demmel, and I. S. Duff. Solving sparse linear systems with sparse backward error. SI AM J. Matrix Anal Appl, 10: 165–190, 1989.

    Article  MathSciNet  MATH  Google Scholar 

  2. C. Ashcraft. A vector implementation of the multifrontal method for large sparse positive definite systems. Technical Report ETA-TR-51, Engineering Technology Division, Boeing Computer Services, 1987.

    Google Scholar 

  3. J. L. Barlow. Error analysis and implementation aspects of deferred correction for equality constrained least squares problems. SIAM J. Numer. Anal., 25:1340–1358, 1988.

    Article  MathSciNet  Google Scholar 

  4. J. L. Barlow and S. L. Handy. The direct solution of weighted and equality constrained least squares problems. SIAM J. Sci. Statist. Comput., 9: 704–716, 1988.

    Article  MathSciNet  MATH  Google Scholar 

  5. Å. Björck. Iterative refinement of linear least squares solutions I. BIT, 7: 257–278, 1967.

    Article  MATH  Google Scholar 

  6. Å. Björck. Comment on the iterative refinement of least squares solutions. J. Amer. Statist. Assoc., 73: 161–166, 1978.

    Article  MathSciNet  Google Scholar 

  7. Å. Björck. A general updating algorithm for constrained linear least squares problems. SIAM J. Sci. Statist. Comput., 5:394–402., 1984.

    Article  MathSciNet  MATH  Google Scholar 

  8. Å. Björck. Stability analysis of the method of semi-normal equations for least squares problems. Linear Algebra Appl, 88 /89: 31–48, 1987.

    Article  MathSciNet  Google Scholar 

  9. Å. Björck. Least squares methods. In R G. Ciarlet and J. L. Lions, editors,Handbook of Numerical Analysis, volume I: Finite Difference Methods-Solution of Equations in R n, pages 466–647. Elsevier/North Holland, Amsterdam, 1990.

    Google Scholar 

  10. Å. Björck. Algorithms for linear least squares problems. In E. Spedicato, editor,Computer Algorithms for Solving Linear Algebraic Equations; The State of the Art., NATO ASI Series F: Computer and Systems Sciences, Vol. 77, pages 57–92, Berlin, 1991. Springer-Verlag.

    Google Scholar 

  11. Å. Björck. Pivoting and stability in the augmented system method. In D. F. Griffiths and G. A. Watson, editors,Numerical Analysis 1991: Proceedings of the 14th Dundee Conference, June 1991, Pitman Research Notes in Mathematics 260, pages 1–16. Longman Scientific and Technical, 1992.

    Google Scholar 

  12. Å. Björck. Numerics of Gram-Schmidt orthogonalization. Linear Algebra and Appl., 197: to appear, 1994.

    Google Scholar 

  13. Å. Björck and I. S. Duff. A direct method for the solution of sparse linear least squares problems. Linear Algebra and Appl., 34: 43–67, 1980.

    Article  MathSciNet  MATH  Google Scholar 

  14. Å. Björck and G. H. Golub. Iterative refinement of linear least squares solution by Householder transformation. BIT, 7: 322–337, 1967.

    Article  Google Scholar 

  15. J. R. Bunch and L. Kaufman. Some stable methods for calculating inertia and solving symmetric linear systems. Mathematics of Computation, 31: 162–179, 1977.

    Article  MathSciNet  Google Scholar 

  16. T. F. Coleman, A. Edenbrandt, and J. R. Gilbert. Predicting fill for sparse orthogonal factorization. J. ACM, 33: 517–532, 1986.

    Article  MathSciNet  MATH  Google Scholar 

  17. M. G. Cox. The least squares solution of overdetermined linear equations having band or augmented band structure. IMA J. Numer. Anal., 1: 3–22, 1981.

    Article  MathSciNet  MATH  Google Scholar 

  18. I. S. Duff. Pivot selection and row orderings in Givens reduction on sparse matrices. Computing, 13: 239–248, 1974.

    Article  MathSciNet  MATH  Google Scholar 

  19. I. S. Duff. On algorithms for obtaining a maximum transversal. ACM Trans. Math. Software, 7: 315–330, 1981.

    Article  Google Scholar 

  20. I. S. Duff, A. M. Erisman, and J. K. Reid. Direct Methods for Sparse Matrices. Oxford University Press, London, 1986.

    MATH  Google Scholar 

  21. I. S. Duff, N. I. M. Gould, J. K. Reid, J. A. Scott, and K. Turner. The factorization of sparse symmetric indefinite matrices. IMA J. Numer. Anal, 11: 181–204, 1991.

    Article  MathSciNet  MATH  Google Scholar 

  22. I. S. Duff and J. K. Reid. MA27—a set of Fortran subroutines for solving sparse symmetric sets of linear equations. Technical Report R. 10533, AERE, Harwell, England, 1982.

    Google Scholar 

  23. I. S. Duff and J. K. Reid. The multifrontal solution of indefinite sparse symmetric linear systems. ACM Trans. Math. Software, 9: 302–325, 1983.

    Article  MathSciNet  MATH  Google Scholar 

  24. S. C. Eisenstat, M. H. Schultz, and A. H. Sherman. Algorithms and data structures for sparse symmetric gaussian elimination. SI AM J. Sci. Statist. Comput., 2: 225–237, 1981.

    Article  MathSciNet  MATH  Google Scholar 

  25. L. Elden. Algorithms for the regularization of ill-conditioned least squares problems. BIT, 17: 134–145, 1977.

    Article  MathSciNet  MATH  Google Scholar 

  26. L. V. Foster. Modifications of the normal equations method that are numerically stable. In G. H. Golub and P. Van Dooren, editors,Numerical Linear Algebra, Digital Signal Processing and Parallel Algorithms, NATO ASI Series, pages 501— 512, Berlin, 1991. Springer-Verlag.

    Google Scholar 

  27. W. Gander. Least squares with a quadratic constraint. Numer. Math., 36:291–307, 1981.

    Article  MathSciNet  MATH  Google Scholar 

  28. W. M. Gentleman. Least squares computations by Givens transformations without square roots. J. Inst. Maths. Applies., 12: 329–336, 1973.

    Article  MathSciNet  MATH  Google Scholar 

  29. J. A. George and M. T. Heath. Solution of sparse linear least squares problems using Givens rotations. Linear Algebra and Its Application, 34: 69–83, 1980.

    Article  MathSciNet  MATH  Google Scholar 

  30. J. A. George and J. H. W. Liu. The evolution of the minimum degree ordering algorithm. SI AM Review, 31: 1–19, 1989.

    Article  MathSciNet  MATH  Google Scholar 

  31. J. A. George and J. W. H. Liu. Computer Solution of Large Sparse Positive Definite Systems. Prentice-Hall, Englewood Cliffs, N.J., 1981.

    MATH  Google Scholar 

  32. J. R. Gilbert, C. Moler, and R. Screiber. Sparse matrices in MATLAB: Design and implementation. SIAM J. Matrix. Anal. Appl, 13: 333–356, 1992.

    Article  MathSciNet  MATH  Google Scholar 

  33. P. E. Gill, W. Murray, M. A. Saunders, and M. H. Wright. A Schur-complement method for sparse quadratic programming. In M.G. Cox and S. Hammarling, editors,Reliable Numerical Computation, pages 113–138, Oxford, 1990. Clarendon Press.

    Google Scholar 

  34. G. H. Golub and J. H. Wilkinson. Note on the iterative refinement of least squares solution. Numer. Math., 9: 139–148, 1966.

    Article  MathSciNet  MATH  Google Scholar 

  35. M. Gulliksson and P.-A. Wedin. Modifying the QR decomposition to constrained and weighted linear least squares. SIAM J. Matrix. Anal. Appl, 13:4: 1298–1313, 1992.

    Article  MathSciNet  MATH  Google Scholar 

  36. M. T. Heath. Some extensions of an algorithm for sparse linear least squares problems. SIAM J. Sci. Statist. Comput., 3: 223–237, 1982.

    Article  MathSciNet  MATH  Google Scholar 

  37. C. L. Lawson and R. J. Hanson. Solving Least Squares Problems. Prentice Hall, Englewood Cliffs, New Jersey, 1974.

    MATH  Google Scholar 

  38. Ö. Leringe and P.-À. Wedin. A comparison between different methods to compute a vector xwhich minimizes ∥Ax — b∥ 2 when Gx = h. Technical Report, Department of Computer Science, Lund University, 1970.

    Google Scholar 

  39. J. G. Lewis, D. J. Pierce, and D. C. Wah. A multifrontal Householder QR factorization. Technical Report ECA-TR-127, Engineering and Scientific Services Division, Boeing Computer Services, 1989.

    Google Scholar 

  40. J. H. W. Liu. On general row merging schemes for sparse Givens transformations. SIAM J. Sci. Statist. Comput., 7: 1190–1211, 1986.

    Article  MathSciNet  MATH  Google Scholar 

  41. J. H. W. Liu. The role of elimination trees in sparse factorization. SIAM J. Matrix Anal. Appi., 11: 134–172, 1990.

    Article  MATH  Google Scholar 

  42. P. Matstoms. The Multifrontal Solution of Sparse Linear Least Squares Problems. Lie. Thesis, Linköping University, 1991.

    Google Scholar 

  43. P. Matstoms. Sparse QR factorization in MATLAB. ACM Trans. Math. Software, to appear, 1994.

    Google Scholar 

  44. C. C. Paige. Computer solution and perturbation analysis of generalized linear least squares problems. Math. Comp., 33: 171–184, 1979.

    Article  MathSciNet  MATH  Google Scholar 

  45. C. C. Paige. Fast numerically stable computations for generalized least squares problems. SIAM J. Numer. Anal., 16: 165–171, 1979.

    Article  MathSciNet  MATH  Google Scholar 

  46. C. C. Paige. The general linear model and the generalized singular value decomposition. Linear Algebra Appi., 70: 269–284, 1985.

    Article  MathSciNet  MATH  Google Scholar 

  47. S. V. Parter. The use of linear graphs in Gauss elimination. SIAM Review, 3: 119–130, 1961.

    Article  MathSciNet  MATH  Google Scholar 

  48. G. Peters and J. H. Wilkinson. The least squares problem and pseudo-inverses. The Computer Journal, 13: 309–316, 1970.

    Article  MATH  Google Scholar 

  49. A. Pothen and C. J. Fan. Computing the block triangular form of a sparse matrix. ACM Trans. Math. Software, 16: 303–324, 1990.

    Article  MathSciNet  MATH  Google Scholar 

  50. M. J. D. Powell and J. K. Reid. On applying Householder’s method to linear least squares problems. In A. J. M. Morell, editor,Proceedings of the IFIP Congress 68, pages 122–126, Amsterdam, 1965. North Holland.

    Google Scholar 

  51. J. K. Reid. A note on the least squares solution of a band system of linear equations by Householder reductions. Comput J., 10:188–189, 1967.

    MathSciNet  MATH  Google Scholar 

  52. C. F. Van Loan. Computing the CS and the generalized singular value decomposition. Numtr. Math., 46: 479–492, 1985.

    Google Scholar 

  53. J. M. Varah. A practical examination of some numerical methods for linear discrete ill-posed problems. SI AM Review, 21: 100–111, 1979.

    Article  MathSciNet  MATH  Google Scholar 

  54. M. H. Wright. Interior methods for constrained optimization. Acta Numerica, 1: 341–407, 1992.

    Article  Google Scholar 

  55. Z. Zlatev. Comparison of two pivotal strategies in sparse plane rotations. Comput. Math. Appl., 8: 119–135, 1982

    Article  MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Mathematics, Linköping University, S-581 83, Linköping, Sweden

    Åke Björck

Authors
  1. Åke Björck
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Department of Mathematics, University of Bergamo, Bergamo, Italy

    Emilio Spedicato

Rights and permissions

Reprints and permissions

Copyright information

© 1994 Kluwer Academic Publishers

About this chapter

Cite this chapter

Björck, Å. (1994). Generalized and Sparse Least Squares Problems. In: Spedicato, E. (eds) Algorithms for Continuous Optimization. NATO ASI Series, vol 434. Springer, Dordrecht. https://doi.org/10.1007/978-94-009-0369-2_3

Download citation

  • DOI: https://doi.org/10.1007/978-94-009-0369-2_3

  • Publisher Name: Springer, Dordrecht

  • Print ISBN: 978-94-010-6652-5

  • Online ISBN: 978-94-009-0369-2

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics

Search

Navigation