Generalized and Sparse Least Squares Problems

  • Åke Björck
Part of the NATO ASI Series book series (ASIC, volume 434)


Least squares problems arise frequently in optimization, e.g., in interior point methods. This paper surveys methods for solving least squares problems of nonstandard form such as generalized and sparse problems. Algorithms for standard and banded problems are first studied. Methods for solving generalized least squares problems are then surveyed. The special case of weighted problems is treated in detail. Iterative refinement is discussed as a general technique for improving the accuracy of computed solutions. Least squares problems where the solution is constrained by linear equality constraints or quadratic constraints are also treated.

Graph theoretic methods for reordering rows and columns to reduce fill in when solving sparse least squares problems are surveyed. The numerical phase of sparse Cholesky and sparse QR factorization is then discussed. In particular the multifrontal method, which currently is the most efficient implementation, is described.


Normal Equation Interior Point Method Cholesky Factor Iterative Refinement Triangular System 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    M. Arioli, J. Demmel, and I. S. Duff. Solving sparse linear systems with sparse backward error. SI AM J. Matrix Anal Appl, 10: 165–190, 1989.MathSciNetzbMATHCrossRefGoogle Scholar
  2. [2]
    C. Ashcraft. A vector implementation of the multifrontal method for large sparse positive definite systems. Technical Report ETA-TR-51, Engineering Technology Division, Boeing Computer Services, 1987.Google Scholar
  3. [3]
    J. L. Barlow. Error analysis and implementation aspects of deferred correction for equality constrained least squares problems. SIAM J. Numer. Anal., 25:1340–1358, 1988.MathSciNetCrossRefGoogle Scholar
  4. [4]
    J. L. Barlow and S. L. Handy. The direct solution of weighted and equality constrained least squares problems. SIAM J. Sci. Statist. Comput., 9: 704–716, 1988.MathSciNetzbMATHCrossRefGoogle Scholar
  5. [5]
    Å. Björck. Iterative refinement of linear least squares solutions I. BIT, 7: 257–278, 1967.zbMATHCrossRefGoogle Scholar
  6. [6]
    Å. Björck. Comment on the iterative refinement of least squares solutions. J. Amer. Statist. Assoc., 73: 161–166, 1978.MathSciNetCrossRefGoogle Scholar
  7. [7]
    Å. Björck. A general updating algorithm for constrained linear least squares problems. SIAM J. Sci. Statist. Comput., 5:394–402., 1984.MathSciNetzbMATHCrossRefGoogle Scholar
  8. [8]
    Å. Björck. Stability analysis of the method of semi-normal equations for least squares problems. Linear Algebra Appl, 88 /89: 31–48, 1987.MathSciNetCrossRefGoogle Scholar
  9. [9]
    Å. Björck. Least squares methods. In R G. Ciarlet and J. L. Lions, editors,Handbook of Numerical Analysis, volume I: Finite Difference Methods-Solution of Equations in R n, pages 466–647. Elsevier/North Holland, Amsterdam, 1990.Google Scholar
  10. [10]
    Å. Björck. Algorithms for linear least squares problems. In E. Spedicato, editor,Computer Algorithms for Solving Linear Algebraic Equations; The State of the Art., NATO ASI Series F: Computer and Systems Sciences, Vol. 77, pages 57–92, Berlin, 1991. Springer-Verlag.Google Scholar
  11. [11]
    Å. Björck. Pivoting and stability in the augmented system method. In D. F. Griffiths and G. A. Watson, editors,Numerical Analysis 1991: Proceedings of the 14th Dundee Conference, June 1991, Pitman Research Notes in Mathematics 260, pages 1–16. Longman Scientific and Technical, 1992.Google Scholar
  12. [12]
    Å. Björck. Numerics of Gram-Schmidt orthogonalization. Linear Algebra and Appl., 197: to appear, 1994.Google Scholar
  13. [13]
    Å. Björck and I. S. Duff. A direct method for the solution of sparse linear least squares problems. Linear Algebra and Appl., 34: 43–67, 1980.MathSciNetzbMATHCrossRefGoogle Scholar
  14. [14]
    Å. Björck and G. H. Golub. Iterative refinement of linear least squares solution by Householder transformation. BIT, 7: 322–337, 1967.CrossRefGoogle Scholar
  15. [15]
    J. R. Bunch and L. Kaufman. Some stable methods for calculating inertia and solving symmetric linear systems. Mathematics of Computation, 31: 162–179, 1977.MathSciNetCrossRefGoogle Scholar
  16. [16]
    T. F. Coleman, A. Edenbrandt, and J. R. Gilbert. Predicting fill for sparse orthogonal factorization. J. ACM, 33: 517–532, 1986.MathSciNetzbMATHCrossRefGoogle Scholar
  17. [17]
    M. G. Cox. The least squares solution of overdetermined linear equations having band or augmented band structure. IMA J. Numer. Anal., 1: 3–22, 1981.MathSciNetzbMATHCrossRefGoogle Scholar
  18. [18]
    I. S. Duff. Pivot selection and row orderings in Givens reduction on sparse matrices. Computing, 13: 239–248, 1974.MathSciNetzbMATHCrossRefGoogle Scholar
  19. [19]
    I. S. Duff. On algorithms for obtaining a maximum transversal. ACM Trans. Math. Software, 7: 315–330, 1981.CrossRefGoogle Scholar
  20. [20]
    I. S. Duff, A. M. Erisman, and J. K. Reid. Direct Methods for Sparse Matrices. Oxford University Press, London, 1986.zbMATHGoogle Scholar
  21. [21]
    I. S. Duff, N. I. M. Gould, J. K. Reid, J. A. Scott, and K. Turner. The factorization of sparse symmetric indefinite matrices. IMA J. Numer. Anal, 11: 181–204, 1991.MathSciNetzbMATHCrossRefGoogle Scholar
  22. [22]
    I. S. Duff and J. K. Reid. MA27—a set of Fortran subroutines for solving sparse symmetric sets of linear equations. Technical Report R. 10533, AERE, Harwell, England, 1982.Google Scholar
  23. [23]
    I. S. Duff and J. K. Reid. The multifrontal solution of indefinite sparse symmetric linear systems. ACM Trans. Math. Software, 9: 302–325, 1983.MathSciNetzbMATHCrossRefGoogle Scholar
  24. [24]
    S. C. Eisenstat, M. H. Schultz, and A. H. Sherman. Algorithms and data structures for sparse symmetric gaussian elimination. SI AM J. Sci. Statist. Comput., 2: 225–237, 1981.MathSciNetzbMATHCrossRefGoogle Scholar
  25. [25]
    L. Elden. Algorithms for the regularization of ill-conditioned least squares problems. BIT, 17: 134–145, 1977.MathSciNetzbMATHCrossRefGoogle Scholar
  26. [26]
    L. V. Foster. Modifications of the normal equations method that are numerically stable. In G. H. Golub and P. Van Dooren, editors,Numerical Linear Algebra, Digital Signal Processing and Parallel Algorithms, NATO ASI Series, pages 501— 512, Berlin, 1991. Springer-Verlag.Google Scholar
  27. [27]
    W. Gander. Least squares with a quadratic constraint. Numer. Math., 36:291–307, 1981.MathSciNetzbMATHCrossRefGoogle Scholar
  28. [28]
    W. M. Gentleman. Least squares computations by Givens transformations without square roots. J. Inst. Maths. Applies., 12: 329–336, 1973.MathSciNetzbMATHCrossRefGoogle Scholar
  29. [29]
    J. A. George and M. T. Heath. Solution of sparse linear least squares problems using Givens rotations. Linear Algebra and Its Application, 34: 69–83, 1980.MathSciNetzbMATHCrossRefGoogle Scholar
  30. [30]
    J. A. George and J. H. W. Liu. The evolution of the minimum degree ordering algorithm. SI AM Review, 31: 1–19, 1989.MathSciNetzbMATHCrossRefGoogle Scholar
  31. [31]
    J. A. George and J. W. H. Liu. Computer Solution of Large Sparse Positive Definite Systems. Prentice-Hall, Englewood Cliffs, N.J., 1981.zbMATHGoogle Scholar
  32. [32]
    J. R. Gilbert, C. Moler, and R. Screiber. Sparse matrices in MATLAB: Design and implementation. SIAM J. Matrix. Anal. Appl, 13: 333–356, 1992.MathSciNetzbMATHCrossRefGoogle Scholar
  33. [33]
    P. E. Gill, W. Murray, M. A. Saunders, and M. H. Wright. A Schur-complement method for sparse quadratic programming. In M.G. Cox and S. Hammarling, editors,Reliable Numerical Computation, pages 113–138, Oxford, 1990. Clarendon Press.Google Scholar
  34. [34]
    G. H. Golub and J. H. Wilkinson. Note on the iterative refinement of least squares solution. Numer. Math., 9: 139–148, 1966.MathSciNetzbMATHCrossRefGoogle Scholar
  35. [35]
    M. Gulliksson and P.-A. Wedin. Modifying the QR decomposition to constrained and weighted linear least squares. SIAM J. Matrix. Anal. Appl, 13:4: 1298–1313, 1992.MathSciNetzbMATHCrossRefGoogle Scholar
  36. [36]
    M. T. Heath. Some extensions of an algorithm for sparse linear least squares problems. SIAM J. Sci. Statist. Comput., 3: 223–237, 1982.MathSciNetzbMATHCrossRefGoogle Scholar
  37. [37]
    C. L. Lawson and R. J. Hanson. Solving Least Squares Problems. Prentice Hall, Englewood Cliffs, New Jersey, 1974.zbMATHGoogle Scholar
  38. [38]
    Ö. Leringe and P.-À. Wedin. A comparison between different methods to compute a vector xwhich minimizes ∥Ax — b∥ 2 when Gx = h. Technical Report, Department of Computer Science, Lund University, 1970.Google Scholar
  39. [39]
    J. G. Lewis, D. J. Pierce, and D. C. Wah. A multifrontal Householder QR factorization. Technical Report ECA-TR-127, Engineering and Scientific Services Division, Boeing Computer Services, 1989.Google Scholar
  40. [40]
    J. H. W. Liu. On general row merging schemes for sparse Givens transformations. SIAM J. Sci. Statist. Comput., 7: 1190–1211, 1986.MathSciNetzbMATHCrossRefGoogle Scholar
  41. [41]
    J. H. W. Liu. The role of elimination trees in sparse factorization. SIAM J. Matrix Anal. Appi., 11: 134–172, 1990.zbMATHCrossRefGoogle Scholar
  42. [42]
    P. Matstoms. The Multifrontal Solution of Sparse Linear Least Squares Problems. Lie. Thesis, Linköping University, 1991.Google Scholar
  43. [43]
    P. Matstoms. Sparse QR factorization in MATLAB. ACM Trans. Math. Software, to appear, 1994.Google Scholar
  44. [44]
    C. C. Paige. Computer solution and perturbation analysis of generalized linear least squares problems. Math. Comp., 33: 171–184, 1979.MathSciNetzbMATHCrossRefGoogle Scholar
  45. [45]
    C. C. Paige. Fast numerically stable computations for generalized least squares problems. SIAM J. Numer. Anal., 16: 165–171, 1979.MathSciNetzbMATHCrossRefGoogle Scholar
  46. [46]
    C. C. Paige. The general linear model and the generalized singular value decomposition. Linear Algebra Appi., 70: 269–284, 1985.MathSciNetzbMATHCrossRefGoogle Scholar
  47. [47]
    S. V. Parter. The use of linear graphs in Gauss elimination. SIAM Review, 3: 119–130, 1961.MathSciNetzbMATHCrossRefGoogle Scholar
  48. [48]
    G. Peters and J. H. Wilkinson. The least squares problem and pseudo-inverses. The Computer Journal, 13: 309–316, 1970.zbMATHCrossRefGoogle Scholar
  49. [49]
    A. Pothen and C. J. Fan. Computing the block triangular form of a sparse matrix. ACM Trans. Math. Software, 16: 303–324, 1990.MathSciNetzbMATHCrossRefGoogle Scholar
  50. [50]
    M. J. D. Powell and J. K. Reid. On applying Householder’s method to linear least squares problems. In A. J. M. Morell, editor,Proceedings of the IFIP Congress 68, pages 122–126, Amsterdam, 1965. North Holland.Google Scholar
  51. [51]
    J. K. Reid. A note on the least squares solution of a band system of linear equations by Householder reductions. Comput J., 10:188–189, 1967.MathSciNetzbMATHGoogle Scholar
  52. [52]
    C. F. Van Loan. Computing the CS and the generalized singular value decomposition. Numtr. Math., 46: 479–492, 1985.Google Scholar
  53. [53]
    J. M. Varah. A practical examination of some numerical methods for linear discrete ill-posed problems. SI AM Review, 21: 100–111, 1979.MathSciNetzbMATHCrossRefGoogle Scholar
  54. [54]
    M. H. Wright. Interior methods for constrained optimization. Acta Numerica, 1: 341–407, 1992.CrossRefGoogle Scholar
  55. [55]
    Z. Zlatev. Comparison of two pivotal strategies in sparse plane rotations. Comput. Math. Appl., 8: 119–135, 1982MathSciNetzbMATHCrossRefGoogle Scholar

Copyright information

© Kluwer Academic Publishers 1994

Authors and Affiliations

  • Åke Björck
    • 1
  1. 1.Department of MathematicsLinköping UniversityLinköpingSweden

Personalised recommendations