Algorithms for Continuous Optimization pp 37-80 | Cite as

# Generalized and Sparse Least Squares Problems

## Abstract

Least squares problems arise frequently in optimization, e.g., in interior point methods. This paper surveys methods for solving least squares problems of nonstandard form such as generalized and sparse problems. Algorithms for standard and banded problems are first studied. Methods for solving generalized least squares problems are then surveyed. The special case of weighted problems is treated in detail. Iterative refinement is discussed as a general technique for improving the accuracy of computed solutions. Least squares problems where the solution is constrained by linear equality constraints or quadratic constraints are also treated.

Graph theoretic methods for reordering rows and columns to reduce fill in when solving sparse least squares problems are surveyed. The numerical phase of sparse Cholesky and sparse QR factorization is then discussed. In particular the multifrontal method, which currently is the most efficient implementation, is described.

## Keywords

Normal Equation Interior Point Method Cholesky Factor Iterative Refinement Triangular System## Preview

Unable to display preview. Download preview PDF.

## References

- [1]M. Arioli, J. Demmel, and I. S. Duff. Solving sparse linear systems with sparse backward error.
*SI AM J. Matrix Anal Appl*, 10: 165–190, 1989.MathSciNetMATHCrossRefGoogle Scholar - [2]C. Ashcraft. A vector implementation of the multifrontal method for large sparse positive definite systems. Technical Report ETA-TR-51, Engineering Technology Division, Boeing Computer Services, 1987.Google Scholar
- [3]J. L. Barlow. Error analysis and implementation aspects of deferred correction for equality constrained least squares problems.
*SIAM J. Numer. Anal.*, 25:1340–1358, 1988.MathSciNetCrossRefGoogle Scholar - [4]J. L. Barlow and S. L. Handy. The direct solution of weighted and equality constrained least squares problems.
*SIAM J. Sci. Statist. Comput.*, 9: 704–716, 1988.MathSciNetMATHCrossRefGoogle Scholar - [5]Å. Björck. Iterative refinement of linear least squares solutions I.
*BIT*, 7: 257–278, 1967.MATHCrossRefGoogle Scholar - [6]Å. Björck. Comment on the iterative refinement of least squares solutions.
*J. Amer. Statist. Assoc.*, 73: 161–166, 1978.MathSciNetCrossRefGoogle Scholar - [7]Å. Björck. A general updating algorithm for constrained linear least squares problems.
*SIAM J. Sci. Statist. Comput.*, 5:394–402., 1984.MathSciNetMATHCrossRefGoogle Scholar - [8]Å. Björck. Stability analysis of the method of semi-normal equations for least squares problems.
*Linear Algebra Appl*, 88 /89: 31–48, 1987.MathSciNetCrossRefGoogle Scholar - [9]Å. Björck. Least squares methods. In R G. Ciarlet and J. L. Lions, editors,
*Handbook of Numerical Analysis*, volume I: Finite Difference Methods-Solution of Equations in*R*^{n}, pages 466–647. Elsevier/North Holland, Amsterdam, 1990.Google Scholar - [10]Å. Björck. Algorithms for linear least squares problems. In E. Spedicato, editor,
*Computer Algorithms for Solving Linear Algebraic Equations; The State of the Art*., NATO ASI Series F: Computer and Systems Sciences, Vol. 77, pages 57–92, Berlin, 1991. Springer-Verlag.Google Scholar - [11]Å. Björck. Pivoting and stability in the augmented system method. In D. F. Griffiths and G. A. Watson, editors,
*Numerical Analysis 1991: Proceedings of the 14th Dundee Conference, June 1991*, Pitman Research Notes in Mathematics 260, pages 1–16. Longman Scientific and Technical, 1992.Google Scholar - [12]Å. Björck. Numerics of Gram-Schmidt orthogonalization.
*Linear Algebra and Appl*., 197: to appear, 1994.Google Scholar - [13]Å. Björck and I. S. Duff. A direct method for the solution of sparse linear least squares problems.
*Linear Algebra and Appl*., 34: 43–67, 1980.MathSciNetMATHCrossRefGoogle Scholar - [14]Å. Björck and G. H. Golub. Iterative refinement of linear least squares solution by Householder transformation.
*BIT*, 7: 322–337, 1967.CrossRefGoogle Scholar - [15]J. R. Bunch and L. Kaufman. Some stable methods for calculating inertia and solving symmetric linear systems.
*Mathematics of Computation*, 31: 162–179, 1977.MathSciNetCrossRefGoogle Scholar - [16]T. F. Coleman, A. Edenbrandt, and J. R. Gilbert. Predicting fill for sparse orthogonal factorization.
*J. ACM*, 33: 517–532, 1986.MathSciNetMATHCrossRefGoogle Scholar - [17]M. G. Cox. The least squares solution of overdetermined linear equations having band or augmented band structure.
*IMA J. Numer. Anal*., 1: 3–22, 1981.MathSciNetMATHCrossRefGoogle Scholar - [18]I. S. Duff. Pivot selection and row orderings in Givens reduction on sparse matrices.
*Computing*, 13: 239–248, 1974.MathSciNetMATHCrossRefGoogle Scholar - [19]I. S. Duff. On algorithms for obtaining a maximum transversal.
*ACM Trans. Math. Software*, 7: 315–330, 1981.CrossRefGoogle Scholar - [20]I. S. Duff, A. M. Erisman, and J. K. Reid.
*Direct Methods for Sparse Matrices*. Oxford University Press, London, 1986.MATHGoogle Scholar - [21]I. S. Duff, N. I. M. Gould, J. K. Reid, J. A. Scott, and K. Turner. The factorization of sparse symmetric indefinite matrices.
*IMA J. Numer. Anal*, 11: 181–204, 1991.MathSciNetMATHCrossRefGoogle Scholar - [22]I. S. Duff and J. K. Reid. MA27—a set of Fortran subroutines for solving sparse symmetric sets of linear equations. Technical Report R. 10533, AERE, Harwell, England, 1982.Google Scholar
- [23]I. S. Duff and J. K. Reid. The multifrontal solution of indefinite sparse symmetric linear systems.
*ACM Trans. Math. Software*, 9: 302–325, 1983.MathSciNetMATHCrossRefGoogle Scholar - [24]S. C. Eisenstat, M. H. Schultz, and A. H. Sherman. Algorithms and data structures for sparse symmetric gaussian elimination.
*SI AM J. Sci. Statist. Comput*., 2: 225–237, 1981.MathSciNetMATHCrossRefGoogle Scholar - [25]L. Elden. Algorithms for the regularization of ill-conditioned least squares problems.
*BIT*, 17: 134–145, 1977.MathSciNetMATHCrossRefGoogle Scholar - [26]L. V. Foster. Modifications of the normal equations method that are numerically stable. In G. H. Golub and P. Van Dooren, editors,
*Numerical Linear Algebra, Digital Signal Processing and Parallel Algorithms*, NATO ASI Series, pages 501— 512, Berlin, 1991. Springer-Verlag.Google Scholar - [27]W. Gander. Least squares with a quadratic constraint.
*Numer. Math*., 36:291–307, 1981.MathSciNetMATHCrossRefGoogle Scholar - [28]W. M. Gentleman. Least squares computations by Givens transformations without square roots.
*J. Inst. Maths. Applies*., 12: 329–336, 1973.MathSciNetMATHCrossRefGoogle Scholar - [29]J. A. George and M. T. Heath. Solution of sparse linear least squares problems using Givens rotations.
*Linear Algebra and Its Application*, 34: 69–83, 1980.MathSciNetMATHCrossRefGoogle Scholar - [30]J. A. George and J. H. W. Liu. The evolution of the minimum degree ordering algorithm.
*SI AM Review*, 31: 1–19, 1989.MathSciNetMATHCrossRefGoogle Scholar - [31]J. A. George and J. W. H. Liu.
*Computer Solution of Large Sparse Positive Definite Systems*. Prentice-Hall, Englewood Cliffs, N.J., 1981.MATHGoogle Scholar - [32]J. R. Gilbert, C. Moler, and R. Screiber. Sparse matrices in MATLAB: Design and implementation.
*SIAM J. Matrix. Anal. Appl*, 13: 333–356, 1992.MathSciNetMATHCrossRefGoogle Scholar - [33]P. E. Gill, W. Murray, M. A. Saunders, and M. H. Wright. A Schur-complement method for sparse quadratic programming. In M.G. Cox and S. Hammarling, editors,
*Reliable Numerical Computation*, pages 113–138, Oxford, 1990. Clarendon Press.Google Scholar - [34]G. H. Golub and J. H. Wilkinson. Note on the iterative refinement of least squares solution.
*Numer. Math*., 9: 139–148, 1966.MathSciNetMATHCrossRefGoogle Scholar - [35]M. Gulliksson and P.-A. Wedin. Modifying the QR decomposition to constrained and weighted linear least squares.
*SIAM J. Matrix. Anal. Appl*, 13:4: 1298–1313, 1992.MathSciNetMATHCrossRefGoogle Scholar - [36]M. T. Heath. Some extensions of an algorithm for sparse linear least squares problems.
*SIAM J. Sci. Statist. Comput*., 3: 223–237, 1982.MathSciNetMATHCrossRefGoogle Scholar - [37]C. L. Lawson and R. J. Hanson.
*Solving Least Squares Problems*. Prentice Hall, Englewood Cliffs, New Jersey, 1974.MATHGoogle Scholar - [38]Ö. Leringe and P.-À. Wedin. A comparison between different methods to compute a vector
*x*which minimizes*∥Ax — b∥*_{2}when*Gx = h*. Technical Report, Department of Computer Science, Lund University, 1970.Google Scholar - [39]J. G. Lewis, D. J. Pierce, and D. C. Wah. A multifrontal Householder QR factorization. Technical Report ECA-TR-127, Engineering and Scientific Services Division, Boeing Computer Services, 1989.Google Scholar
- [40]J. H. W. Liu. On general row merging schemes for sparse Givens transformations.
*SIAM J. Sci. Statist. Comput*., 7: 1190–1211, 1986.MathSciNetMATHCrossRefGoogle Scholar - [41]J. H. W. Liu. The role of elimination trees in sparse factorization.
*SIAM J. Matrix Anal. Appi*., 11: 134–172, 1990.MATHCrossRefGoogle Scholar - [42]P. Matstoms. The Multifrontal Solution of Sparse Linear Least Squares Problems. Lie. Thesis, Linköping University, 1991.Google Scholar
- [43]P. Matstoms. Sparse QR factorization in MATLAB.
*ACM Trans. Math. Software*, to appear, 1994.Google Scholar - [44]C. C. Paige. Computer solution and perturbation analysis of generalized linear least squares problems.
*Math. Comp*., 33: 171–184, 1979.MathSciNetMATHCrossRefGoogle Scholar - [45]C. C. Paige. Fast numerically stable computations for generalized least squares problems.
*SIAM J. Numer. Anal*., 16: 165–171, 1979.MathSciNetMATHCrossRefGoogle Scholar - [46]C. C. Paige. The general linear model and the generalized singular value decomposition.
*Linear Algebra Appi*., 70: 269–284, 1985.MathSciNetMATHCrossRefGoogle Scholar - [47]S. V. Parter. The use of linear graphs in Gauss elimination.
*SIAM Review*, 3: 119–130, 1961.MathSciNetMATHCrossRefGoogle Scholar - [48]G. Peters and J. H. Wilkinson. The least squares problem and pseudo-inverses.
*The Computer Journal*, 13: 309–316, 1970.MATHCrossRefGoogle Scholar - [49]A. Pothen and C. J. Fan. Computing the block triangular form of a sparse matrix.
*ACM Trans. Math. Software*, 16: 303–324, 1990.MathSciNetMATHCrossRefGoogle Scholar - [50]M. J. D. Powell and J. K. Reid. On applying Householder’s method to linear least squares problems. In A. J. M. Morell, editor,
*Proceedings of the IFIP Congress**68*, pages 122–126, Amsterdam, 1965. North Holland.Google Scholar - [51]J. K. Reid. A note on the least squares solution of a band system of linear equations by Householder reductions.
*Comput J*., 10:188–189, 1967.MathSciNetMATHGoogle Scholar - [52]C. F. Van Loan. Computing the CS and the generalized singular value decomposition.
*Numtr. Math*., 46: 479–492, 1985.Google Scholar - [53]J. M. Varah. A practical examination of some numerical methods for linear discrete ill-posed problems.
*SI AM Review*, 21: 100–111, 1979.MathSciNetMATHCrossRefGoogle Scholar - [54]M. H. Wright. Interior methods for constrained optimization.
*Acta Numerica*, 1: 341–407, 1992.CrossRefGoogle Scholar - [55]Z. Zlatev. Comparison of two pivotal strategies in sparse plane rotations.
*Comput. Math. Appl*., 8: 119–135, 1982MathSciNetMATHCrossRefGoogle Scholar