Abstract
The effectiveness of sparse matrix techniques for directly solving large-scale linear least-squares problems is severely limited if the system matrix A has one or more nearly dense rows. In this paper, we partition the rows of A into sparse rows and dense rows (As and Ad) and apply the Schur complement approach. A potential difficulty is that the reduced normal matrix AsTAs is often rank-deficient, even if A is of full rank. To overcome this, we propose explicitly removing null columns of As and then employing a regularization parameter and using the resulting Cholesky factors as a preconditioner for an iterative solver applied to the symmetric indefinite reduced augmented system. We consider complete factorizations as well as incomplete Cholesky factorizations of the shifted reduced normal matrix. Numerical experiments are performed on a range of large least-squares problems arising from practical applications. These demonstrate the effectiveness of the proposed approach when combined with either a sparse parallel direct solver or a robust incomplete Cholesky factorization algorithm.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Andersen, E.D., Gondzio, J, Mészáros, C., Xu, X.: Implementation of Interior Point Methods for Large Scale Linear Programming. HEC/Université de Genève (1996)
Andersen, K.D: A modified Schur complement method for handling dense columns in interior point methods for linear programming (1996)
Anderson, O.D.: An improved approach to inverting the autocovariance matrix of a general mixed autoregressive moving average time process. Australian J. Statist. 18(1–2), 73–75 (1976)
Anderson, O.D.: On the inverse of the autocovariance matrix for a general moving average process. Biometrika 63(2), 391–394 (1976)
Avron, H., Ng, E., Toledo, S.: Using perturbed Q R factorizations to solve linear least-squares problems. SIAM J. Matrix Anal. Appl. 31(2), 674–693 (2009)
Benzi, M.: Preconditioning techniques for large linear systems: a survey. J. Comput. Phys. 182(2), 418–477 (2002)
Björck, Å.: A general updating algorithm for constrained linear least squares problems. SIAM J. Sci. Statist. Comput. 5(2), 394–402 (1984)
Björck, Å: Numerical Methods for Least Squares Problems. SIAM, Philadelphia (1996)
Davis, T.A., Hu, Y.: The University of Florida sparse matrix collection. ACM Trans. Math. Softw. 38(1), 1, 1–28 (2011)
Diniz, P.S.R.: Adaptive Filtering: Algorithms and Practical Implementation. Springer, 4th ed 2013 edition (2012)
Dollar, H.S., Scott, J.A.: A note on fast approximate minimum degree orderings for matrices with some dense rows. Numer. Linear Algebra Appl. 17, 43–55 (2010)
Duff, I.S.: MA57—a code for the solution of sparse symmetric definite and indefinite systems. ACM Trans. Math. Softw. 30, 118–154 (2004)
Fong, D.C.-L., Saunders, M.A.: LSMR: an iterative algorithm for sparse least-squares problems. SIAM J. Sci. Comput. 33(5), 2950–2971 (2011)
George, A., Heath, M.T.: Solution of sparse linear least squares problems using Givens rotations. Linear Algebra Appl. 34, 69–83 (1980)
Gill, P.E., Murray, W., Ponceleon, D.B., Saunders, M.A.: Solving reduced KKT systems in barrier methods for linear and quadratic programming. Technical Report SOL 91-7, Department of Operations Research Stanford University (1991)
Gill, P.E., Saunders, M.A., Shinnerl, J.R.: On the stability of Cholesky factorization for symmetric quasidefinite systems. SIAM J. Matrix Anal. Appl. 17 (1), 35–46 (1996)
Goldfarb, D., Scheinberg, K.: A product-form Cholesky factorization method for handling dense columns in interior point methods for linear programming. Math. Program. Series A 99, 1–34 (2004)
Golub, G.H., Van Loan, C.F.: Unsymmetric positive definite linear systems. Linear Algebra Appl. 28, 85–97 (1979)
Gould, N.I.M., Orban, D., Toint, Ph.L: CUTEst: a constrained and unconstrained testing environment with safe threads for mathematical optimization. Comput. Optim. Appl. 60, 545–557 (2015)
Gould, N.I.M., Scott, J.A.: The state-of-the-art of preconditioners for sparse linear least squares problems. ACM Trans. Math. Softw. 43(4), 36,1–35 (2017)
Greif, C., He, S., Liu, P.: SYM-ILDL C++ package for incomplete factorizations of symmetric indefinite matrices https://github.com/inutard/matrix-factor (2013)
Hogg, J.D., Reid, J.K., Scott, J.A.: Design of a multicore sparse Cholesky factorization using DAGs. SIAM J. Sci. Comput. 32, 3627–3649 (2010)
Hogg, J.D., Scott, J.A.: HSL_MA97: A bit-compatible multifrontal code for sparse symmetric systems. Technical Report RAL-TR-2011-024 Rutherford Appleton Laboratory (2011)
HSL: A collection of Fortran codes for large-scale scientific computation. http://www.hsl.rl.ac.uk (2017)
Kalman, R.E.: A new approach to linear filtering and prediction problems. Trans. ASME–J. Basic Eng. 82(Series D), 35–45 (1960)
Karypis, G., Kumar, V.: METIS: software package for partitioning unstructured graphs, partitioning meshes, and computing fill-reducing orderings of sparse matrices (version 3.0). Technical report, University of Minnesota Department of Computer Science and Army HPC Research Center (1997)
Lin, C.-J., Moré, J.J.: Incomplete Cholesky factorizations with limited memory. SIAM J. Sci. Comput. 21(1), 24–45 (1999)
Lustig, I.J., Marsten, R.E., Shanno, D.F.: Computational experience with a primal-dual interior point method for linear programming. Linear Algebra Appl. 152, 191–222 (1991)
Manteuffel, T.A.: An incomplete factorization technique for positive definite linear systems. Math. Comput. 34, 473–497 (1980)
Marxen, A.: Primal barrier methods for linear programming. Technical Report SOL 89-6, Department of Operations Research Stanford University (1989)
MUMPS: A MUltifrontal Massively Parallel sparse direct Solver. http://graal.ens-lyon.fr/MUMPS/ (2016)
Ng, E.: A scheme for handling rank-deficiency in the solution of sparse linear least squares problems. SIAM J. Sci. Statist. Comput. 12(5), 1173–1183 (1991)
Okulicka-DłuŻewska, F., Rozložník, M., Smoktunowicz, A.: Cholesky-like factorization of symmetric indefinite matrices and orthogonalization with respect to bilinear forms. SIAM J. Matrix Anal. Appl. 36(2), 727–751 (2015)
Orban, D.: Limited-memory LDL⊤ factorization of symmetric quasi-definite matrices with application to constrained optimization. Numer. Algor. 70(1), 9–41 (2015)
Orban, D., Arioli, M.: Iterative Solution of Symmetric Quasi-Definite Linear Systems, volume 3 of SIAM Spotlights Society for Industrial and Applied Mathematics. SIAM, Philadelphia (2017)
Paige, C.C., Saunders, M.A.: Solution of sparse indefinite systems of linear equations. SIAM J. Numer. Anal. 12(4), 617–629 (1975)
Paige, C.C., Saunders, M.A.: Algorithm 583; LSQR: Sparse linear equations and least-squares problems. ACM Trans. Math. Softw. 8(2), 195–209 (1982)
Paige, C.C., Saunders, M.A: LSQR: an algorithm for sparse linear equations and sparse least squares. ACM Trans. Math. Softw. 8(1), 43–71 (1982)
Reid, J.K., Scott, J.A.: An out-of-core sparse Cholesky solver. ACM Trans. Math. Softw. 36(2), 9,1–36 (2009)
Saad, Y.: Iterative Methods for Sparse Linear Systems, 2nd edn. SIAM, Philadelphia (2003)
Saad, Y., Schultz, M.H: GMRES: a generalized minimal residual algorithm for solving nonsymmetric linear systems. SIAM J. Sci. Stat. Comput. 7, 856–869 (1986)
Saunders, M.A.: Cholesky-based methods for sparse least squares: The benefits of regularization. Technical Report SOL 95-1, Department of Operations Research, Stanford University, 1995. In: Adams, L., Nazareth, J. L. (eds.) Linear and Nonlinear Conjugate Gradient-Related Methods, pp 92–100. SIAM, Philadelphia (1996)
Sayed, A.H: Fundamentals of Adaptive Filtering. Wiley (2003)
Scott, J.A.: On using Cholesky-based factorizations for solving rank-deficient sparse linear least-squares problems. SIAM J. Sci. Comput. 39(4), C319–C339 (2017)
Scott, J.A., Tůma, M.: HSL_MI28: an efficient and robust limited-memory incomplete Cholesky factorization code. ACM Trans. Math. Softw 40(4), 24,1–19 (2014)
Scott, J.A., Tůma, M.: On positive semidefinite modification schemes for incomplete Cholesky factorization. SIAM J. Sci. Comput. 36(2), A609–A633 (2014)
Scott, J.A., Tůma, M.: On signed incomplete Cholesky factorization preconditioners for saddle-point systems. SIAM J. Sci. Comput. 36(6), A2984–A3010 (2014)
Scott, J.A., Tůma, M.: Solving mixed sparse-dense linear least-squares problems by preconditioned iterative methods. SIAM J. Sci. Comput. 39(6), A2422–A2437 (2017)
Sun, C.: Dealing with dense rows in the solution of sparse linear least squares problems. Research Report CTC95TR227, Advanced Computing Research Institute Cornell Theory Center; Cornell University (1995)
Sun, C.: Parallel solution of sparse linear least squares problems on distributed-memory multiprocessors. Parallel Comput. 23(13), 2075–2093 (1997)
Vanderbei, R.J.: Splitting dense columns in sparse linear systems. Linear Algebra Appl. 152, 107–117 (1991)
Vanderbei, R.J.: Symmetric quasidefinite matrices. SIAM J. Optim. 5(1), 100–113 (1995)
WSMP: Watson sparse matrix package (WSMP). http://researcher.watson.ibm.com/researcher/view_group.php?id=1426 (2016)
Acknowledgements
We are very grateful to Michael Saunders and to an anonymous reviewer for their careful reading of our manuscript and for their constructive comments and suggestions that led to improvements in the presentation of this paper.
Author information
Authors and Affiliations
Corresponding author
Additional information
Jennifer Scott was supported by EPSRC grant EP/M025179/1. Miroslav Tu̇ma was supported by projects 17-04150J and 18-12719S of the Grant Agency of the Czech Republic.
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
About this article
Cite this article
Scott, J., Tůma, M. A Schur complement approach to preconditioning sparse linear least-squares problems with some dense rows. Numer Algor 79, 1147–1168 (2018). https://doi.org/10.1007/s11075-018-0478-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11075-018-0478-2