Abstract
Partial eigenvalue decomposition (PEVD) and partial singular value decomposition (PSVD) of large sparse matrices are of fundamental importance in a wide range of applications, including latent semantic indexing, spectral clustering, and kernel methods for machine learning. The more challenging problems are when a large number of eigenpairs or singular triplets need to be computed. We develop practical and efficient algorithms for these challenging problems. Our algorithms are based on a filter-accelerated block Davidson method. Two types of filters are utilized, one is Chebyshev polynomial filtering, the other is rational-function filtering by solving linear equations. The former utilizes the fastest growth of the Chebyshev polynomial among same degree polynomials; the latter employs the traditional idea of shift-invert, for which we address the important issue of automatic choice of shifts and propose a practical method for solving the shifted linear equations inside the block Davidson method. Our two filters can efficiently generate high-quality basis vectors to augment the projection subspace at each Davidson iteration step, which allows a restart scheme using an active projection subspace of small dimension. This makes our algorithms memory-economical, thus practical for large PEVD/PSVD calculations. We compare our algorithms with representative methods, including ARPACK, PROPACK, the randomized SVD method, and the limited memory SVD method. Extensive numerical tests on representative datasets demonstrate that, in general, our methods have similar or faster convergence speed in terms of CPU time, while requiring much lower memory comparing with other methods. The much lower memory requirement makes our methods more practical for large-scale PEVD/PSVD computations.
Similar content being viewed by others
References
Bache K, Lichman M. UCI machine learning repository. http://archive.ics.uci.edu/ml, 2013
Baer R, Head-Gordon M. Chebyshev expansion methods for electronic structure calculations on large molecular systems. J Chem Phys, 1997, 107: 10003–10013
Baglama J, Calvetti D, Reichel L. IRBL: An implicitly restarted block-Lanczos method for large-scale Hermitian eigenproblems. SIAM J Sci Comput, 2003, 24: 1650–1677
Bai Z, Demmel J, Dongarra J, et al. Templates for the Solution of Algebraic Eigenvalue Problems: A Practical Guide. Philadelphia: SIAM, 2000
Belabbas M-A, Wolfe P J. Spectral methods in machine learning and new strategies for very large datasets. Proc Natl Acad Sci USA, 2009, 106: 369–374
Berns-Müller J, Graham I G, Spence A. Inexact inverse iteration for symmetric matrices. Linear Algebra Appl, 2006, 416: 389–413
Cai D, He X, Han J. Spectral regression: A unified subspace learning framework for content-based image retrieval. In: Proceedings of the 15th International Conference on Multimedia. New York: ACM, 2007, 403–412
Cai J F, Candès E J, Shen Z. A singular value thresholding algorithm for matrix completion. SIAM J Optim, 2010, 20: 1956–1982
Calvetti D, Reichel L, Sorensen D C. An implicit restarted Lanczos method for large symmetric eigenvalue problem. Elec Trans Numer Anal, 1994, 1: 237–263
Chebyshev P L. Sur les fonctions qui s’écartent peu de zéro pour certaines valeurs de la variable. Oeuvreas, 1881, 2: 335–356
Chen J, Saad Y. Lanczos vectors versus singular vectors for effective dimension reduction. IEEE Trans Knowl Data Eng, 2009, 21: 1091–1103
Davidson E R. The iterative calculation of a few of the lowest eigenvalues and corresponding eigenvectors of large real-symmetric matrices. J Comput Phys, 1975, 17: 87–94
Deerwester S, Dumais S T, Furnas G W, et al. Indexing by latent semantic analysis. J Amer Soc Inform Sci, 1990, 41: 391–407
Fan R-E, Lin C-J. Libsvm data: Classification, regression, and multi-label. http://www.csie.ntu.edu.tw/~cjlin/libs vmtools/datasets/
Fokkema D R, Sleijpen G L G, van der Vorst H A. Jacobi-Davidson style QR and QZ algorithms for the reduction of matrix pencils. SIAM J Sci Comput, 1998, 20: 94–125
Freitag M A. Inner-outer Iterative Methods for Eigenvalue Problems—Convergence and Preconditioning. PhD Thesis. Bath: University of Bath, 2007
Gleich D F, Gray A P, Greif C, et al. An inner-outer iteration for computing PageRank. SIAM J Sci Comput, 2010, 32: 349–371
Golub G H, Kahan W. Calculating the singular values and pseudo-inverse of a matrix. SIAM J Numer Anal, 1965, 2: 205–224
Golub G H, Van Loan C F. Matrix Computations. Baltimore: The Johns Hopkins University Press, 1996
Golub G H, Ye Q. Inexact preconditioned conjugate gradient method with inner-outer iteration. SIAM J Sci Comput, 1999, 21: 1305–1320
Halko N, Martinsson P-G, Tropp J A. Finding structure with randomness: Stochastic algorithms for constructing approximate matrix decompositions. SIAM Rev, 2009, 53: 217–288
Knyazev A V. Toward the optimal preconditioned eigensolver: Locally optimal block preconditioned conjugate gradient method. SIAM J Sci Comput, 2001, 23: 517–541
Ko Y. A study of term weighting schemes using class information for text classification. In: SIGIR’ 12 Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval. New York: ACM, 2012, 1029–1030
Kohanoff J. Electronic Structure Calculations for Solids and Molecules: Theory and Computational Methods. Cambridge: Cambridge University Press, 2006
Kunegis J, Lommatzsch A. Learning spectral graph transformations for link prediction. In: Proceedings of the 26th International Conference on Machine Learning. New York: ACM, 2009, 561–568
Lanczos C. An iteration method for the solution of the eigenvalue problem of linear differential and integral operators. J Res Nat Bur Standards, 1950, 45: 255–282
Larsen R M. Lanczos bidiagonalization with partial reorthogonalization. Technical Report DAIMI PB-357. Aarhus: Aarhus University, 1998
LeCun Y, Cortes C, Burges C J C. The mnist database of handwritten digits. http://yann.lecun.com/exdb/mnist/
Lehoucq R B, Meerbergen K. Using generalized Cayley transformations within an inexact rational Krylov sequence method. SIAM J Matrix Anal Appl, 1998, 20: 131–148
Lehoucq R B, Sorensen D C, Yang C. ARPACK Users’ Guide: Solution of Large-Scale Eigenvalue Problems with Implicitly Restarted Arnoldi Methods. Phildelphia: SIAM, 1998
Leskovec J, Krevl A. SNAP Datasets: Stanford large network dataset collection. http://snap.stanford.edu/data, 2014
Li J, White J. Low rank solution of Lyapunov equations. SIAM J Matrix Anal Appl, 2002, 24: 260–280
Liang Q, Ye Q. Computing singular values of large matrices with inverse free preconditioned krylov subspace method. Electron Trans Numer Anal, 2014, 42: 197–221
Liu X, Wen Z, Zhang Y. Limited memory block Krylov subspace optimization for computing dominant singular value decompositions. SIAM J Sci Comput, 2013, 35: A1641–A1668
Ma S, Goldfarb D, Chen L. Fixed point and Bregman iterative methods for matrix rank minimization. Math Program, 2011, 128: 321–353
Martin R M. Electronic Structure: Basic Theory and Practical Methods. Cambridge: Cambridge University Press, 2004
Mazumder R, Hastie T, Tibshirani R. Spectral regularization algorithms for learning large incomplete matrices. J Mach Learn Res, 2010, 11: 2287–2322
Morgan R B, Scott D S. Generalizations of Davidson’s method for computing eigenvalues of sparse symmetric matrices. SIAM J Sci Stat Comput, 1986, 7: 817–825
Ng A Y, Jordan M I, Weiss Y. On spectral clustering: Analysis and an algorithm. Adv Neural Inf Process Syst, 2002, 14: 849–856
Notay Y. Combination of Jacobi-Davidson and conjugate gradients for the partial symmetric eigenproblem. Numer Linear Algebra Appl, 2002, 9: 21–44
Ovtchinnikov E E. Jacobi correction equation, line search, and conjugate gradients in hermitian eigenvalue computation II: Computing several extreme eigenvalues. SIAM J Numer Anal, 2008, 46: 2593–2619
Parlett B N. The Symmetric Eigenvalue Problem. Philadelphia: SIAM, 1998
Penzl T. A cyclic low-rank Smith method for large sparse Lyapunov equations. SIAM J Sci Comput, 2000, 21: 1401–1418
Peters G, Wilkinson J H. Inverse iteration, ill-conditioned equations, and Newton’s method. SIAM Rev, 1979, 21: 339–360
Prateek J, Meka R, Dhillon I S. Guaranteed rank minimization via singular value projection. Adv Neural Inf Process Syst, 2010, 23: 937–945
Rennie J. 20 newsgroups. http://qwone.com/~jason/20Newsgroups/, 2008
Rivlin T J. Chebyshev Polynomials. New York: John Wiley & Sons, 1974
Rokhlin V, Szlam A, Tygert M. A randomized algorithm for principal component analysis. SIAM J Matrix Anal Appl, 2009, 31: 1100–1124
Roman J E, Campos C, Romero E, et al. SLEPc users manual. Technical Report DSIC-II/24/02-Revision 3.6. València: Universitat Politècnica de València, 2015
Ruhe A, Skoogh D. Rational Krylov algorithm for eigenvalue computation and model reduction. In: Proceedings of the 4th International Workshop on Applied Parallel Computing. Large Scale Scientific and Industrial Problems. New York: Springer-Verlag, 1998: 49150–2
Ruhe A, Wiberg T. The method of conjugate gradients used in inverse iteration. BIT, 1972, 12: 543–554
Saad Y. A flexible inner-outer preconditioned GMRES algorithm. SIAM J Sci Comput, 1993, 14: 461–469
Saad Y. Iterative Methods for Sparse Linear Systems. Philadelphia: SIAM, 2003
Saad Y. Numerical Methods for Large Eigenvalue Problems. Philadelphia: SIAM, 2011
Salton G, Buckley C. Term-weighting approaches in automatic text retrieval. Inform Process Manag, 1988, 24: 513–523
Sankey O F, Drabold D A, Gibson A. Projected random vectors and the recursion method in the electronic-structure problem. Phys Rev B, 1994, 50: 1376–1381
Simon H D. The Lanczos algorithm with partial reorthogonalization. Math Comput, 1984, 42: 115–142
Simoncini V, Eldén L. Inexact Rayleigh quotient-type methods for eigenvalue computations. BIT, 2002, 42: 159–182
Sleijpen G L G, van der Vorst H A. A Jacobi-Davidson iteration method for linear eigenvalue problems. SIAM J Matrix Anal Appl, 1996, 17: 401–425
Sorensen D C. Implicit application of polynomial filters in a k-step Arnoldi method. SIAM J Matrix Anal Appl, 1992, 13: 357–385
Sorensen D C. Implicitly restarted Arnoldi/Lanczos methods for large scale eigenvalue calculations. Technical Report TR-96-40. Houston: Rice University, 1996
Stathopoulos A, McCombs J R. Nearly optimal preconditioned methods for hermitian eigenproblems under limited memory, part II: Seeking many eigenvalues. SIAM J Sci Comput, 2007, 29: 2162–2188
Stathopoulos A, McCombs J R. PRIMME: PReconditioned Iterative MultiMethod Eigensolver: Methods and software description. ACM Trans Math Software, 2010, 37: 21–30
Toh K C, Yun S. An accelerated proximal gradient algorithm for nuclear norm regularized linear least squares problems. Pac J Optim, 2010, 6: 615–640
von Luxburg U. A tutorial on spectral clustering. Stat Comput, 2007, 17: 395–416
Wu K, Simon H D. A parallel Lanczos method for symmetric generalized eigenvalue problems. Report 41284. Lawrence: Lawrence Berkeley National Laboratory, 1997
Wu L, Stathopoulos A. PRIMME SVDS: A preconditioned SVD solver for computing accurately singular triplets of large matrices based on the PRIMME eigensolver. ArXiv:1408.5535, 2014
Xue F, Elman H C. Fast inexact subspace iteration for generalized eigenvalue problems with spectral transformation. Linear Algebra Appl, 2011, 435: 601–622
Zhou Y. Studies on Jacobi-Davidson, rayleigh quotient iteration, inverse iteration generalized Davidson and Newton updates. Numer Linear Algebra Appl, 2006, 13: 621–642
Zhou Y. A block Chebyshev-Davidson method with inner-outer restart for large eigenvalue problems. J Comput Phys, 2010, 229: 9188–9200
Zhou Y. Practical acceleration for computing the HITS ExpertRank vectors. J Comput Appl Math, 2012, 236: 4398–4409
Zhou Y, Chelikowsky J R, Saad Y. Chebyshev-filtered subspace iteration method free of sparse diagonalization for solving the Kohn-Sham equation. J Comput Phys, 2014, 274: 770–782
Zhou Y, Li R-C. Bounding the spectrum of large Hermitian matrices. Linear Algebra Appl, 2011, 435: 480–493
Zhou Y, Saad Y. A Chebyshev-Davidson algorithm for large symmetric eigenvalue problems. SIAM J Matrix Anal Appl, 2007, 29: 954–971
Zhu X, Kandola J, Lafferty J, et al. Graph kernels by spectral transforms. In: Chapelle O, Scholkopf B, Zien A, eds. Semi-Supervised Learning. Cambridge: MIT Press, 2006, 277–291
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Zhou, Y., Wang, Z. & Zhou, A. Accelerating large partial EVD/SVD calculations by filtered block Davidson methods. Sci. China Math. 59, 1635–1662 (2016). https://doi.org/10.1007/s11425-016-0274-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11425-016-0274-0