Skip to main content
Log in

Accelerating large partial EVD/SVD calculations by filtered block Davidson methods

Science China Mathematics Aims and scope Submit manuscript

Abstract

Partial eigenvalue decomposition (PEVD) and partial singular value decomposition (PSVD) of large sparse matrices are of fundamental importance in a wide range of applications, including latent semantic indexing, spectral clustering, and kernel methods for machine learning. The more challenging problems are when a large number of eigenpairs or singular triplets need to be computed. We develop practical and efficient algorithms for these challenging problems. Our algorithms are based on a filter-accelerated block Davidson method. Two types of filters are utilized, one is Chebyshev polynomial filtering, the other is rational-function filtering by solving linear equations. The former utilizes the fastest growth of the Chebyshev polynomial among same degree polynomials; the latter employs the traditional idea of shift-invert, for which we address the important issue of automatic choice of shifts and propose a practical method for solving the shifted linear equations inside the block Davidson method. Our two filters can efficiently generate high-quality basis vectors to augment the projection subspace at each Davidson iteration step, which allows a restart scheme using an active projection subspace of small dimension. This makes our algorithms memory-economical, thus practical for large PEVD/PSVD calculations. We compare our algorithms with representative methods, including ARPACK, PROPACK, the randomized SVD method, and the limited memory SVD method. Extensive numerical tests on representative datasets demonstrate that, in general, our methods have similar or faster convergence speed in terms of CPU time, while requiring much lower memory comparing with other methods. The much lower memory requirement makes our methods more practical for large-scale PEVD/PSVD computations.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Bache K, Lichman M. UCI machine learning repository. http://archive.ics.uci.edu/ml, 2013

    Google Scholar 

  2. Baer R, Head-Gordon M. Chebyshev expansion methods for electronic structure calculations on large molecular systems. J Chem Phys, 1997, 107: 10003–10013

    Article  Google Scholar 

  3. Baglama J, Calvetti D, Reichel L. IRBL: An implicitly restarted block-Lanczos method for large-scale Hermitian eigenproblems. SIAM J Sci Comput, 2003, 24: 1650–1677

    Article  MathSciNet  MATH  Google Scholar 

  4. Bai Z, Demmel J, Dongarra J, et al. Templates for the Solution of Algebraic Eigenvalue Problems: A Practical Guide. Philadelphia: SIAM, 2000

    Book  MATH  Google Scholar 

  5. Belabbas M-A, Wolfe P J. Spectral methods in machine learning and new strategies for very large datasets. Proc Natl Acad Sci USA, 2009, 106: 369–374

    Article  Google Scholar 

  6. Berns-Müller J, Graham I G, Spence A. Inexact inverse iteration for symmetric matrices. Linear Algebra Appl, 2006, 416: 389–413

    Article  MathSciNet  MATH  Google Scholar 

  7. Cai D, He X, Han J. Spectral regression: A unified subspace learning framework for content-based image retrieval. In: Proceedings of the 15th International Conference on Multimedia. New York: ACM, 2007, 403–412

    Chapter  Google Scholar 

  8. Cai J F, Candès E J, Shen Z. A singular value thresholding algorithm for matrix completion. SIAM J Optim, 2010, 20: 1956–1982

    Article  MathSciNet  MATH  Google Scholar 

  9. Calvetti D, Reichel L, Sorensen D C. An implicit restarted Lanczos method for large symmetric eigenvalue problem. Elec Trans Numer Anal, 1994, 1: 237–263

    MathSciNet  MATH  Google Scholar 

  10. Chebyshev P L. Sur les fonctions qui s’écartent peu de zéro pour certaines valeurs de la variable. Oeuvreas, 1881, 2: 335–356

    Google Scholar 

  11. Chen J, Saad Y. Lanczos vectors versus singular vectors for effective dimension reduction. IEEE Trans Knowl Data Eng, 2009, 21: 1091–1103

    Article  Google Scholar 

  12. Davidson E R. The iterative calculation of a few of the lowest eigenvalues and corresponding eigenvectors of large real-symmetric matrices. J Comput Phys, 1975, 17: 87–94

    Article  MathSciNet  MATH  Google Scholar 

  13. Deerwester S, Dumais S T, Furnas G W, et al. Indexing by latent semantic analysis. J Amer Soc Inform Sci, 1990, 41: 391–407

    Article  Google Scholar 

  14. Fan R-E, Lin C-J. Libsvm data: Classification, regression, and multi-label. http://www.csie.ntu.edu.tw/~cjlin/libs vmtools/datasets/

  15. Fokkema D R, Sleijpen G L G, van der Vorst H A. Jacobi-Davidson style QR and QZ algorithms for the reduction of matrix pencils. SIAM J Sci Comput, 1998, 20: 94–125

    Article  MathSciNet  MATH  Google Scholar 

  16. Freitag M A. Inner-outer Iterative Methods for Eigenvalue Problems—Convergence and Preconditioning. PhD Thesis. Bath: University of Bath, 2007

    Google Scholar 

  17. Gleich D F, Gray A P, Greif C, et al. An inner-outer iteration for computing PageRank. SIAM J Sci Comput, 2010, 32: 349–371

    Article  MathSciNet  MATH  Google Scholar 

  18. Golub G H, Kahan W. Calculating the singular values and pseudo-inverse of a matrix. SIAM J Numer Anal, 1965, 2: 205–224

    MathSciNet  MATH  Google Scholar 

  19. Golub G H, Van Loan C F. Matrix Computations. Baltimore: The Johns Hopkins University Press, 1996

    MATH  Google Scholar 

  20. Golub G H, Ye Q. Inexact preconditioned conjugate gradient method with inner-outer iteration. SIAM J Sci Comput, 1999, 21: 1305–1320

    Article  MathSciNet  MATH  Google Scholar 

  21. Halko N, Martinsson P-G, Tropp J A. Finding structure with randomness: Stochastic algorithms for constructing approximate matrix decompositions. SIAM Rev, 2009, 53: 217–288

    Article  MathSciNet  MATH  Google Scholar 

  22. Knyazev A V. Toward the optimal preconditioned eigensolver: Locally optimal block preconditioned conjugate gradient method. SIAM J Sci Comput, 2001, 23: 517–541

    Article  MathSciNet  MATH  Google Scholar 

  23. Ko Y. A study of term weighting schemes using class information for text classification. In: SIGIR’ 12 Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval. New York: ACM, 2012, 1029–1030

    Chapter  Google Scholar 

  24. Kohanoff J. Electronic Structure Calculations for Solids and Molecules: Theory and Computational Methods. Cambridge: Cambridge University Press, 2006

    Book  MATH  Google Scholar 

  25. Kunegis J, Lommatzsch A. Learning spectral graph transformations for link prediction. In: Proceedings of the 26th International Conference on Machine Learning. New York: ACM, 2009, 561–568

    Google Scholar 

  26. Lanczos C. An iteration method for the solution of the eigenvalue problem of linear differential and integral operators. J Res Nat Bur Standards, 1950, 45: 255–282

    Article  MathSciNet  Google Scholar 

  27. Larsen R M. Lanczos bidiagonalization with partial reorthogonalization. Technical Report DAIMI PB-357. Aarhus: Aarhus University, 1998

    Google Scholar 

  28. LeCun Y, Cortes C, Burges C J C. The mnist database of handwritten digits. http://yann.lecun.com/exdb/mnist/

  29. Lehoucq R B, Meerbergen K. Using generalized Cayley transformations within an inexact rational Krylov sequence method. SIAM J Matrix Anal Appl, 1998, 20: 131–148

    Article  MathSciNet  MATH  Google Scholar 

  30. Lehoucq R B, Sorensen D C, Yang C. ARPACK Users’ Guide: Solution of Large-Scale Eigenvalue Problems with Implicitly Restarted Arnoldi Methods. Phildelphia: SIAM, 1998

    Book  MATH  Google Scholar 

  31. Leskovec J, Krevl A. SNAP Datasets: Stanford large network dataset collection. http://snap.stanford.edu/data, 2014

    Google Scholar 

  32. Li J, White J. Low rank solution of Lyapunov equations. SIAM J Matrix Anal Appl, 2002, 24: 260–280

    Article  MathSciNet  MATH  Google Scholar 

  33. Liang Q, Ye Q. Computing singular values of large matrices with inverse free preconditioned krylov subspace method. Electron Trans Numer Anal, 2014, 42: 197–221

    MathSciNet  MATH  Google Scholar 

  34. Liu X, Wen Z, Zhang Y. Limited memory block Krylov subspace optimization for computing dominant singular value decompositions. SIAM J Sci Comput, 2013, 35: A1641–A1668

    Article  MathSciNet  MATH  Google Scholar 

  35. Ma S, Goldfarb D, Chen L. Fixed point and Bregman iterative methods for matrix rank minimization. Math Program, 2011, 128: 321–353

    Article  MathSciNet  MATH  Google Scholar 

  36. Martin R M. Electronic Structure: Basic Theory and Practical Methods. Cambridge: Cambridge University Press, 2004

    Book  MATH  Google Scholar 

  37. Mazumder R, Hastie T, Tibshirani R. Spectral regularization algorithms for learning large incomplete matrices. J Mach Learn Res, 2010, 11: 2287–2322

    MathSciNet  MATH  Google Scholar 

  38. Morgan R B, Scott D S. Generalizations of Davidson’s method for computing eigenvalues of sparse symmetric matrices. SIAM J Sci Stat Comput, 1986, 7: 817–825

    Article  MathSciNet  MATH  Google Scholar 

  39. Ng A Y, Jordan M I, Weiss Y. On spectral clustering: Analysis and an algorithm. Adv Neural Inf Process Syst, 2002, 14: 849–856

    Google Scholar 

  40. Notay Y. Combination of Jacobi-Davidson and conjugate gradients for the partial symmetric eigenproblem. Numer Linear Algebra Appl, 2002, 9: 21–44

    Article  MathSciNet  MATH  Google Scholar 

  41. Ovtchinnikov E E. Jacobi correction equation, line search, and conjugate gradients in hermitian eigenvalue computation II: Computing several extreme eigenvalues. SIAM J Numer Anal, 2008, 46: 2593–2619

    Article  MathSciNet  MATH  Google Scholar 

  42. Parlett B N. The Symmetric Eigenvalue Problem. Philadelphia: SIAM, 1998

    Book  MATH  Google Scholar 

  43. Penzl T. A cyclic low-rank Smith method for large sparse Lyapunov equations. SIAM J Sci Comput, 2000, 21: 1401–1418

    Article  MathSciNet  MATH  Google Scholar 

  44. Peters G, Wilkinson J H. Inverse iteration, ill-conditioned equations, and Newton’s method. SIAM Rev, 1979, 21: 339–360

    Article  MathSciNet  MATH  Google Scholar 

  45. Prateek J, Meka R, Dhillon I S. Guaranteed rank minimization via singular value projection. Adv Neural Inf Process Syst, 2010, 23: 937–945

    Google Scholar 

  46. Rennie J. 20 newsgroups. http://qwone.com/~jason/20Newsgroups/, 2008

    Google Scholar 

  47. Rivlin T J. Chebyshev Polynomials. New York: John Wiley & Sons, 1974

    MATH  Google Scholar 

  48. Rokhlin V, Szlam A, Tygert M. A randomized algorithm for principal component analysis. SIAM J Matrix Anal Appl, 2009, 31: 1100–1124

    Article  MathSciNet  MATH  Google Scholar 

  49. Roman J E, Campos C, Romero E, et al. SLEPc users manual. Technical Report DSIC-II/24/02-Revision 3.6. València: Universitat Politècnica de València, 2015

    Google Scholar 

  50. Ruhe A, Skoogh D. Rational Krylov algorithm for eigenvalue computation and model reduction. In: Proceedings of the 4th International Workshop on Applied Parallel Computing. Large Scale Scientific and Industrial Problems. New York: Springer-Verlag, 1998: 49150–2

    Google Scholar 

  51. Ruhe A, Wiberg T. The method of conjugate gradients used in inverse iteration. BIT, 1972, 12: 543–554

    Article  MathSciNet  MATH  Google Scholar 

  52. Saad Y. A flexible inner-outer preconditioned GMRES algorithm. SIAM J Sci Comput, 1993, 14: 461–469

    Article  MathSciNet  MATH  Google Scholar 

  53. Saad Y. Iterative Methods for Sparse Linear Systems. Philadelphia: SIAM, 2003

    Book  MATH  Google Scholar 

  54. Saad Y. Numerical Methods for Large Eigenvalue Problems. Philadelphia: SIAM, 2011

    Book  MATH  Google Scholar 

  55. Salton G, Buckley C. Term-weighting approaches in automatic text retrieval. Inform Process Manag, 1988, 24: 513–523

    Article  Google Scholar 

  56. Sankey O F, Drabold D A, Gibson A. Projected random vectors and the recursion method in the electronic-structure problem. Phys Rev B, 1994, 50: 1376–1381

    Article  Google Scholar 

  57. Simon H D. The Lanczos algorithm with partial reorthogonalization. Math Comput, 1984, 42: 115–142

    Article  MathSciNet  MATH  Google Scholar 

  58. Simoncini V, Eldén L. Inexact Rayleigh quotient-type methods for eigenvalue computations. BIT, 2002, 42: 159–182

    Article  MathSciNet  MATH  Google Scholar 

  59. Sleijpen G L G, van der Vorst H A. A Jacobi-Davidson iteration method for linear eigenvalue problems. SIAM J Matrix Anal Appl, 1996, 17: 401–425

    Article  MathSciNet  MATH  Google Scholar 

  60. Sorensen D C. Implicit application of polynomial filters in a k-step Arnoldi method. SIAM J Matrix Anal Appl, 1992, 13: 357–385

    Article  MathSciNet  MATH  Google Scholar 

  61. Sorensen D C. Implicitly restarted Arnoldi/Lanczos methods for large scale eigenvalue calculations. Technical Report TR-96-40. Houston: Rice University, 1996

    MATH  Google Scholar 

  62. Stathopoulos A, McCombs J R. Nearly optimal preconditioned methods for hermitian eigenproblems under limited memory, part II: Seeking many eigenvalues. SIAM J Sci Comput, 2007, 29: 2162–2188

    Article  MathSciNet  MATH  Google Scholar 

  63. Stathopoulos A, McCombs J R. PRIMME: PReconditioned Iterative MultiMethod Eigensolver: Methods and software description. ACM Trans Math Software, 2010, 37: 21–30

    Article  Google Scholar 

  64. Toh K C, Yun S. An accelerated proximal gradient algorithm for nuclear norm regularized linear least squares problems. Pac J Optim, 2010, 6: 615–640

    MathSciNet  MATH  Google Scholar 

  65. von Luxburg U. A tutorial on spectral clustering. Stat Comput, 2007, 17: 395–416

    Article  MathSciNet  Google Scholar 

  66. Wu K, Simon H D. A parallel Lanczos method for symmetric generalized eigenvalue problems. Report 41284. Lawrence: Lawrence Berkeley National Laboratory, 1997

    Book  MATH  Google Scholar 

  67. Wu L, Stathopoulos A. PRIMME SVDS: A preconditioned SVD solver for computing accurately singular triplets of large matrices based on the PRIMME eigensolver. ArXiv:1408.5535, 2014

    Google Scholar 

  68. Xue F, Elman H C. Fast inexact subspace iteration for generalized eigenvalue problems with spectral transformation. Linear Algebra Appl, 2011, 435: 601–622

    Article  MathSciNet  MATH  Google Scholar 

  69. Zhou Y. Studies on Jacobi-Davidson, rayleigh quotient iteration, inverse iteration generalized Davidson and Newton updates. Numer Linear Algebra Appl, 2006, 13: 621–642

    Article  MathSciNet  MATH  Google Scholar 

  70. Zhou Y. A block Chebyshev-Davidson method with inner-outer restart for large eigenvalue problems. J Comput Phys, 2010, 229: 9188–9200

    Article  MathSciNet  MATH  Google Scholar 

  71. Zhou Y. Practical acceleration for computing the HITS ExpertRank vectors. J Comput Appl Math, 2012, 236: 4398–4409

    Article  MathSciNet  MATH  Google Scholar 

  72. Zhou Y, Chelikowsky J R, Saad Y. Chebyshev-filtered subspace iteration method free of sparse diagonalization for solving the Kohn-Sham equation. J Comput Phys, 2014, 274: 770–782

    Article  Google Scholar 

  73. Zhou Y, Li R-C. Bounding the spectrum of large Hermitian matrices. Linear Algebra Appl, 2011, 435: 480–493

    Article  MathSciNet  MATH  Google Scholar 

  74. Zhou Y, Saad Y. A Chebyshev-Davidson algorithm for large symmetric eigenvalue problems. SIAM J Matrix Anal Appl, 2007, 29: 954–971

    Article  MathSciNet  MATH  Google Scholar 

  75. Zhu X, Kandola J, Lafferty J, et al. Graph kernels by spectral transforms. In: Chapelle O, Scholkopf B, Zien A, eds. Semi-Supervised Learning. Cambridge: MIT Press, 2006, 277–291

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Yunkai Zhou.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhou, Y., Wang, Z. & Zhou, A. Accelerating large partial EVD/SVD calculations by filtered block Davidson methods. Sci. China Math. 59, 1635–1662 (2016). https://doi.org/10.1007/s11425-016-0274-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11425-016-0274-0

Keywords

MSC(2010)

Navigation