Abstract
Eigenvector computation, e.g., k-SVD for finding top-k singular subspaces, is often of central importance to many scientific and engineering tasks. There has been resurgent interest recently in analyzing relevant methods in terms of singular value gap dependence. Particularly, when the gap vanishes, the convergence of k-SVD is considered to be capped by a gap-free sub-linear rate. We argue in this work both theoretically and empirically that this is not necessarily the case, refreshing our understanding on this significant problem. Specifically, we leverage the recently proposed structured gap in a careful analysis to establish a unified linear convergence of k-SVD to one of the ground-truth solutions, regardless of what target matrix and how large target rank k are given. Theoretical results are evaluated and verified by experiments on synthetic or real data.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Notes
They use a different term “generalized gap-dependent analysis”.
The k-th gap freeness, which does not depend on the k-th gap but still depends on a certain gap, is different from the gap freeness which does not depend on any gap.
Minor fluctuations occur in the convergence curves under setting II with \(\varDelta ''=0.30\). We believe that they are due to the relatively large step size compared to the small errors at the scale of \(10^{-15}\). Nevertheless, the overall trend of the convergence curves still concurs with our theoretical results.
References
Absil PA, Mahony R, Sepulchre R (2008) Optimization algorithms on matrix manifolds. Princeton University Press, Princeton
Allen-Zhu Z, Li Y (2016) Even faster SVD decomposition yet without agonizing pain. In: Advances in Neural Information Processing Systems, pp 974–982
Balsubramani A, Dasgupta S, Freund Y (2013) The fast convergence of incremental PCA. In: Advances in Neural Information Processing Systems 26, December 5-8, 2013, Lake Tahoe, Nevada, United States., 3174–3182
Dai W, Xu T, Wang W (2012) Simultaneous codeword optimization (simco) for dictionary update and learning. IEEE Trans Signal Process 60(12):6340–6353
Edelman A, Arias TA, Smith ST (1998) The geometry of algorithms with orthogonality constraints. SIAM J Matrix Anal Appl 20(2):303–353
Faddeeva V (1959) Computational methods of linear algebra. Dover books on advanced mathematics. Dover Publications. https://books.google.com.sa/books?id=WORQAAAAMAAJ
Fan J, Sun Q, Zhou WX, Zhu Z (2018) Principal component analysis for big data. arXiv preprint arXiv:1801.01602
Garber D, Hazan E, Jin C, Kakade SM, Musco C, Netrapalli P, Sidford A (2016) Faster eigenvector computation via shift-and-invert preconditioning. In: International Conference on Machine Learning, 2626–2634
Golub G, Van Loan C (2013) Matrix Computations. Johns Hopkins Studies in the Mathematical Sciences. Johns Hopkins University Press, Baltimore
Halko N, Martinsson P, Tropp JA (2011) Finding structure with randomness: probabilistic algorithms for constructing approximate matrix decompositions. SIAM Rev 53(2):217–288. https://doi.org/10.1137/090771806
Hardt M, Price E (2014) The noisy power method: A meta algorithm with applications. In: Advances in Neural Information Processing Systems 27: 28th Annual Conference on Neural Information Processing Systems 2014, December 8–13, 2014 Montreal. Quebec, Canada, pp 2861–2869
Hardt M, Price E (2014) The noisy power method: A meta algorithm with applications. In: Advances in neural information processing systems, pp 2861–2869
Helmke U, Moore JB (2012) Optimization and dynamical systems. Springer, Berlin
Mohar B, Poljak S (1993) Combinatorial and graph-theoretical problems in linear algebra. Eigenvalues in combinatorial optimization. Springer, New York, pp 107–151
Musco C, Musco C, (2015) Randomized block krylov methods for stronger and faster approximate singular value decomposition. In: Advances in Neural Information Processing Systems 28: 29th Annual Conference on Neural Information Processing Systems 2015, December, 7–12, 2015 Montreal. Quebec, Canada, pp 1396–1404
Ng AY, Jordan MI, Weiss Y (2001) On spectral clustering: Analysis and an algorithm. In: Advances in Neural Information Processing Systems 14 [Neural Information Processing Systems: Natural and Synthetic, NIPS 2001, December 3-8, 2001, Vancouver, British Columbia, Canada], pp. 849–856
Pitaval R, Dai W, Tirkkonen O (2015) Convergence of gradient descent for low-rank matrix approximation. IEEE Trans Information Theory 61(8):4451–4457. https://doi.org/10.1109/TIT.2015.2448695
Rudelson M, Vershynin, R (2010) Non-asymptotic theory of random matrices: extreme singular values. arXiv preprint arXiv:1003.2990
Saad Y, Chelikowsky JR, Shontz SM (2010) Numerical methods for electronic structure calculations of materials. SIAM Rev 52(1):3–54. https://doi.org/10.1137/060651653
Shamir O (2015) A stochastic PCA and SVD algorithm with an exponential convergence rate. In: Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, pp. 144–152
Shamir O (2016) Convergence of stochastic gradient descent for PCA. In: ICML, pp. 257–265
Shamir O (2016) Fast stochastic algorithms for SVD and PCA: convergence properties and convexity. In: International Conference on Machine Learning, pp. 248–256
Torbjorn Ringertz U (1997) Eigenvalues in optimum structural design. Inst Math Appl 92:135
Wang J, Wang W, Garber D, Srebro N (2018) Efficient coordinate-wise leading eigenvector computation. In: Algorithmic Learning Theory, ALT 2018, 7-9 April 2018, Lanzarote, Canary Islands, Spain, pp 806–820. http://proceedings.mlr.press/v83/wang18a.html
Wen Z, Yin W (2013) A feasible method for optimization with orthogonality constraints. Math Program 142(1–2):397–434
Woodruff DP (2014) Sketching as a tool for numerical linear algebra. Found Trends Theor Comput Sci 10(1–2):1–157. https://doi.org/10.1561/0400000060
Xu P, He BD, Sa CD, Mitliagkas I, Ré C (2018) Accelerated stochastic power iteration. In: International Conference on Artificial Intelligence and Statistics, AISTATS 2018, 9-11 April 2018, Playa Blanca, Lanzarote, Canary Islands, Spain, pp. 58–67. http://proceedings.mlr.press/v84/xu18a.html
Xu Z, Gao X (2018) On truly block eigensolvers via riemannian optimization. International Conference on Artificial Intelligence and Statistics, AISTATS 2018, 9–11 April 2018. Playa Blanca, Lanzarote, Canary Islands, Spain, pp 168–177
Ye K, Lim L (2016) Schubert varieties and distances between subspaces of different dimensions. SIAM J Matrix Anal Appl 37(3):1176–1197. https://doi.org/10.1137/15M1054201
Acknowledgements
We thank the reviewers for their comments, which helped improve this paper considerably. The work is partially supported by a research project jointly funded by Hutchinson Research & Innovation Singapore Pte. Ltd. and Energy Research Institute @ NTU (ERI@N).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Xu, Z., Ke, Y., Cao, X. et al. A unified linear convergence analysis of k-SVD. Memetic Comp. 12, 343–353 (2020). https://doi.org/10.1007/s12293-020-00315-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12293-020-00315-4