Fast Algorithms for DR Approximation
In nonlinear dimensionality reduction, the kernel dimension is the square of the vector number in the data set. In many applications, the number of data vectors is very large. The spectral decomposition of a large dimensioanl kernel encounters difficulties in at least three aspects: large memory usage, high computational complexity, and computational instability. Although the kernels in some nonlinear DR methods are sparse matrices, which enable us to overcome the difficulties in memory usage and computational complexity partially, yet it is not clear if the instability issue can be settled. In this chapter, we study some fast algorithms that avoid the spectral decomposition of large dimensional kernels in DR processing, dramatically reducing memory usage and computational complexity, as well as increasing numerical stability. In Section 15.1, we introduce the concepts of rank revealings. In Section 15.2, we present the randomized low rank approximation algorithms. In Section 15.3, greedy lank-revealing algorithms (GAT) and randomized anisotropic transformation algorithms (RAT), which approximate leading eigenvalues and eigenvectors of DR kernels, are introduced. Numerical experiments are shown in Section 15.4 to illustrate the validity of these algorithms. The justification of RAT algorithms is included in Section 15.5.
KeywordsFast Algorithm Random Matrix Random Projection Nonlinear Dimensionality Reduction Swiss Roll
Unable to display preview. Download preview PDF.
- Stewart, G.W.: Matrix Algorithms Volume I: Basic Decompositions. SIAM, Philadelphia (1998).Google Scholar
- Belabbas, M.A., Wolfe, P.J.: Fast low-rank approximation for covariance matrices. In: Proceedings of the 2nd IEEE International Workshop on Computational Advances in Multi-Sensor Adaptive Processing (2007).Google Scholar
- Belabbas, M.A., Wolfe, P.J.: On sparse representations of linear operators and the approximation of matrix products. In: Proceedings of the 42nd Annual Conference on Information Sciences and Systems, pp. 258–263 (2008).Google Scholar
- Fowlkes, C., Belongie, S., Chung, F., Malik, J.: Spectral grouping using the Nyström method. IEEE Trans. Patt. Anal. Mach. Intell. pp. 214–225 (2004).Google Scholar
- Parker, P., Wolfe, P.J., Tarokh, V.: A signal processing application of randomized low-rank approximations. In: IEEE Worksh. Statist. Signal Process., pp. 345–350 (2005).Google Scholar
- Williams, C.K.I., Seeger, M.: Using the Nyström method to speed up kernel machines. In: Neural Information Processing Systems, pp. 682–688 (2000).Google Scholar
- Martinsson, P.G., Rokhlin, V., Tygert, M.: A randomized algorithm for the approximation of matrices. Tech. Rep. 1361, Dept. of Computer Science, Yale University (2006).Google Scholar
- Martinsson, P.G., Rokhlin, V., Tygert, M.: On interpolation and integration in finite-dimensional spaces of bounded functions. Comm. Appl. Math. Comput. Sci. pp. 133–142 (2006).Google Scholar
- Chui, C., Wang, J.: Dimensionality reduction of hyper-spectral imagery data for feature classification. In: W. Freeden, Z. Nashed, T. Sonar (eds.) Handbook of Geomathematics. Springer, Berlin (2010).Google Scholar
- Xiao, L., Sun, J., Boyd, S.P.: A duality view of spectral methods for dimensionality reduction. In: W.W. Cohen, A. Moore (eds.) Machine Learning: Proceedings of the Twenty-Third International Conference, ACM International Conference Proceeding Series, vol. 148, pp. 1041–1048. ICML, Pittsburgh, Pennsylvania, USA (2006).Google Scholar