Advertisement

Abstract

Principal component analysis (PCA) is a very important linear method for dimensionality reduction. It measures data distortion globally by the Frobenius norm of the matrix of data difference. The reduced data of PCA consists of several leading eigenvectors of the covariance matrix of the data set. Hence, PCA may not preserve the local separation of the original data. To respect local properties of data in dimensionality reduction (DR), we employ Lipschitz embedding. Random projection is a powerful method to construct Lipschitz mappings to realize dimensionality reduction with a high probability. Random projection does not introduce a significant distortion when the dimension and cardinality of data both are large. It randomly projects the original high-dimensional data into a lower-dimensional subspace. Because the projection costs linear computational time, the method is computationally efficient, yet produces sufficient accuracy with a high probability. In Section 7.1, we give a review of Lipschitz embedding. In Section 7.2, we introduce random matrices and random projection algorithms. In Section 7.3, the justification of the validity of random projection is presented in detail. Particularly, Johnson and Lindenstrauss Lemma will be proved in this section. The applications of random projection are given in Section 7.4.

Keywords

Principal Component Analysis Dimensionality Reduction Face Recognition Singular Value Decomposition Random Matrice 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. [1]
    Johnson, W.B., Lindenstrauss, J.: Extensions of Lipschitz mappings into a Hilbert space. AMS Contemp. Math. 26, 189–206 (1984).MathSciNetzbMATHCrossRefGoogle Scholar
  2. [2]
    Indyk, P., Motwani, R.: Approximate nearest neighbors: towards removing the curse of dimensionality. In: Proc. 30th Symp. on Theory of Computing, vol. 13, pp. 604–613 (1998).Google Scholar
  3. [3]
    Achlioptas, D.: Database-friendly random projections. Proc. 20th PODS pp. 274–281 (2001).Google Scholar
  4. [4]
    Arriaga, R.I., Vempala, S.: An algorithmic theory of learning: Robust concepts and random projection. Machine Learning 63(2), 161–182 (2006). URL http://dx.doi.org/10.1007/s10994-006-6265-7. Accessed 1 December 2011.zbMATHCrossRefGoogle Scholar
  5. [5]
    Klivans, A.R., Servedio, R.A.: Learning intersections of halfspaces with a margin. Proc. 17th COLT 100, 348–362 (2004).MathSciNetGoogle Scholar
  6. [6]
    Szlam, A.: Non-stationary analysis on datasets and applications. Ph.D. thesis, Yale University (2006).Google Scholar
  7. [7]
    Frankl, P., Maehara, H.: The Johnson-Lindenstrauss lemma and the sphericity of some graphs. Journal of Combinatorial Theory 44, 355–362 (1988).MathSciNetzbMATHCrossRefGoogle Scholar
  8. [8]
    Dasgupta, S., Gupta, A.: An elementary proof of a theorem of Johnson and Lindenstrauss. Random Structures and Algorithms 22(1), 60–65 (1999).MathSciNetCrossRefGoogle Scholar
  9. [9]
    Goel, N., Bebis, G., Nefian, A.: Face recognition experiments with random projection (2004).Google Scholar
  10. [10]
    Dasgupta, S.: Experiments with random projection. Proc. Uncertainty in Artificial Intelligence (2000).Google Scholar
  11. [11]
    Martinez, A.M., Benavente, R.: Cvc technical report 24. Ph.D. Thesis, University of Cambridge and ATT Laboratories Cambridge (1998).Google Scholar
  12. [12]
    Bingham, E., Mannila, H.: Random projection in dimensionality reduction: applications to image and text data. In: Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining, KDD 01, pp. 245–250. ACM, New York, NY, USA (2001).CrossRefGoogle Scholar
  13. [13]
    Kaski, S.: Dimensionality reduction by random mapping:fast similarity computation for clustering. In: Proc. Int. Joint Conf. on Neural Networks, vol. 1, pp. 413–418 (1998).Google Scholar
  14. [14]
    Lin, J., Gunopulos, D.: Dimensionality reduction by random projection and latent semantic indexing. In: Proc. of SDM (2003).Google Scholar
  15. [15]
    Sulić, V., J. Perš, Kristan, M., Kovačič, S.: Dimensionality reduction for distributed vision systems using random projection. In: International Conference on Pattern Recognition (2010).Google Scholar
  16. [16]
    Berry, M.W., Pulatova, S.A., Stewart, G.W.: Algorithm 844: computing sparse reduced-rank approximations to sparse matrices. ACM Trans Math Softw 31(2), 252–269 (2005).MathSciNetzbMATHCrossRefGoogle Scholar
  17. [17]
    Chan, T.F., Hansen, P.C.: Some applications of the rank revealing QR factorization. SIAM J. on Sci. Statist. Comput. 13, 727–741 (1992).MathSciNetzbMATHCrossRefGoogle Scholar
  18. [18]
    Chen, Z., Dongarra, J.J.: Condition numbers of Gaussian random matrices. SIAM J. on Matrix Anal. Appl. 27, 603–620 (2005).MathSciNetCrossRefGoogle Scholar
  19. [19]
    Goldstine, H.H., von Neumann, J.: Numerical inverting of matrices of high order II. Amer. Math. Soc. Proc. 2, 188–202 (1951).zbMATHCrossRefGoogle Scholar
  20. [20]
    Goreinov, S.A., Tyrtyshnikov, E.E., Zamarashkin, N.L.: A theory of pseudo-skeleton approximations. Linear Algebra and Its Applications 261, 1–21 (1997).MathSciNetzbMATHCrossRefGoogle Scholar
  21. [21]
    Gu, M., Eisenstat, S.C.: Efficient algorithms for computing a strong rank-revealing QR factorization. SIAM J. on Sci. Comput. 17, 848–869 (1996).MathSciNetzbMATHCrossRefGoogle Scholar
  22. [22]
    Martinsson, P.G., Rokhlin, V., Tygert, M.: A randomized algorithm for the approximation of matrices. Tech. Rep. 1361, Dept. of Computer Science, Yale University (2006).Google Scholar
  23. [23]
    Tyrtyshnikov, E.: Matrix bruhat decompositions with a remark on the QR (GR) algorithm. Linear Algebra Appl. 250, 61–68 (1997).MathSciNetzbMATHCrossRefGoogle Scholar
  24. [24]
    Tyrtyshnikov, E., Zamarashkin, N.: Thin structure of eigenvalue clusters for non-Hermitian Toeplitz matrices. Linear Algebra Appl. 292, 297–310 (1999).MathSciNetzbMATHCrossRefGoogle Scholar
  25. [25]
    Woolfe, F., Liberty, E., Rokhlin, V., Tygert, M.: A fast randomized algorithm for the approximation of matrices. Appl. Comput. Harmon. Anal. 25(3), 335–366 (2008).MathSciNetzbMATHCrossRefGoogle Scholar
  26. [26]
    Zamarashkin, N., Tyrtyshnikov, E.: Eigenvalue estimates for Hankel matrices. Sbornik: Mathematics 192, 59–72 (2001).MathSciNetCrossRefGoogle Scholar

Copyright information

© Higher Education Press, Beijing and Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Jianzhong Wang
    • 1
  1. 1.Department of Mathematics and StatisticsSam Houston State UniversityHuntsvilleUSA

Personalised recommendations