Skip to main content
SpringerLink
Log in

On general matrix exponential discriminant analysis methods for high dimensionality reduction

  • Published:
Calcolo Aims and scope Submit manuscript

Abstract

Recently, some matrix exponential-based discriminant analysis methods were proposed for high dimensionality reduction. It has been shown that they often have more discriminant power than the corresponding discriminant analysis methods. However, one has to solve some large-scale matrix exponential eigenvalue problems which constitutes the bottleneck in this type of methods. The main contribution of this paper is twofold: First, we propose a framework of fast implementation on general matrix exponential-based discriminant analysis methods. The key is to equivalently transform large-scale matrix computation problems into much smaller ones. On the other hand, it was mentioned in Wang et al. (IEEE Trans Image Process 23:920–930, 2014) that the exponential model is more reliable than the original one and suppresses the sensitivity to pertubations. However, the interpretation is only heuristic, and to the best of our knowledge, there is no theoretical justification for reliability and stability of the matrix discriminant analysis methods. To fill in this gap, the second contribution of our work is to provide stability analysis for the fast exponential discriminant analysis method from a theoretical point of view. Numerical experiments illustrate the numerical behavior of the proposed algorithm, and demonstrate that our algorithm is more stable than many state-of-the-art algorithms for high dimensionality reduction.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

Notes

  1. http://cvc.yale.edu/projects/yalefaces/yalefaces.html.

  2. http://www.ri.cmu.edu/projects/project_418.html.

  3. http://rvl1.ecn.purdue.edu/aleix/aleix_face_DB.html.

  4. http://www.nist.gov/itl/iad/ig/colorferet.cfm.

References

  1. Ahmed, N.: Exponential discriminant regularization using nonnegative constraint and image descriptor. In: IEEE 9th international conference on emerging technologies, pp. 1–6 (2013)

  2. Belhumeur, P., Hespanha, J., Kriegman, D.: Eigenfaces vs Fisherface: recognition using class-specific linear projection. IEEE Trans. Pattern Anal. Mach. Intell. 19, 711–720 (1997)

    Article  Google Scholar 

  3. Chang, X., Paige, C.C., Stewart, G.W.: Preturbation analysis for the QR factorization. SIAM J. Matrix Anal. Appl. 18, 775–791 (1997)

    Article  MathSciNet  MATH  Google Scholar 

  4. Chen, H., Chang, H., Liu, T.: Local discriminant embedding and its variants. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 846–853 (2005)

  5. Chen, L., Liao, H., Ko, M., Lin, J., Yu, G.: A new LDA-based face recognition system which can solve the small sample size problem. Pattern Recognit. 33, 1713–1726 (2000)

    Article  Google Scholar 

  6. Chu, D., Thye, G.: A new and fast implementation for null space based linear discriminant analysis. Pattern Recognit. 43, 1373–1379 (2010)

    Article  MATH  Google Scholar 

  7. Cover, T., Hart, P.: Nearest neighbor pattern classification. IEEE Trans. Inf. Theory 13, 21–27 (1967)

    Article  MATH  Google Scholar 

  8. Dornaika, F., Bosaghzadeh, A.: Exponential local discriminant embedding and its application to face recognition. IEEE Trans. Syst. Man Cybern. Part B Cybern. 43, 921–934 (2013)

    Google Scholar 

  9. Dornaika, F., El Traboulsi, Y.: Matrix exponential based semi-supervised discriminant embedding for image classification. Pattern Recognit. 61, 92–103 (2017)

    Article  MATH  Google Scholar 

  10. Duda, R., Hart, P., Stock, D.: Pattern Classification, 2nd edn. Wiley, New York (2004)

    Google Scholar 

  11. Friedman, J.: Regularized discriminant analysis. J. Am. Stat. Assoc. 84, 165–175 (1989)

    Article  MathSciNet  Google Scholar 

  12. Fukunaga, K.: Introduction to Statistical Pattern Recognition, 2nd edn. Elsevier Academic Press, London (1999)

    MATH  Google Scholar 

  13. Golub, G.H., Van Loan, C.F.: Matrix Computations, 4th edn. The Johns Hopkins university Press, Baltimore (2013)

    MATH  Google Scholar 

  14. Higham, N.J.: Functions of Matrices: Theory and Computation. SIAM, Philadelphia (2008)

    Book  MATH  Google Scholar 

  15. He, J., Ding, L., Cui, M., Hu, Q.: Marginal Fisher analysis based on matrix exponential transformation. Chin. J. Comput. 10, 2196–2205 (2014). (in Chinese)

    MathSciNet  Google Scholar 

  16. He, X., Niyogi, P.: Locality preserving projections. In: Thrun, S., Saul, L., Schölkopf, B. (eds.) Advances in Neural Information Processing Systems, vol. 16, pp. 153–160. MIT Press, Cambridge (2004)

    Google Scholar 

  17. He, X., Yan, S.C., Hu, Y., Niyogi, P., Zhang, H.J.: Face recognition using laplacianfaces. IEEE Trans. Pattern Anal. Mach. Intell. 27, 328–340 (2005)

    Article  Google Scholar 

  18. Howland, P., Park, H.: Generalizing discriminant analysis using the generalized singular value decomposition. IEEE Trans. Pattern Anal. Mach. Intell. 26, 995–1006 (2004)

    Article  Google Scholar 

  19. Jolliffe, I.: Principal Component Analysis. Springer, New York (1986)

    Book  MATH  Google Scholar 

  20. Kokiopoulou, E., Chen, J., Saad, Y.: Trace optimization and eigenproblems in dimension reduction methods. Numer. Linear Algebra Appl. 18, 565–602 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  21. Kokiopoulou, E., Saad, Y.: Orthogonal neighborhood preserving projections. In: Houston, T.X., Han, J., et al. (eds.) IEEE 5th International Conference on Data Mining (ICDM05), pp. 234–241. IEEE, New York (2005)

    Chapter  Google Scholar 

  22. Kokiopoulou, E., Saad, Y.: Orthogonal neighborhood preserving projections: a projection-based dimensionality reduction technique. IEEE Trans. Pattern Anal. Mach. Intell. 29, 2143–2156 (2007)

    Article  Google Scholar 

  23. Kramer, R., Young, A., Burton, A.: Understanding face familiarity. Cognition 172, 46–58 (2018)

    Article  Google Scholar 

  24. Mohri, M., Rostamizadeh, A., Talwalkar, A.: Foundations of Machine Learning. The MIT Press, Cambridge (2012)

    MATH  Google Scholar 

  25. Park, C., Park, H.: A comparision of generalized linear discriminant analysis algorithms. Pattern Recognit. 41, 1083–1097 (2008)

    Article  MATH  Google Scholar 

  26. Qiao, L., Chen, S., Tan, X.: Sparsity preserving projections with applications to face recognition. Pattern Recognit. 43, 331–341 (2010)

    Article  MATH  Google Scholar 

  27. Seghouane, A., Shokouhi, N., Koch, I.: Sparse principal component analysis with preserved sparsity pattern. IEEE Trans. Image Process. 28, 3274–3285 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  28. Shi, W., Wu, G.: Why PCA plus graph embedding methods can be unstable for extracting classification features? (submitted for publication)

  29. Stewart, G.W.: Perturbation bounds for the definite generalized eigenvalue problem. Linear Algebra Appl. 23, 69–85 (1979)

    Article  MathSciNet  MATH  Google Scholar 

  30. Sun, J.: The perturbation bounds for eigenspaces of a definite matrix-pair. Numer. Math. 41, 321–343 (1983)

    Article  MathSciNet  MATH  Google Scholar 

  31. Tenenbaum, J., de Silva, V., Langford, J.: A global geometric framework for nonlinear dimensionality reduction. Science 290(5500), 2319–2323 (2000)

    Article  Google Scholar 

  32. Torgerson, W.: Multidimensional scaling: I. Theory and method. Psychometrika 17, 401–419 (1952)

    Article  MathSciNet  MATH  Google Scholar 

  33. Turk, M., Pentland, A.: Eigenfaces for recognition. J. Cognit. Neurosci. 3, 71–86 (1991)

    Article  Google Scholar 

  34. Van Loan, C.F.: The sensitivity of the matrix exponential. SIAM J. Numer. Anal. 14, 971–981 (1977)

    Article  MathSciNet  MATH  Google Scholar 

  35. Wang, J.: Geometric Structure of High-Dimensional Data and Dimensionality Reduction. Higher Education Press, Beijing (2011)

    Book  Google Scholar 

  36. Wang, S., Chen, H., Peng, X., Zhou, C.: Exponential locality preserving projections for small sample size problem. Neurocomputing 74, 36–54 (2011)

    Google Scholar 

  37. Wang, S., Yan, S., Yang, J., et al.: A general exponential framework for dimensionality reduction. IEEE Trans. Image Process. 23, 920–930 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  38. Webb, A.: Statistical Pattern Recognition, 2nd edn. Wiley, Hoboken (2002)

    Book  MATH  Google Scholar 

  39. Wu, G., Feng, T., Zhang, L., Yang, M.: Inexact implementation using Krylov subspace methods for large scale exponential discriminant analysis with applications to high dimensionality reduction problems. Pattern Recogniti. 66, 328–341 (2017)

    Article  Google Scholar 

  40. Wu, G., Xu, W., Leng, H.: Inexact and incremental bilinear Lanczos components algorithms for high dimensionality reduction and image reconstruction. Pattern Recognit. 48, 244–263 (2015)

    Article  MATH  Google Scholar 

  41. Yan, L., Pan, J.: Two-dimensional exponential discriminant analysis and its application to face recognition. In: International Conference on Computational Aspects of Social Networks (CASoN), pp. 528–531 (2010)

  42. Yan, S., Xu, D., Zhang, B., Zhang, H., Yang, Q., Lin, S.: Graph embedding and extensions: a general framework for dimensionality reduction. IEEE Trans. Pattern Anal. Mach. Intell. 29, 40–51 (2007)

    Article  Google Scholar 

  43. Yang, J., Zhang, D., Yang, J., Niu, B.: Globally maximizing, locally minimizing: unsupervised discriminant projection with applications to face and palm biometrics. IEEE Trans. Pattern Anal. Mach. Intell. 29, 650–664 (2007)

    Article  Google Scholar 

  44. Ye, J., Li, Q.: LDA/QR: an efficient and effective dimension reduction algorithm and its theoretical foundation. Pattern Recognit. 37, 851–854 (2004)

    Article  Google Scholar 

  45. Yuan, S., Mao, X.: Exponential elastic preserving projections for facial expression recognition. Neurocomputing 275, 711–724 (2018)

    Article  Google Scholar 

  46. Zhang, T., Fang, B., Tang, Y., Shang, Z., Xu, B.: Generalized discriminant analysis: a matrix exponential approach. IEEE Trans. Syst. Man Cybern. Part B Cybern. 40, 186–197 (2010)

    Article  Google Scholar 

Download references

Acknowledgements

We would like to express our sincere thanks to the anonymous referees and our editor for insightful comments and suggestions that greatly improved the representation of this paper.

Author information

Authors and Affiliations

  1. School of Mathematics, China University of Mining and Technology, Xuzhou, 221116, Jiangsu, People’s Republic of China

    Wenya Shi, Youwei Luo & Gang Wu

Authors
  1. Wenya Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Youwei Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gang Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gang Wu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This work is supported by the Natural Science Foundation of Jiangsu Province under grant BK20171185, and the Fundamental Research Funds for the Central Universities of China under grant 2019XKQYMS89.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shi, W., Luo, Y. & Wu, G. On general matrix exponential discriminant analysis methods for high dimensionality reduction. Calcolo 57, 18 (2020). https://doi.org/10.1007/s10092-020-00366-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10092-020-00366-6

Keywords

Mathematics Subject Classification

Search

Navigation