Skip to main content
SpringerLink
Log in

A New and Fast Implementation of Orthogonal LDA Algorithm and Its Incremental Extension

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

Linear discriminant analysis (LDA) is one of the most popular dimension reduction methods and has been widely used in many applications. In the last decades many LDA-based dimension reduction algorithms have been reported. Among these methods, orthogonal LDA (OLDA) is a famous one and several different implementations of OLDA have been proposed. In this paper, we propose a new and fast implementation of OLDA. Compared with the other OLDA implementations, our proposed implementation of OLDA is the fastest one when the dimensionality d is larger than the sample size n. Then, based on our proposed implementation of OLDA, we present an incremental OLDA algorithm which can accurately update the projection matrix of OLDA when new samples are added into the training set. The effectiveness of our proposed new OLDA algorithm and its incremental version are demonstrated by some real-world data sets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Fukunaga K (1990) Introduction to statistical pattern recognition, 2nd edn. Academic Press, Boston

    MATH  Google Scholar 

  2. Duda RO, Hart PE, Stork DG (2000) Pattern classification, 2nd edn. John Wiley & Sons, New York

    MATH  Google Scholar 

  3. Jin Z, Yang JY, Hu ZS, Lou Z (2001) Face recognition based on the uncorrelated discriminant transformation. Pattern Recognit 34(7):1405–1416

    Article  MATH  Google Scholar 

  4. Belhumeour PN, Hespanha JP, Kriegman DJ (1997) Eigenfaces vs. Fisherfaces: recognition using class specific linear projection. IEEE Trans Pattern Anal Mach Intell 19(7):711–720

    Article  Google Scholar 

  5. Lu J, Plataniotis KN, Venetsanopoulos AN (2003) Face recognition using LDA-based algorithms. IEEE Trans Neural Netw 14(1):195–200

    Article  Google Scholar 

  6. Bishop CM (2006) Pattern recognition and machine learning. Springer, New York

    MATH  Google Scholar 

  7. Howland P, Jeon M, Park H (2003) Structure preserving dimension reduction for clustered text data based on the generalized singular value decomposition. SIAM J Matrix Anal Appl 25(1):165–179

    Article  MathSciNet  MATH  Google Scholar 

  8. Berry MW, Dumais ST, O’Brie GW (1995) Using linear algebra for intelligent information retrieval. SIAM Rev 37:573–595

    Article  MathSciNet  MATH  Google Scholar 

  9. Cai D, He X, Han J (2008) SRDA: an efficient algorithm for large scale discriminant analysis. IEEE Trans Knowl Data Eng 20(1):1–12

    Article  Google Scholar 

  10. Dudoit S, Fridlyand J, Speed TP (2002) Comparison of discrimination methods for the classfication of tumors using gene expression data. J Am Stat Assoc 97(457):77–87

    Article  MathSciNet  MATH  Google Scholar 

  11. Baldi P, Hatfield GW (2002) DNA microarrays and gene expression: from experiments to data analysis and modeling. Cambridge University Press, Cambridge

    Book  Google Scholar 

  12. Howland P, Wang J, Park H (2006) Solving the small sample size problem in face recognition using generalized discriminant analysis. Pattern Recognit 39(2):227–287

    Google Scholar 

  13. Krzanowski WJ, Jonathan P, McCarthy WV, Thomas MR (1995) Discriminant analysis with singular covariance matrices: methods and applications to spectroscopic data. Appl Stat 44(1):101–115

    Article  MATH  Google Scholar 

  14. Chu D, Thye GS (2010) A new and fast implementation for null space based linear discriminant analysis. Pattern Recognit 43(4):1373–1379

    Article  MATH  Google Scholar 

  15. Chen LF, Liao HYM, Ko MT, Yu GJ (2000) A new LDA-based face recognition system which can solve the small sample size problem. Pattern Recognit 33(10):1713–1726

    Article  Google Scholar 

  16. Huang R, Liu Q, Lu H, Ma S (2002) Solving the small size problem of LDA. In: Proceedings of 16th Internatinal Conference. Pattern Recognition, IEEE Computer Society, Quebec, Canada, pp 29–32

  17. Lu G-F, Wang Y (2012) Feature extraction using a fast null space based linear discriminant analysis algorithm. Inf Sci 193(6):72–80

    Article  MathSciNet  Google Scholar 

  18. Sharma A, Paliwal KK (2012) A new perspective to null linear discriminant analysis method and its fast implementation using random matrix multiplication with scatter matrices. Pattern Recognit 45(6):2205–2213

    Article  MATH  Google Scholar 

  19. Howland P, Park H (2004) Generalizing discriminant analysis using the generalized singular value decomposition. IEEE Trans Pattern Anal Mach Intell 26(8):995–1006

    Article  Google Scholar 

  20. Ye J, Janardan R, Park CH, Park H (2004) An optimization criterion for generalized discriminant analysis on undersampled problems. IEEE Trans Pattern Anal Mach Intell 26(8):982–994

    Article  Google Scholar 

  21. Ye J (2005) Characterization of a family of algorithms for generalized discriminant analysis on undersampled problems. J Mach Learn Res 6:483–502

    MathSciNet  MATH  Google Scholar 

  22. Ye J, Janardan R, Li Q, Park H (2006) Feature reduction via generalized uncorrelated linear discriminant analysis. IEEE Trans Knowl Data Eng 18(10):1312–1322

    Article  Google Scholar 

  23. Jin Z, Yang JY, Tang ZM, Hu ZS (2001) A theorem on the uncorrelated optimal discriminant vectors. Pattern Recognit 34(10):2041–2047

    Article  MATH  Google Scholar 

  24. Ye J, Xiong T (2006) Computational and theoretical analysis of null space and orthogonal linear discriminant analysis. J Mach Learn Res 7:1183–1204

    MathSciNet  MATH  Google Scholar 

  25. Ching W-K, Chu D, Liao L-Z, Wang X (2012) Regularized orthogonal linear discriminant analysis. Pattern Recognit 45:2719–2732

    Article  MATH  Google Scholar 

  26. Chu D, Goh ST (2010) A new and fast orthogonal linear discriminant analysis on undersampled problems. SIAM J Sci Comput 32(4):2274–2297

    Article  MathSciNet  MATH  Google Scholar 

  27. Park H, Drake BL, Lee S, Park CH (2007) Fast linear discriminant analysis using QR decomposition and regularization. Department of Computer Science and Engineering, University of Minnesota, Minneaplis

  28. Yang J, Yang JY (2003) Why can LDA be performed in PCA transformed space? Pattern Recognit 36(3):563–566

    Article  Google Scholar 

  29. Lu G-F, Zou J, Wang Y (2012) Incremental complete LDA for face recognition. Pattern Recognit 45(7):2510–2521

    Article  MATH  Google Scholar 

  30. Lu G-F, Zou J, Wang Y (2012) Incremental learning of complete linear discriminant analysis for face recognition. Knowl Based Syst 31(7):19–27

    Article  Google Scholar 

  31. Wang X, Tang X (2004) Dual-space linear discriminant analysis for face recognition, In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’04), Washington, USA, pp 564–569

  32. Zheng W, Tang X (2009) Fast algorithm for updating the discriminnat vectors of dual-space LDA. IEEE Trans Inf Forensics Secur 4(3):418–427

    Article  MathSciNet  Google Scholar 

  33. Hamsici OC, Martinez AM (2008) Bayes optimality in linear discriminant analysis. IEEE Trans Pattern Anal Mach Intell 30(4):647–657

    Article  Google Scholar 

  34. Ye J (2007) Least squares linear discriminant analysis, In: The twenty-fourth international conference on machine learning (ICML 2007), pp 1087–1093

  35. Ye J, Li Q, Xiong H, Park H (2005) IDR/QR: an incremental dimension reduction algorithm via QR decomposition. IEEE Trans Knowl Data Eng 17(9):1208–1222

    Article  Google Scholar 

  36. Ye J, Li Q, Xiong H, Park H, Janardan R, Kumar V (2004) IDR/QR: an incremental dimension reduction algorithm via QR decomposition. In: ACM SIGKDD Proceedings, pp 364–373

  37. Lu G-F, Zou J, Wang Y (2012) Incremental learning of discriminant common vectors for feature extraction. Appl Math Comput 218(22):11269–11278

    MATH  Google Scholar 

  38. Yan J, Zhang B, Yan S, Yang Q, Li H, Chen Z, Xi W, Fan W, Ma W-Y, Cheng Q (2004) IMMC: Incremental maximum margin criterion. In: Proceedings of International conference knowledge discovery and data mining (KDD’04), ACM, Seattle, Washington, USA, pp 1–6

  39. Yan J, Zhang B, Yan S, Liu N, Yang Q, Cheng Q, Li H, Chen Z, Ma W-Y (2006) A scalable supervised algorithm for dimensionality reduction on streaming data. Inf Sci 176(14):2042–2065

    Article  Google Scholar 

  40. Pang S, Ozawa S, Kasabov N (2005) Incremental linear discriminant analysis for classification of data streams. IEEE Trans Syst Man Cybern Part B 35(5):905–914

    Article  Google Scholar 

  41. Zhao H, Yuen PC (2008) Incremental linear discriminant analysis for face recognition. IEEE Trans Syst Man Cybern Part B 38(1):210–221

    Article  Google Scholar 

  42. Kim T-K, Wong SF, Stenger B, Kittler J, Cipolla R (2007) Incremental linear discriminant analysis using sufficient spanning set approximations. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR’07). Minneapolis, MN, pp 1–7

  43. Kim T-K, Stenger B, Kittler J, Cipolla R (2011) Incremental linear discriminant analysis using sufficient spanning sets and its applications. Int J Comput Vis 91(2):216–232

    Article  MathSciNet  MATH  Google Scholar 

  44. Ross DA, Lim J, Lin R-S (2008) Incremental learning for robust visual tracking. Int J Comput Vis 77(1–3):125–141

    Article  Google Scholar 

  45. Golub GH, Loan CFV (1996) Matrix computations, 3rd edn. The Johns Hopkins University Press, Baltimore

    MATH  Google Scholar 

  46. Lang K (1995) Newsweeder: learning to filter Netnews. In: International Conference on Machine Learning (ICML), pp 331–339

  47. Xiong T, Ye J, Cherkassky V (2006) Kernel uncorrelated and orthogonal discriminant analysis: a unified approach. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), pp 125–131

Download references

Acknowledgments

This research is supported by Anhui Provincial Natural Science Foundation (No. 1308085MF95), the Pre-research Foundation of NSFC of Anhui Polytechnic University (zryy1305), the Key Laboratory of Intelligent Perception and Systems for High-Dimensional Information (Nanjing University of Science and Technology), Ministry of Education (Grant No. 30920130122005), China Postdoctoral Science Foundation (2013M531251).

Author information

Authors and Affiliations

  1. School of Computer and Information, AnHui Polytechnic University, Wuhu, 241000, Anhui, China

    Gui-Fu Lu, Jian Zou & Yong Wang

Authors
  1. Gui-Fu Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jian Zou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yong Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gui-Fu Lu.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lu, GF., Zou, J. & Wang, Y. A New and Fast Implementation of Orthogonal LDA Algorithm and Its Incremental Extension. Neural Process Lett 43, 687–707 (2016). https://doi.org/10.1007/s11063-015-9441-6

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-015-9441-6

Keywords

Search

Navigation