Skip to main content
SpringerLink
Log in

Linear discriminant analysis for the small sample size problem: an overview

  • Original Article
  • Published:
International Journal of Machine Learning and Cybernetics Aims and scope Submit manuscript

Abstract

Dimensionality reduction is an important aspect in the pattern classification literature, and linear discriminant analysis (LDA) is one of the most widely studied dimensionality reduction technique. The application of variants of LDA technique for solving small sample size (SSS) problem can be found in many research areas e.g. face recognition, bioinformatics, text recognition, etc. The improvement of the performance of variants of LDA technique has great potential in various fields of research. In this paper, we present an overview of these methods. We covered the type, characteristics and taxonomy of these methods which can overcome SSS problem. We have also highlighted some important datasets and software/packages.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. These four spaces can also be represented in Fig. 2 without performing a preprocessing step. In that case, r t in the figure will be replaced by the dimensionality d and the size of the spaces will change accordingly.

  2. For this experiment, first we project the original feature vectors onto the range space of \({\mathbf{S}}_{T}\) matrix as a pre-processing step. Then all the spaces are utilized individually to do dimensionality reduction and to classify a test feature vector, the nearest neighbor classifier is used. To obtain performance in terms of average classification accuracy, \(k\)-fold cross-validation process has been applied, where k = 5. The details of the datasets have been given later in Sect. 10.1.

  3. For more datasets on face see Ralph Gross [19], Zhao et al. [70] and http://www.face-rec.org/databases/. For bio-medical data see Kent Ridge Bio-medical Repository (http://datam.i2r.a-star.edu.sg/datasets/krbd/).

References

  1. Aas K, Eikvil L (1999) Text categorization: a survey. Norwegian Computing Center Report NR 941

  2. Alon U, Barkai N, Notterman DA, Gish K, Ybarra S, Mack D, Levine AJ (1999) Broad patterns of gene expression revealed by clustering of tumor and normal colon tissues probed by oligonucleotide arrays. PNAS 96:6745–6750

    Article  Google Scholar 

  3. Armstrong SA, Staunton JE, Silverman LB, Pieters R, den Boer ML, Minden MD, Sallan SE, Lander ES, Golub TR, Korsemeyer SJ (2002) MLL translocations specify a distinct gene expression profile that distinguishes a unique leukemia. Nat Genet 30:41–47

    Article  Google Scholar 

  4. Beer DG, Kardia SLR, Huang C–C, Giordano TJ, Levin AM, Misek DE, Lin L, Chen G, Gharib TG, Thomas DG, Lizyness ML, Kuick R, Hayasaka S, Taylor JMG, Iannettoni MD, Orringer MB, Hanash S (2002) Gene-expression profiles predict survival of patients with lung adenocarcinoma. Nat Med 8:816–824

    Google Scholar 

  5. Belhumeur PN, Hespanhaand JP, Kriegman DJ (1997) Eigenfaces vs. fisherfaces: recognition using class specific linear projection. IEEE Trans Pattern Anal Machine Intell 19(7):711–720

    Article  Google Scholar 

  6. Bhattacharjee A, Richards WG, Staunton J, Li C, Monti S, Vasa P, Ladd C, Beheshti J, Bueno R, Gillette M, Loda M, Weber G, Mark EJ, Lander ES, Wong W, Johnson BE, Golub TR, Sugarbaker DJ, Meyerson M (2001) Classification of human lung carcinomas by mRNA expression profiling reveals distinct adenocarcinoma sub-classes. PNAS 98(24):13790–13795

    Article  Google Scholar 

  7. Blake CL, Merz CJ (1998) UCI repository of machine learning databases, Irvine, CA, University of Calif., Dept. of Information and Comp. Sci. http://www.ics.uci.edu/_mlearn

  8. Cevikalp H, Neamtu M, Wilkes MA, Barkana A (2005) Discriminative common vectors for face recognition. IEEE Trans Pattern Anal Machine Intell 27(1):4–13

    Article  Google Scholar 

  9. Chen L-F, Liao H-YM, Ko M-T, Lin J-C, Yu G-J (2000) A new LDA-based face recognition system which can solve the small sample size problem. Pattern Recogn 33:1713–1726

    Article  Google Scholar 

  10. Chu D, Thye GS (2010) A new and fast implementation for null space based linear discriminant analysis. Pattern Recogn 43:1373–1379

    Article  MATH  Google Scholar 

  11. Cui X, Zhao H, Wilson J (2010) Optimized ranking and selection methods for feature selection with application in microarray experiments. J Biopharm Stat 20(2):223–239

    Article  MathSciNet  Google Scholar 

  12. Dai DQ, Yuen PC (2007) Face recognition by regularized discriminant analysis. IEEE Trans SMC Part B 37(4):1080–1085

    Google Scholar 

  13. Dudoit S, Fridlyand J, Speed TP (2002) Comparison of discrimination methods for the classification of tumors using gene expression data. J Am Stat Assoc 97(457):77–87

    Article  MATH  MathSciNet  Google Scholar 

  14. Friedman JH (1989) Regularized discriminant analysis. J Am Stat Assoc 84(405):165–175

    Article  Google Scholar 

  15. Fukunaga K (1990) Introduction to statistical pattern recognition. Academic Press Inc., Hartcourt Brace Jovanovich, San Diego, CA

  16. Furey TS, Cristianini N, Duffy N, Bednarski DW, Schummer M, Haussler D (2000) Support vector machine classification and validation of cancer tissue samples using microarray expression data. Bioinformatics 16:906–914

    Article  Google Scholar 

  17. Golub TR, Slonim DK, Tamayo P, Huard C, Gaasenbeek M, Mesirov JP, Coller H, Loh ML, Downing JR, Caligiuri MA, Bloomfield CD, Lander ES (1999) Molecular classification of cancer: class discovery and class prediction by gene expression monitoring. Science 286:531–537

    Article  Google Scholar 

  18. Gordon GJ, Jensen RV, Hsiao L–L, Gullans SR, Blumenstock JE, Ramaswamy S, Richards WG, Sugarbaker DJ, Bueno R (2002) Translation of microarray data into clinically relevant cancer diagnostic tests using gene expression ratios in lung cancer and mesothelioma. Cancer Res 62:4963–4967

    Google Scholar 

  19. Gross R (2005) Face databases. In: Handbook of face recognition. Springer, New York, pp 301–327

    Chapter  Google Scholar 

  20. Hastie T, Buja A, Tibshirani R (1995) Penalized discriminant analysis. Ann Stat 23:73–102

    Article  MATH  MathSciNet  Google Scholar 

  21. Huang R, Liu Q, Lu H, Ma S (2002) Solving the small sample size problem of LDA. Proc ICPR 3(2002):29–32

    Google Scholar 

  22. Jiang X, Mandal B, Kot A (2008) Eigenfeature regularization and extraction in face recognition. IEEE Trans Pattern Anal Machine Intell 30(3):383–394

    Article  Google Scholar 

  23. Khan J, Wei JS, Ringner M, Saal LH, Ladanyi M, Westermann F, Berthold F, Schwab M, Antonescu CR, Peterson C, Meltzer PS (2001) Classification and diagnostic prediction of cancers using gene expression profiling and artificial neural network. Nat Med 7:673–679

    Article  Google Scholar 

  24. Lewis DD (1999) Reuters-21578 text categorization test collection distribution 1.0. http://www.daviddlewis.com/resources/testcollections/reuters21578/

  25. Li H, Zhang K, Jiang T (2005) Robust and accurate cancer classification with gene expression profiling. In: Proceedings of IEEE Comput. Syst. Bioinform. Conf., pp 310–321

  26. Li H, Jiang T, Zhang K (2003) Efficient and robust feature extraction by maximum margin criterion. In: Advances in neural information processing systems

  27. Liu J, Chen SC, Tan XY (2007) Efficient pseudo-inverse linear discriminant analysis and its nonlinear form for face recognition. Int J Pattern Recogn Artif Intell 21(8):1265–1278

    Article  Google Scholar 

  28. Lu J, Plataniotis K, Venetsanopoulos A (2003) Face recognition using kernel direct discriminant analysis algorithms. IEEE Trans Neural Netw 14(1):117–126

    Article  Google Scholar 

  29. Lu J, Plataniotis KN, Venetsanopoulos AN (2003) Regularized discriminant analysis for the small sample. Pattern Recogn Lett 24:3079–3087

    Article  Google Scholar 

  30. Lu J, Plataniotis KN, Venetsanopoulos AN (2005) Regularization studies of linear discriminant analysis in small sample size scenarios with application to face recognition. Pattern Recogn Lett 26(2):181–191

    Article  Google Scholar 

  31. Mak MW, Kung SY (2006) A solution to the curse of dimensionality problem in pairwise scoring techniques. In: Int. Conf. on Neural Info. Process. (ICONIP’06), pp 314–323

  32. Martinez AM (2002) Recognizing imprecisely localized, partially occluded, and expression variant faces from a single sample per class. IEEE Trans Pattern Anal Machine Intell 24(6):748–763

    Article  Google Scholar 

  33. Moghaddam B, Weiss Y, Avidan S (2006) Generalized spectral bounds for sparse LDA. In: Int. Conf. Mach. Learn., ICML’06, pp 641–648

  34. Paliwal KK, Sharma A (2012) Improved pseudoinverse linear discriminant analysis method for dimensionality reduction. Int J Pattern Recogn Artif Intell 26(1):1250002-1–1250002-9

    MathSciNet  Google Scholar 

  35. Paliwal KK, Sharma A (2011) Approximate LDA technique for dimensionality reduction in the small sample size case. J Pattern Recogn Res 6(2):298–306

    Article  Google Scholar 

  36. Paliwal KK, Sharma A (2010) Improved direct LDA and its application to DNA microarray gene expression data. Pattern Recogn Lett 31:2489–2492

    Article  Google Scholar 

  37. Petricoin EF III, Ardekani AM, Hitt BA, Levine PJ, Fusaro VA, Steinberg MS, Mills GB, Simone C, Fishman DA, Kohn EC, Liotta LA (2002) Use of proteomic patterns in serum to identify ovarian cancer. Lancet 359:572–577

    Article  Google Scholar 

  38. Phillips PJ, Moon H, Rauss PJ, Rizvi S (2000) The FERET evaluation methodology for face recognition algorithms. IEEE Trans Pattern Anal Mach Intell 22(10):1090–1104

    Article  Google Scholar 

  39. Pomeroy SL, Tamayo P, Gaasenbeek M, Sturla LM, Angelo M, McLaughlin ME, Kim JYH, Goumnerova LC, Black PM, Lau C, Allen JC, Zagzag D, Olson JM, Curran T, Wetmore C, Biegel JA, Poggio T, Mukherjee S, Rifkin R, Califano A, Stolovitzky G, Louis DN, Mesirov JP, Lander ES, Golub TR (2002) Gene expression-based classification and outcome prediction of central nervous system embryonal tumors. Nature 415:436–442

    Article  Google Scholar 

  40. Ramaswamy S, Tamayo P, Rifkin R, Mukherjee S, Yeang C-H, Angelo M, Ladd C, Reich M, Latulippe E, Mesirov JP, Poggio T, Gerald W, Loda M, Lander ES, Golub TR (2001) Multiclass cancer diagnosis using tumor gene expression signatures. Proc Natl Acad Sci USA 98(26):15149–15154

    Article  Google Scholar 

  41. Samaria F, Harter A (1994) Parameterization of a stochastic model for human face identification. In: Proceedings of the Second IEEE Workshop Appl. of Comp. Vis., pp 138–142

  42. Sanderson C, Paliwal KK (2003) Fast features for face authentication under illumination direction changes. Pattern Recogn Lett 24:2409–2419

    Article  Google Scholar 

  43. Sharma A, Paliwal KK (2006) Class-dependent PCA, LDA and MDC: a combined classifier for pattern classification. Pattern Recogn 39(7):1215–1229

    Article  MATH  Google Scholar 

  44. Sharma A, Paliwal KK (2007) Fast principal component analysis using fixed-point algorithm. Pattern Recogn Lett 28(10):1151–1155

    Article  Google Scholar 

  45. Sharma A, Paliwal KK (2008) Cancer classification by gradient LDA technique using microarray gene expression data. Data Knowl Eng 66(2):338–347

    Article  Google Scholar 

  46. Sharma A, Paliwal KK (2008) Rotational linear discriminant analysis technique for dimensionality reduction. IEEE Trans Knowl Data Eng 20(10):1336–1347

    Article  Google Scholar 

  47. Sharma A, Paliwal KK (2010) Regularisation of eigenfeatures by extrapolation of scatter-matrix in face-recognition problem. Electron Lett IEEE 46(10):450–475

    Article  Google Scholar 

  48. Sharma A, Imoto S, Miyano S, Sharma V (2011) Null space based feature selection method for gene expression data. Int J Mach Learn Cybernet. doi:10.1007/s13042-011-0061-9

    Google Scholar 

  49. Sharma A, Paliwal KK (2012) A new perspective to null linear discriminant analysis method and its fast implementation using random matrix multiplication with scatter matrices. Pattern Recogn 45:2205–2213

    Article  MATH  Google Scholar 

  50. Sharma A, Paliwal KK (2012) A two-stage linear discriminant analysis for face-recognition. Pattern Recogn Lett 33:1157–1162

    Article  Google Scholar 

  51. Sharma A, Imoto S, Miyano S (2012) A top-r feature selection algorithm for microarray gene expression data. IEEE/ACM Trans Comput Biol Bioinf 9(3):754–764

    Article  Google Scholar 

  52. Sharma A, Imoto S, Miyano S (2012) A between-class overlapping filter-based method for transcriptome data analysis. J Bioinf Comput Biol 10(5):1250010-1–1250010-20

    Article  Google Scholar 

  53. Sharma A, Imoto S, Miyano S (2012) A filter based feature selection algorithm using null space of covariance matrix for DNA microarray gene expression data. Curr Bioinf 7(3):6

    Article  Google Scholar 

  54. Sharma A, Paliwal KK, Imoto S, Miyano S (2013) A feature selection method using improved regularized linear discriminant analysis. Mach Vis Appl. doi:10.1007/s00138-013-0577-y

    Google Scholar 

  55. Singh D, Febbo PG, Ross K, Jackson DG, Manola J, Ladd C, Tamayo P, Renshaw AA, D’Amico AV, Richie JP, Lander ES, Loda M, Kantoff PW, Golub TR, Sellers WR (2002) Gene expression correlates of clinical prostate cancer behavior. Cancer Cell 1:203–209

    Article  Google Scholar 

  56. Song F, Zhang D, Wang J, Liu H, Tao Q (2007) A parameterized direct LDA and its application to face recognition. Neurocomputing 71:191–196

    Article  Google Scholar 

  57. Swets DL, Weng J (1996) Using discriminative eigenfeatures for image retrieval. IEEE Trans Pattern Anal Mach Intell 18(8):831–836

    Article  Google Scholar 

  58. Thomaz CE, Kitani EC, Gillies DF (2005) A maximum uncertainty LDA-based approach for limited sample size problems with application to face recognition. In: Proceedings of 18th Brazilian Symp. On Computer Graphics and Image Processing, (IEEE CS Press), pp 89–96

  59. Tian Q, Barbero M, Gu ZH, Lee SH (1986) Image classification by the Foley-Sammon transform. Opt Eng 25(7):834–840

    Article  Google Scholar 

  60. van’t Veer LJ, Dai H, van de Vijver MJ, He YD, Hart AMH, Mao M, Peterse HL, van der Kooy K, Marton MJ, Witteveen AT, Schreiber GJ, Kerkhoven RM, Roberts C, Linsley PS, Bernards R, Friend SH (2002) Gene expression profiling predicts clinical outcome of breast cancer. Lett Nat Nat 415:530–536

    Article  Google Scholar 

  61. Witten IH, Frank E (2000) Data mining: practical machine learning tools with java implementations. Morgan Kaufmann, San Francisco. http:/www.cs.waikato.ac.nz/ml/weka/

  62. Ye J (2005) Characterization of a family of algorithms for generalized discriminant analysis on undersampled problems. J Mach Learn Res 6:483–502

    MATH  MathSciNet  Google Scholar 

  63. Ye J, Janardan R, Li Q, Park H (2004) Feature extraction via generalized uncorrelated linear discriminant analysis. In: The Twenty-First International Conference on Machine Learning, pp 895–902

  64. Ye J, Li Q (2005) A two-stage linear discriminant analysis via QR-decomposition. IEEE Trans Pattern Anal Mach Intell 27(6):929–941

    Article  Google Scholar 

  65. Ye J, Xiong T (2006) Computational and theoretical analysis of null space and orthogonal linear discriminant analysis. J Mach Learn Res 7:1183–1204

    MATH  MathSciNet  Google Scholar 

  66. Yeoh EJ, Ross ME, Shurtleff SA, Williams WK, Patel D, Mahfouz R, Behm FG, Raimondi SC, Relling MV, Patel A, Cheng C, Campana D, Wilkins D, Zhou X, Li J, Liu H, Pui CH, Evans WE, Naeve C, Wong L, Downing JR (2002) Classification, subtype discovery, and prediction of outcome in pediatric acute lymphoblastic leukemia by gene expression profiling. Cancer 1(2):133–143

    Google Scholar 

  67. Yu H, Yang J (2001) A direct LDA algorithm for high-dimensional data-with application to face recognition. Pattern Recogn 34:2067–2070

    Article  MATH  Google Scholar 

  68. Zhao W, Chellappa R, Krishnaswamy A (1998) Discriminant analysis of principal components for face recognition. In: Proceedings of Thir Int. Conf. on Automatic Face and Gesture Recognition, Nara, Japan, pp 336–341

  69. Zhao W, Chellappa R, Phillips PJ (1999) Subspace linear discriminant analysis for face recognition, Technical Report CAR-TR-914, CS-TR-4009. University of Maryland at College Park, USA

    Google Scholar 

  70. Zhao W, Chellappa R, Phillips PJ (2003) Face recognition: a literature survey. ACM Comput Surv 35(4):399–458

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Engineering, Griffith University, Brisbane, Australia

    Alok Sharma & Kuldip K. Paliwal

  2. School of Engineering and Physics, University of the South Pacific, Suva, Fiji

    Alok Sharma

Authors
  1. Alok Sharma
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kuldip K. Paliwal
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alok Sharma.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Sharma, A., Paliwal, K.K. Linear discriminant analysis for the small sample size problem: an overview. Int. J. Mach. Learn. & Cyber. 6, 443–454 (2015). https://doi.org/10.1007/s13042-013-0226-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13042-013-0226-9

Keywords

Search

Navigation