Neural Computing and Applications

, Volume 28, Issue 9, pp 2809–2822 | Cite as

DC programming and DCA for sparse Fisher linear discriminant analysis

  • Hoai An Le ThiEmail author
  • Duy Nhat Phan
Original Article


We consider the supervised pattern classification in the high-dimensional setting, in which the number of features is much larger than the number of observations. We present a novel approach to the sparse Fisher linear discriminant problem using the \(\ell _0\)-norm. The resulting optimization problem is nonconvex, discontinuous and very hard to solve. We overcome the discontinuity by using appropriate approximations to the \(\ell _0\)-norm such that the resulting problems can be formulated as difference of convex functions (DC) programs to which DC programming and DC Algorithms (DCA) are investigated. The experimental results on both simulated and real datasets demonstrate the efficiency of the proposed algorithms compared to some state-of-the-art methods.


Classification Feature selection Fisher linear discriminant analysis DC programming DCA 



This research is funded by Foundation for Science and Technology Development of Ton Duc Thang University (FOSTECT), website:, under Grant FOSTECT.2015.BR.15. The authors would like to thank the referees for their valuable comments which helped to improve the manuscript.


  1. 1.
    Alon U, Barkai N, Notterman DA, Gish K, Ybarra S, Mack D, Levine AJ (1999) Broad patterns of gene expression revealed by clustering analysis of tumor and normal colon tissues probed by oligonucleotide arrays. Proc Natl Acad Sci USA 96(12):6745–6750CrossRefGoogle Scholar
  2. 2.
    Bickel PJ, Levina E (2004) Some theory for Fisher’s linear discriminant function, naive Bayes, and some alternatives when there are many more variables than observations. Bernoulli 10(6):989–1010MathSciNetCrossRefzbMATHGoogle Scholar
  3. 3.
    Boyd S, Parikh N, Chu E, Peleato B, Eckstein J (2011) Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3(1):1–124CrossRefzbMATHGoogle Scholar
  4. 4.
    Bradley PS, Mangasarian OL (1998) Feature selection via concave minimization and support vector machines. In: Proceeding of international conference on machine learning ICML98Google Scholar
  5. 5.
    Chen X, Xu FM, Ye Y (2010) Lower bound theory of nonzero entries in solutions of l2-lp minimization. SIAM J Sci Comput 32(5):2832–2852MathSciNetCrossRefzbMATHGoogle Scholar
  6. 6.
    Cheng S, Le Thi HA (2013) Learning sparse classifiers with difference of convex functions algorithms. Optim Methods Softw 28(4):830–854MathSciNetCrossRefzbMATHGoogle Scholar
  7. 7.
    Clemmensen L, Hansen M, Ersboll B, Frisvad J (2007) A method for comparison of growth media in objective identification of penicillium based on multi-spectral imaging. J Microbiol Methods 69:249–255CrossRefGoogle Scholar
  8. 8.
    Clemmensen L, Hastie T, Witten D, Ersbøll B (2011) Sparse discriminant analysis. Technometrics 53(4):406–413MathSciNetCrossRefGoogle Scholar
  9. 9.
    Collobert R, Sinz F, Weston J, Bottou L (2006) Trading convexity for scalability. In Proceedings of the 23rd international conference on machine learning, NY, USA, pp 201–208Google Scholar
  10. 10.
    Fisher RA (1936) The use of multiple measurements in taxonomic problems. Ann Eugen 7:179–188CrossRefGoogle Scholar
  11. 11.
    Friedman J, Hastie T, Hoefling H, Tibshirani R (2007) Pathwise coordinate optimization. An Appl Stat 1:302–332MathSciNetCrossRefzbMATHGoogle Scholar
  12. 12.
    Gasso G, Rakotomamonjy A, Canu S (2009) Recovering sparse signals with a certain family of nonconvex penalties and dc programming. IEEE Trans Signal Process 57:4686–4698MathSciNetCrossRefGoogle Scholar
  13. 13.
    Gordon GJ, Jensen RV, Hsiao LL, Gullans SR, Blumenstock JE, Ramaswamy S, Richards WG, Sugarbaker DJ, Bueno R (2002) Translation of microarray data into clinically relevant cancer diagnostic tests using gene expression ratios in lung cancer and mesothelioma. Cancer Res 62:4963–4967Google Scholar
  14. 14.
    Grosenick L, Greer S, Knutson B (2008) Interpretable classifiers for fmri improve prediction of purchases. IEEE Trans Neural Syst Rehabil Eng 16(6):539–547CrossRefGoogle Scholar
  15. 15.
    Guo Y, Hastie T, Tibshirani R (2007) Regularized linear discriminant analysis and its application in microarrays. Biostatistics 8(1):86–100CrossRefzbMATHGoogle Scholar
  16. 16.
    Hastie T, Buja A, Tibshirani R (1995) Penalized discriminant analysis. Ann Stat 23(1):73–102MathSciNetCrossRefzbMATHGoogle Scholar
  17. 17.
    Hastie T, Tibshirani R, Friedman J (2009) The elements of statistical learning. Springer Verlag, New YorkCrossRefzbMATHGoogle Scholar
  18. 18.
    Khan J, Wei JS, Ringner M, Saal LH, Ladanyi M, Westermann F, Berthold F, Schab M, Antonescu CR, Peterson C, Meltzer PS (2001) Classification and diagnostic prediction of cancers using expression profiling and artificial neural networks. Nat Med 7:673–679CrossRefGoogle Scholar
  19. 19.
    Krause N, Singer Y (2004) Leveraging the margin more carefully. In: Proceedings of the twenty first international conference on machine learning, NY, USAGoogle Scholar
  20. 20.
    Krzanowski W, Jonathan P, Mccarthy W, Thomas M (1995) Discriminant analysis with singular covariance matrices: methods and applications to spectroscopic data. J R Stat Soc 44(1):101–115zbMATHGoogle Scholar
  21. 21.
    Le Hoai M, Le Thi HA, Pham Dinh T, Huynh VN (2013) Block clustering based on difference of convex functions (DC) programming and DC algorithms. Neural Comput 25:259–278MathSciNetCrossRefGoogle Scholar
  22. 22.
    Le Thi HA (2000) An efficient algorithm for globally minimizing a quadratic function under convex quadratic constraints. Math Program 87:401–426MathSciNetCrossRefzbMATHGoogle Scholar
  23. 23.
    Le Thi HA, Le Hoai M, Nguyen VV, Pham Dinh T (2008) A DC programming approach for feature selection in support vector machines learning. J Adv Data Anal Classif 2(3):259–278MathSciNetCrossRefzbMATHGoogle Scholar
  24. 24.
    Le Thi HA, Le Hoai M, Pham Dinh T (2007) Optimization based DC programming and DCA for hierarchical clustering. Eur J Oper Res 183:1067–1085MathSciNetCrossRefzbMATHGoogle Scholar
  25. 25.
    Le Thi HA, Le HM, Dinh TP (2014a) New and efficient DCA based algorithms for minimum sum-of-squares clustering. Pattern Recognit 47:388–401CrossRefzbMATHGoogle Scholar
  26. 26.
    Le Thi HA, Le Hoai M, Pham Dinh T (2015a) Feature selection in machine learning: an exact penalty approach using a difference of convex function algorithm. Mach Learn 101:163–186MathSciNetCrossRefzbMATHGoogle Scholar
  27. 27.
    Le Thi HA, Nguyen MC (2014) Self-organizing maps by difference of convex functions optimization. Data Min Knowl Discov 28:1336–1365MathSciNetCrossRefzbMATHGoogle Scholar
  28. 28.
    Le Thi HA, Nguyen VV, Ouchani S (2009) Gene selection for cancer classification using DCA. J Front Comput Sci Technol 3:612–620Google Scholar
  29. 29.
    Le Thi HA, Pham Dinh T (2005) The DC (difference of convex functions) programming and DCA revisited with DC models of real world nonconvex optimization problems. Ann Oper Res 133:23–46MathSciNetCrossRefzbMATHGoogle Scholar
  30. 30.
    Le Thi HA, Pham Dinh T, Huynh VN (2012) Exact penalty and error bounds in DC programming. J Glob Optim 52(3):509–535MathSciNetCrossRefzbMATHGoogle Scholar
  31. 31.
    Le Thi HA, Pham Dinh T, Le Hoai M, Vo Xuan T (2015b) DC approximation approaches for sparse optimization. Eur J Oper Res 244:26–44MathSciNetCrossRefzbMATHGoogle Scholar
  32. 32.
    Le Thi HA, Vo Xuan T, Pham Dinh T (2014b) Feature selection for linear SVMs under uncertain data: robust optimization based on difference of convex functions algorithms. Neural Netw 59:36–50CrossRefzbMATHGoogle Scholar
  33. 33.
    Leng C (2008) Sparse optimal scoring for multiclass cancer diagnosis and biomarker detection using microarray data. Comput Biol Chem 32:417–425MathSciNetCrossRefzbMATHGoogle Scholar
  34. 34.
    Liu Y, Shen X, Doss H (2005) Multicategory \(\psi \)-learning and support vector machine: computational tools. J Comput Graph Stat 14:219–236MathSciNetCrossRefGoogle Scholar
  35. 35.
    Mai Q, Zou H (2013) A note on the connection and equivalence of three sparse linear discriminant analysis methods. Technometrics 55(2):243–246MathSciNetCrossRefGoogle Scholar
  36. 36.
    Mai Q, Zou H, Yuan M (2012) A direct approach to sparse discriminant analysis in ultra-high dimensions. Biometrika 99(1):29–42MathSciNetCrossRefzbMATHGoogle Scholar
  37. 37.
    Mardia KV, Kent JT, Bibby JM (1979) Multivariate Analysis. Academic Press, London, New York, Toronto, Sydney, San FranciscoGoogle Scholar
  38. 38.
    Nakayama R, Nemoto T, Takahashi H, Ohta T, Kawai A, Yoshida T, Toyama Y, Ichikawa H, Hasegama T (2007) Gene expression analysis of soft tissue sarcomas: characterization and reclassification of malignant fibrous histiocytoma. Mod Pathol 20(7):749–759CrossRefGoogle Scholar
  39. 39.
    Neumann J, Schnorr G, Steidl G (2005) Combined SVM-based feature selection and classification. Mach Learn 61:129–150CrossRefzbMATHGoogle Scholar
  40. 40.
    Peleg D, Meir R (2008) A bilinear formulation for vector sparsity optimization. Signal Process 88(2):375–389CrossRefzbMATHGoogle Scholar
  41. 41.
    Pham Dinh T, Le Thi HA (1997) Convex analysis approach to D.C. programming: theory, algorithms and applications. Acta Math Vietnam 22(1):289–355MathSciNetzbMATHGoogle Scholar
  42. 42.
    Pham Dinh T, Le Thi HA (1998) A DC optimization algorithm for solving the trust-region subproblem. SIAM J Optim 8(2):476–505MathSciNetCrossRefzbMATHGoogle Scholar
  43. 43.
    Pham Dinh T, Le Thi HA (2014) Recent advances in dc programming and dca. Trans Comput Collect Intell 8342:1–37Google Scholar
  44. 44.
    Sun L, Hui A, Su Q, Vortmeyer A, Kotliarov Y, Pastorino S, Passaniti A, Menon J, Wlling J, Bailey R, Rosenblum M, Mikkelsen T, Fine H (2006) Neuronal and glioma-derived stem cell factor induces angiogenesis within the brain. Cancer Cell 9:287–300CrossRefGoogle Scholar
  45. 45.
    Tibshirani R (1996) Regression shrinkage and selection via the lasso. J R Stat Soc 58:267–288MathSciNetzbMATHGoogle Scholar
  46. 46.
    Tibshirani R, Hastie T, Narasimhan B, Chu G (2002) Diagnosis of multiple cancer types by shrunken centroids of gene expression. Proc Natl Acad Sci 99:6567–6572CrossRefGoogle Scholar
  47. 47.
    Tibshirani R, Hastie T, Narasimhan B, Chu G (2003) Class prediction by nearest shrunken centroids, with applications to DNA microarrays. Stat Sci 18(1):104–117MathSciNetCrossRefzbMATHGoogle Scholar
  48. 48.
    Trendafilov NT, Jolliffe IT (2007) Dalass: Variable selection in discriminant analysis via the lasso. Comput Stat Data Anal 51:3718–3736MathSciNetCrossRefzbMATHGoogle Scholar
  49. 49.
    Witten D, Tibshirani R (2011) Penalized classification using Fisher’s linear discriminant. J R Stat Soc B 73:753–772MathSciNetCrossRefzbMATHGoogle Scholar
  50. 50.
    Wu M, Zhang L, Wang Z, Christiani D, Lin X (2009) Sparse linear discriminant analysis for simultaneous testing for the significance of a gene set/pathway and gene selection. Bioinformatics 25:1145–1151CrossRefGoogle Scholar
  51. 51.
    Xu P, Brock GN, Parrish RS (2009) Modified linear discriminant analysis approaches for classification of high-dimensional microarray data. Comput Stat Data Anal 53:1674–1687MathSciNetCrossRefzbMATHGoogle Scholar
  52. 52.
    Yeoh EJ, Ross ME, Shurtleff SA, Williams WK, Patel D, Mahfouz R, Behm FG, Raimondi SC, Relling MV, Patel A et al (2002) Classification, subtype discovery, and prediction of outcome in pediatric lymphoblastic leukemia by gene expression profiling. Cancer Cell 1:133–143CrossRefGoogle Scholar

Copyright information

© The Natural Computing Applications Forum 2016

Authors and Affiliations

  1. 1.Department for Management of Science and Technology Development, Faculty of Mathematics StatisticsTon Duc Thang UniversityHo Chi Minh CityVietnam
  2. 2.Laboratory of Theoretical and Applied Computer ScienceUniversity of LorraineMetzFrance

Personalised recommendations