Semi-supervised orthogonal discriminant analysis with relative distance : integration with a MOO approach

  • Rakesh Kumar SanodiyaEmail author
  • Sriparna Saha
  • Jimson Mathew
Methodologies and Application


In discriminant analysis, trace ratio is an important criterion for minimizing the between-class similarity and maximizing the within-class similarity, simultaneously. In brief, we address the trace ratio problem associated with many semi-supervised discriminant analysis algorithms as they use the normal Euclidean distances between training data samples. Based on this problem, we propose a new semi-supervised orthogonal discriminant analysis technique with relative distance constraints called SSODARD. Different from the existing semi-supervised dimensionality reduction algorithms, our algorithm is more consistent in propagating the label information from the labeled data to the unlabeled data because of the use of relative distance function instead of normal Euclidean distance function. For finding this appropriate relative distance function, we use pairwise constraints generated from labeled data and satisfy them using Bregman projection. Since the projection is not orthogonal, we require an appropriate subset of constraints. In order to select such a subset of constraints, we further develop a framework called MO-SSODARD, which uses evolutionary algorithm while optimizing various validity indices simultaneously. The experimental results on various datasets show that our proposed approaches are superior than the state-of-the-art discriminant algorithms with respect to various validity indices.


Semi-supervised learning Dimension reduction Data clustering Bregman projection Validity indices Trace ratio 


Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflict of interest.

Human participants

This study does not contain any studies with human participants or animals performed by any of the authors.


  1. Baldi P, Long AD (2001) A Bayesian framework for the analysis of microarray expression data: regularized t-test and statistical inferences of gene changes. Bioinformatics 17(6):509–519CrossRefGoogle Scholar
  2. Belhumeur PN, Hespanha JP, Kriegman DJ (1997) Eigenfaces vs. fisherfaces: recognition using class specific linear projection. IEEE Trans Pattern Anal Mach Intell 19(7):711–720CrossRefGoogle Scholar
  3. Belkin M, Niyogi P, Sindhwani V (2006) Manifold regularization: a geometric framework for learning from labeled and unlabeled examples. J Mach Learn Res 7(Nov):2399–2434MathSciNetzbMATHGoogle Scholar
  4. Blum A, Mitchell T (1998) Combining labeled and unlabeled data with co-training. In: Proceedings of the eleventh annual conference on computational learning theory, pp 92–100. ACMGoogle Scholar
  5. Cai D, He X, Han J (2007) Semi-supervised discriminant analysis. In: IEEE 11th international conference on computer vision, 2007. ICCV 2007, pp 1–7. IEEEGoogle Scholar
  6. Chen LF, Liao HYM, Ko MT, Lin JC, Yu GJ (2000) A new IDA-based face recognition system which can solve the small sample size problem. Pattern Recognit 33(10):1713–1726CrossRefGoogle Scholar
  7. Chung MK (2012) Gaussian kernel smoothing. Lecture notes, pp 1–10Google Scholar
  8. Ciro GC, Dugardin F, Yalaoui F, Kelly R (2016) A nsga-ii and nsga-iii comparison for solving an open shop scheduling problem with resource constraints. IFAC-PapersOnLine 49(12):1272–1277CrossRefGoogle Scholar
  9. Corne DW, Knowles JD, Oates MJ (2000) The pareto envelope-based selection algorithm for multiobjective optimization. In: International conference on parallel problem solving from nature, pp 839–848. SpringerGoogle Scholar
  10. Das I, Dennis JE (1998) Normal-boundary intersection: a new method for generating the pareto surface in nonlinear multicriteria optimization problems. SIAM J Optim 8(3):631–657MathSciNetCrossRefzbMATHGoogle Scholar
  11. Deb K (2014) Multi-objective optimization. In: Search methodologies, pp 403–449. SpringerGoogle Scholar
  12. Deb K, Jain H (2014) An evolutionary many-objective optimization algorithm using reference-point-based nondominated sorting approach, part i: solving problems with box constraints. IEEE Trans Evol Comput 18(4):577–601CrossRefGoogle Scholar
  13. Deb K, Pratap A, Agarwal S, Meyarivan TAMT (2002) A fast and elitist multiobjective genetic algorithm: Nsga-ii. IEEE Trans Evol Comput 6(2):182–197CrossRefGoogle Scholar
  14. Duchene J, Leclercq S (1988) An optimal transformation for discriminant and principal component analysis. IEEE Trans Pattern Anal Mach Intell 10(6):978–983CrossRefzbMATHGoogle Scholar
  15. Fisher Ronald A (1936) The use of multiple measurements in taxonomic problems. Annals Hum Genet 7(2):179–188Google Scholar
  16. Hand DJ (1982) Kernel discriminant analysis. JOHN WILEY & SONS, INC., ONE WILEY DR., SOMERSET, N. J. 08873, 1982, 264Google Scholar
  17. He X, Niyogi P (2004) Locality preserving projections. In: Advances in neural information processing systems, pp 153–160Google Scholar
  18. Huang S, Elgammal A, Huangfu L, Yang D, Zhang X (2014) Globality-locality preserving projections for biometric data dimensionality reduction. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 15–20Google Scholar
  19. Huang Yi, Xu Dong, Nie Feiping (2012) Semi-supervised dimension reduction using trace ratio criterion. IEEE Trans Neural Netw Learn Syst 23(3):519–526CrossRefGoogle Scholar
  20. Izenman AJ (2013) Linear discriminant analysis. In: Modern multivariate statistical techniques, pp 237–280. SpringerGoogle Scholar
  21. Jain H, Deb K (2014) An evolutionary many-objective optimization algorithm using reference-point based nondominated sorting approach, part ii: handling constraints and extending to an adaptive approach. IEEE Trans Evol Comput 18(4):602–622CrossRefGoogle Scholar
  22. Jin Z, Yang JY, Hu ZS, Lou Z (2001) Face recognition based on the uncorrelated discriminant transformation. Pattern Recognit 34(7):1405–1416CrossRefzbMATHGoogle Scholar
  23. Joachims T (1999) Transductive inference for text classification using support vector machines. In: ICML, vol 99, pp 200–209Google Scholar
  24. Krzanowski WJ, Jonathan P, McCarthy WV, Thomas MR (1995) Discriminant analysis with singular covariance matrices: methods and applications to spectroscopic data. Appl Stat 44(1):101–115CrossRefzbMATHGoogle Scholar
  25. Kulis B, Sustik MA, Dhillon IS (2009) Low-rank kernel learning with bregman matrix divergences. J Mach Learn Res 10(Feb):341–376MathSciNetzbMATHGoogle Scholar
  26. Li Ming, Yuan Baozong (2005) 2d-lda: a statistical linear discriminant analysis for image matrix. Pattern Recognit Lett 26(5):527–532CrossRefGoogle Scholar
  27. Lu J, Zhou X, Tan YP, Shang Y, Zhou Jie (2012) Cost-sensitive semi-supervised discriminant analysis for face recognition. IEEE Trans Inf Forensics Secur 7(3):944–953CrossRefGoogle Scholar
  28. Mao-Guo G, Li-Cheng J, Dong-Dong Y, Wen-Ping M (2009) Evolutionary multi-objective optimization algorithmsGoogle Scholar
  29. Mishra S, Saha S, Mondal S (2016) Divide and conquer based non-dominated sorting for parallel environment. In: 2016 IEEE congress on evolutionary computation (CEC), pp 4297–4304. IEEEGoogle Scholar
  30. Murata T, Ishibuchi H (1995) Moga: multi-objective genetic algorithms. In: IEEE international conference on evolutionary computation, 1995. vol 1, p 289. IEEEGoogle Scholar
  31. Nie F, Xiang S, Jia Y, Zhang C (2009) Semi-supervised orthogonal discriminant analysis via label propagation. Pattern Recognit 42(11):2615–2627CrossRefzbMATHGoogle Scholar
  32. Nie F, Xu D, Tsang IWH, Zhang C (2010) Flexible manifold embedding: a framework for semi-supervised and unsupervised dimension reduction. IEEE Trans Image Process 19(7):1921–1932MathSciNetCrossRefzbMATHGoogle Scholar
  33. Rao CR (1948) The utilization of multiple measurements in problems of biological classification. J R Stat Soc Ser B (Methodological) 10(2):159–203MathSciNetzbMATHGoogle Scholar
  34. Roweis ST, Saul LK (2000) Nonlinear dimensionality reduction by locally linear embedding. Science 290(5500):2323–2326CrossRefGoogle Scholar
  35. Seada H, Deb K (2015) U-nsga-iii: a unified evolutionary optimization procedure for single, multiple, and many objectives: proof-of-principle results. In: International conference on evolutionary multi-criterion optimization, pp 34–49. SpringerGoogle Scholar
  36. Si S, Tao D, Geng B (2010) Bregman divergence-based regularization for transfer subspace learning. IEEE Trans Knowl Data Eng 22(7):929–942CrossRefGoogle Scholar
  37. Srinivas N, Deb K (1994) Multiobjective optimization using nondominated sorting in genetic algorithms. Evolu Comput 2(3):221–248CrossRefGoogle Scholar
  38. Sugiyama M, Idé T, Nakajima S, Sese J (2010) Semi-supervised local fisher discriminant analysis for dimensionality reduction. Mach Learn 78(1–2):35MathSciNetCrossRefGoogle Scholar
  39. Wang H, Yan S, Xu D, Tang X, Huang T (2007) Trace ratio vs. ratio trace for dimensionality reduction. In: IEEE Conference on computer vision and pattern recognition, 2007. CVPR’07. pp 1–8. IEEEGoogle Scholar
  40. Wang Sheng, Lu Jianfeng, Gu Xingjian, Du Haishun, Yang Jingyu (2016) Semi-supervised linear discriminant analysis for dimension reduction and classification. Pattern Recognit 57:179–189CrossRefGoogle Scholar
  41. Wang Xiaogang, Tang Xiaoou (2004) Dual-space linear discriminant analysis for face recognition. In: Proceedings of the 2004 IEEE computer society conference on computer vision and pattern recognition, 2004. CVPR 2004. vol 2, pp II–II. IEEEGoogle Scholar
  42. Wold S, Esbensen K, Geladi P (1987) Principal component analysis. Chemom Intell Lab Syst 2(1–3):37–52CrossRefGoogle Scholar
  43. Xu Y, Pan SJ, Xiong H, Wu Q, Luo Ronghua, Min Huaqing, Song Hengjie (2017) A unified framework for metric transfer learning. IEEE Trans Knowl Data Eng 29(6):1158–1171CrossRefGoogle Scholar
  44. Ye J, Li Q (2005) A two-stage linear discriminant analysis via qr-decomposition. IEEE Trans Pattern Anal Mach Intell 27(6):929–941CrossRefGoogle Scholar
  45. Yu W, Teng X, Liu C (2006) Face recognition using discriminant locality preserving projections. Image Vision Comput 24(3):239–248CrossRefGoogle Scholar
  46. Zhang Y, Yeung DY (2008a) Semi-supervised discriminant analysis using robust path-based similarity. In: IEEE conference on computer vision and pattern recognition, 2008. CVPR 2008. pp 1–8. IEEEGoogle Scholar
  47. Zhang Y, Yeung DY (2008b) Semi-supervised discriminant analysis via cccp. In: Joint European conference on machine learning and knowledge discovery in databases, pp 644–659. SpringerGoogle Scholar
  48. Zhou D, Bousquet O, Lal TN, Weston J, Schölkopf B (2004) Learning with local and global consistency. In: Advances in neural information processing systems, pp 321–328Google Scholar
  49. Zitzler E, Laumanns M, Thiele L (2001) Spea2: improving the strength pareto evolutionary algorithm. TIK-report, 103Google Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  • Rakesh Kumar Sanodiya
    • 1
    Email author
  • Sriparna Saha
    • 1
  • Jimson Mathew
    • 1
  1. 1.Department of Computer Science and EngineeringIndian Institute of Technology PatnaPatnaIndia

Personalised recommendations