Abstract
Most existing dimensionality reduction methods have been applied as a separable data preprocessing step before classification algorithms. This reduces the flexibility of classification algorithms. To handle this limitation, recently, a novel method, namely maximum margin projection pursuit (MMPP), was developed by simultaneously taking into account dimensionality reduction and classification in the criterion function. MMPP alternatively updates the projection matrix and normal vector of classifications hyperplane by optimizing the min-max problem. This results in the following two problems: (1) It does not guarantee both the convergence of the proposed iterative algorithm in real applications and minimization of the objective function; (2) It heavily depends on learning rate and does not get the global optimal solution. In this paper, we simultaneously solve both the projection matrix and norm vector of classification hyperplane by non-iterative method which not only optimizes the criterion function but also is faster than traditional MMPP algorithm. Furthermore, we extend our method to solve multiclass classification problems. Experiments on the Yale, ORL, AR and COIL20 databases have been conducted to evaluate our method. The results illustrate that, compared with the iterative algorithm, our no-iteration algorithm achieves higher efficiency, more stable recognition, and smaller objective value.
Similar content being viewed by others
References
Adankon MM, Cheriet M (2009) Support vector machine. Encyclopedia of biometrics: 1303–1308. Springer
Anton B, Fein J, To T, Li X, Silberstein L, Evans CJ (1996) Immunohistochemical localization of orl-1 in the central nervous system of the rat. J Comp Neurol 368(2):229–251
Belhumeur PN, Hespanha JP, Kriegman DJ (1997) Eigenfaces vs. fisherfaces: recognition using class specific linear projection. IEEE Trans Pattern Anal Mach Intell 19(7):711–720
Belkin M, Niyogi P (2003) Laplacian eigenmaps for dimensionality reduction and data representation. Neural Comput 15(6):1373–1396
Cai D, He X, Han J, Zhang HJ (2006) Orthogonal laplacianfaces for face recognition. IEEE Trans Image Process 15(11):3608–3614
Cai D, He X, Han J et al. (2007) Isometric projection. AAAI: 528–533
Cai D, He X, Zhou K, Han J, Bao H (2007) Locality sensitive discriminant analysis. IJCAI 2007:1713–1726
Chen HT, Chang HW, Liu TL (2005) Local discriminant embedding and its variants. Computer vision and pattern recognition, 2005. CVPR 2005. IEEE Comput Soc Conf IEEE 2:846–853
Crammer K, Singer Y (2002) On the learnability and design of output codes for multiclass problems. Mach Learn 47(2):201–233
Donoho DL (2006) Compressed sensing. IEEE Trans Inf Theory 52(4):1289–1306
Fan RE, Chang KW, Hsieh CJ, Wang XR, Lin CJ (2008) Liblinear: a library for large linear classification. J Mach Learn Res 9(Aug):1871–1874
Fletcher R (2013) Practical methods of optimization. John Wiley & Sons
Gao Q, Liu J, Cui K, Zhang H, Wang X (2014) Stable locality sensitive discriminant analysis for image recognition. Neural Netw 54:49–56
Gao Q, Ma J, Zhang H, Gao X, Liu Y (2013) Stable orthogonal local discriminant embedding for linear dimensionality reduction. IEEE Trans Image Process 22(7):2521–2531
He X, Cai D, Han J (2008) Learning a maximum margin subspace for image retrieval. IEEE Trans Knowl Data Eng 20(2):189–201
He X, Cai D, Yan S, Zhang HJ (2005) Neighborhood preserving embedding. Comput Vision, 2005. ICCV 2005. Tenth IEEE Int Conf IEEE 2:1208–1213
He X, Niyogi P (2004) Locality preserving projections. Advances in neural information processing systems: 153–160
He X, Yan S, Hu Y, Niyogi P, Zhang HJ (2005) Face recognition using laplacianfaces. IEEE Trans Pattern Anal Mach Intell 27(3):328–340
Hsieh CC, Hsih MH, Jiang MK, Cheng YM, Liang EH (2016) Effective semantic features for facial expressions recognition using svm. Multimed Tools Appl 75(11):6663–6682
Ji S, Ye J (2009) Linear dimensionality reduction for multi-label classification. IJCAI 9:1077–1082
Jolliffe IT (1986) Principal component analysis and factor analysis. Principal component analysis: 115–128. Springer
Kokiopoulou E, Saad Y (2007) Orthogonal neighborhood preserving projections: a projection-based dimensionality reduction technique. IEEE Trans Pattern Anal Mach Intell 29(12):2143–2156
Liu Y, Gao Q, Gao X, Shao L (2018) L2, 1-norm discriminant manifold learning. IEEE Access 6:40723–40734
Liu Y, Gao Q, Miao S, Gao X, Nie F, Li Y (2017) A non-greedy algorithm for l1-norm lda. IEEE Trans Image Process 26(2):684–695
Majumdar A, Ward RK (2010) Robust classifiers for data reduced via random projections. IEEE Trans Syst Man Cybernet Part B (Cybernetics) 40(5):1359–1371
Nene SA, Nayar S, Murase H (1996) Columbia object image library (COIL-20). Technical report CUCS-005-96
Nie F, Xiang S, Song Y, Zhang C (2009) Orthogonal locality minimizing globality maximizing projections for feature extraction. Opt Eng 48(1):017202–017202
Nikitidis S, Tefas A, Pitas I (2014) Maximum margin projection subspace learning for visual data analysis. IEEE Trans Image Process 23(10):4413–4425
Paul S, Boutsidis C, Magdon-Ismail M, Drineas P (2013) Random projections for support vector machines. Artificial intelligence and statistics: 498–506
Roweis ST, Saul LK (2000) Nonlinear dimensionality reduction by locally linear embedding. Science 290(5500):2323–2326
Shi Q, Shen C, Hill R, Hengel AVD (2012) Is margin preserved after random projection? arXiv preprint arXiv:1206.4651
Sugiyama M (2007) Dimensionality reduction of multimodal labeled data by local fisher discriminant analysis. J Mach Learn Res 8(May):1027–1061
Tenenbaum JB, De Silva V, Langford JC (2000) A global geometric framework for nonlinear dimensionality reduction. Science 290(5500):2319–2323
Varatharajan R, Manogaran G, Priyan M (2018) A big data classification approach using lda with an enhanced svm method for ecg signals in cloud computing. Multimed Tools Appl 77(8):10195–10215
Wang R, Nie F, Hong R, Chang X, Yang X, Yu W (2017) Fast and orthogonal locality preserving projections for dimensionality reduction. IEEE Trans Image Process 26(10):5019–5030
Xu Y, Fang X, Wu J, Li X, Zhang D (2016) Discriminative transfer subspace learning via low-rank and sparse representation. IEEE Trans Image Process 25(2):850–863
Xu Y, Yang JY, Jin Z (2003) Theory analysis on fslda and ulda. Pattern Recogn 36(12):3031–3033
Xu Y, Yang JY, Jin Z (2004) A novel method for fisher discriminant analysis. Pattern Recogn 37(2):381–384
Yan S, Xu D, Zhang B, Zhang HJ, Yang Q, Lin S (2007) Graph embedding and extensions: a general framework for dimensionality reduction. IEEE Trans Pattern Anal Mach Intell 29(1):40–51
Yang B (2009) Svm-induced dimensionality reduction and classification. 2009 Second Int Conf Intell Comput Technol Autom 4:275–278
Yang X, Liu G, Yu Q, Wang R (2017) Stable and orthogonal local discriminant embedding using trace ratio criterion for dimensionality reduction. Multimed Tools Appl 77(3):3071–3081
Acknowledgements
The work was supported by the Natural Science Foundation of China: 61773302, and the Natural Science Foundation of Ningbo: 2018A610049.
Author information
Authors and Affiliations
Corresponding authors
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Xie, D., Nie, F. & Gao, Q. On the optimal solution to maximum margin projection pursuit. Multimed Tools Appl 79, 35441–35461 (2020). https://doi.org/10.1007/s11042-019-07749-0
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-019-07749-0