Skip to main content
SpringerLink
Log in

Kernel ensemble support vector machine with integrated loss in shared parameters space

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In this paper, we propose a kernel ensemble SVM with integrated loss in shared parameters space. Different from the traditional multiple kernel learning methods of seeking linear combinations of basis kernels as a unified kernel, the proposed method aims to find multiple solutions in corresponding Reproducing Kernel Hilbert Spaces (RKHSs) simultaneously. To achieve this goal, we draw on the idea of multi-view data processing, and the individual kernel gram matrix is considered as one view of the data. We, therefore, propose an ensemble idea to combine those multiple individual kernel losses into a whole one through an integrated loss design. Therefore, each model can co-optimize to learn its optimal parameters by minimizing the integrated loss in multiple RKHSs. Besides, another feature of our method is the introduction of shared and specific parameters in multiple RKHSs for learning. In this manner, the proposed model can learn the common and individual structures of the data from its parameters space, thereby improving the accuracy of the classification task and further enhancing the robustness of the proposed ensemble model. Experimental results on several UCI classification and image datasets demonstrate that our method performs best among state-of-the-art MKL methods, such as SimpleMKL, EasyMKL, MREKLM, and MRMKL.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Algorithm 1
Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Notes

  1. http://cam-orl.co.uk/facedatabase.html

  2. http://vision.ucsd.edu/~leekc/ExtYaleDatabase/ExtYaleB.html

  3. http://yann.lecun.com/exdb/mnist/

  4. https://www.kaggle.com/bistaumanga/usps-dataset

  5. https://www.cs.columbia.edu/CAVE/software/softlib/coil-20.php

  6. https://www.cs.toronto.edu/~kriz/cifar.html

References

  1. Aiolli F, Donini M (2015) Easymkl: a scalable multiple kernel learning algorithm. Neurocomputing 169:215–224

    Article  Google Scholar 

  2. Anstreicher KM (2012) On convex relaxations for quadratically constrained quadratic programming. Math Program 136(2):233–251

    Article  MathSciNet  MATH  Google Scholar 

  3. Bach FR, Lanckriet GR, Jordan MI (2004) Multiple kernel learning, conic duality, and the smo algorithm. In: Proceedings of the twenty-first international conference on machine learning, p 6

  4. Bach FR, Thibaux R, Jordan MI (2005) Computing regularization paths for learning multiple kernels. Adv Neural Inf Process Syst 17:73–80

    Google Scholar 

  5. Cortes C, Mohri M, Rostamizadeh A (2012) Algorithms for learning kernels based on centered alignment. J Machine Learn Res 13(1):795–828

    MathSciNet  MATH  Google Scholar 

  6. Crammer K, Keshet J, Singer Y (2002) Kernel design using boosting. Adv Neural Inf Process Syst 15:553–560

    Google Scholar 

  7. Cristianini N, Shawe-Taylor J et al (2000) An introduction to support vector machines and other kernel-based learning methods. Cambridge University Press, Cambridge

    Book  MATH  Google Scholar 

  8. Fan Q, Wang Z, Zha H, Gao D (2017) Mreklm: a fast multiple empirical kernel learning machine. Pattern Recogn 61:197–209

    Article  MATH  Google Scholar 

  9. Gönen M, Alpaydın E (2011) Multiple kernel learning algorithms. J Machine Learn Res 12:2211–2268

    MathSciNet  MATH  Google Scholar 

  10. Gönen M, Khan S, Kaski S (2013) Kernelized bayesian matrix factorization. In: International conference on machine learning, PMLR, pp 864–872

  11. Han Y, Yang Y, Li X, Liu Q, Ma Y (2018) Matrix-regularized multiple kernel learning via (r,p) norms. IEEE Trans Neur Netw Learn Syst 29 (10):4997–5007

    Article  MathSciNet  Google Scholar 

  12. Han Y, Yang K, Yang Y, Ma Y (2016) Localized multiple kernel learning with dynamical clustering and matrix regularization. IEEE Trans Neural Netw Learn Systs 29(2):486–499

    Article  MathSciNet  Google Scholar 

  13. Hinrichs C, Singh V, Peng J, Johnson S (2012) Q-mkl: Matrix-induced regularization in multi-kernel learning with applications to neuroimaging. Adv Neural Inf Process Syst 25:1421–1429

    Google Scholar 

  14. Hu M, Chen Y, Kwok JT-Y (2009) Building sparse multiple-kernel svm classifiers. IEEE Trans Neur Netw 20(5):827–839

    Article  Google Scholar 

  15. Kim TK (2015) T test as a parametric statistic. Korean J Anesthesiol 68(6):540

    Article  MathSciNet  Google Scholar 

  16. Kimeldorf G, Wahba G (1971) Some results on tchebycheffian spline functions. J Math Anal Appl 33(1):82–95

    Article  MathSciNet  MATH  Google Scholar 

  17. Kloft M, Brefeld U, Sonnenburg S, Zien A (2011) Lp-norm multiple kernel learning. J Machine Learn Res 12:953–997

    MathSciNet  MATH  Google Scholar 

  18. Kwok J-Y, Tsang I-H (2004) The pre-image problem in kernel methods. IEEE Trans Neural Networks 15(6):1517–1525

    Article  Google Scholar 

  19. Lanckriet GR, Cristianini N, Bartlett P, Ghaoui LE, Jordan MI (2004) Learning the kernel matrix with semidefinite programming. J Mach Learn Res 5(Jan):27–72

    MathSciNet  MATH  Google Scholar 

  20. Lanckriet GR, De Bie T, Cristianini N, Jordan MI, Noble WS (2004) A statistical framework for genomic data fusion. Bioinformatics 20(16):2626–2635

    Article  Google Scholar 

  21. Li Z, Jing X-Y, Zhu X, Zhang H (2017) Heterogeneous defect prediction through multiple kernel learning and ensemble learning. In: 2017 IEEE international conference on software maintenance and evolution (ICSME). IEEE, pp 91–102

  22. Li J, Li Z, Lu G, Xu Y, Zhang B, Zhang D (2021) Asymmetric gaussian process multi-view learning for visual classification. Inf Fusion 65:108–118

    Article  Google Scholar 

  23. Liu J, Chen Y, Zhang J, Xu Z (2014) Enhancing low-rank subspace clustering by manifold regularization. IEEE Trans Image Process 23(9):4022–4030

    Article  MathSciNet  MATH  Google Scholar 

  24. Liu H, Wang W, Jiao P (2019) Content based image retrieval via sparse representation and feature fusion. In: 2019 IEEE 8th data driven control and learning systems conference (DDCLS). IEEE, pp 18–23

  25. Mao Q, Tsang IW, Gao S, Wang L (2014) Generalized multiple kernel learning with data-dependent priors. IEEE Transactions on Neural Networks and Learning Systems 26(6):1134–1148

    Article  MathSciNet  Google Scholar 

  26. Margineantu DD, Dietterich TG (1997) Pruning adaptive boosting. In: ICML, vol 97, Citeseer, pp 211–218

  27. Ong CS, Smola A, Williamson R et al (2022) Learning the kernel with hyperkernels

  28. Qi C, Zhou Z, Hu L, Wang Q (2016) A framework of multiple kernel ensemble learning for hyperspectral classification. In: 2016 intl IEEE conferences on ubiquitous intelligence & computing, advanced and trusted computing, scalable computing and communications, cloud and big data computing, internet of people, and smart world congress (UIC/ATC/ScalCom/CBDCom/IoP/SmartWorld). IEEE, pp 456–460

  29. Rakotomamonjy A, Bach F, Canu S, Grandvalet Y (2008) Simplemkl. J Machine Learn Res 9:2491–2521

    MathSciNet  MATH  Google Scholar 

  30. Recht B, Roelofs R, Schmidt L, Shankar V (2022) Do cifar-10 classifiers generalize to cifar-10? arXiv:1806.00451

  31. Scholkopf B, Smola AJ (2001) Learning with kernels: Support vector machines, regularization, optimization and beyond

  32. Shen X-J, Ni C, Wang L, Zha Z-J (2021) Sliker: Sparse loss induced kernel ensemble regression. Pattern Recogn 109:107587

    Article  Google Scholar 

  33. Sonnenburg S, Rätsch G, Schäfer C (2005) A general and efficient multiple kernel learning algorithm. Adv Neural Inf Process Syst 18:1273–1280

    MATH  Google Scholar 

  34. Sonnenburg S, Rätsch G, Schäfer C, Schölkopf B (2006) Large scale multiple kernel learning. J Machine Learn Res 7:1531–1565

    MathSciNet  MATH  Google Scholar 

  35. Sun T, Jiao L, Liu F, Wang S, Feng J (2013) Selective multiple kernel learning for classification with ensemble strategy. Pattern Recogn 46(11):3081–3090

    Article  Google Scholar 

  36. Sun L, Zhao C, Yan Z, Liu P, Duckett T, Stolkin R (2018) A novel weakly-supervised approach for rgb-d-based nuclear waste object detection. IEEE Sensors J 19(9):3487–3500

    Article  Google Scholar 

  37. Varma M, Babu BR (2009) More generality in efficient multiple kernel learning. In: Proceedings of the 26th annual international conference on machine learning, pp 1065–1072

  38. Vishwanathan S, Murty MN (2002) Ssvm: a simple svm algorithm. In: Proceedings of the 2002 international joint conference on neural networks. IJCNN’02 (Cat. No. 02CH37290), vol 3. IEEE, pp 2393–2398

  39. Wang Q, Ding Z, Tao Z, Gao Q, Fu Y (2020) Generative partial multi-view clustering. arXiv:2003.13088

  40. Wu D, Wang B, Precup D, Boulet B (2019) Multiple kernel learning-based transfer regression for electric load forecasting. IEEE Transactions on Smart Grid 11(2):1183–1192

    Article  Google Scholar 

  41. Xia H, Hoi SC (2012) Mkboost: a framework of multiple kernel boosting. IEEE Trans Knowl Data Eng 25(7):1574–1586

    Article  Google Scholar 

  42. Xu X, Tsang IW, Xu D (2013) Soft margin multiple kernel learning. IEEE Trans Neur Netw Learn Syst 24(5):749–761

    Article  Google Scholar 

  43. Ye J, Ji S, Chen J (2020) Multi-class discriminant kernel learning via convex programming. Journal of Machine Learning Research 9(4)

  44. Zheng-Peng W, Zhang X-G (2011) Elastic multiple kernel learning. Acta Automatica Sinica 37(6):693–699

    Article  MathSciNet  MATH  Google Scholar 

  45. Zhou J, Jiang Z, Chung F-L, Wang S (2022) Formulating ensemble learning of svms into a single svm formulation by negative agreement learning. IEEE Transactions on Systems Man and Cybernetics Systems

  46. Zhou J, Liu J, Luo X (2015) Rademacher complexity bound for domain adaptation regression. In: 2015 conference on technologies and applications of artificial intelligence (TAAI). IEEE, pp 273–280

Download references

Funding

This work was funded in part by the Graduate Research and Innovation Projects of Jiangsu Province (No.SJCX21_1694) and the Science and Technology Planning Social Development Project of Zhenjiang City (No.SH2021006).

Author information

Authors and Affiliations

  1. School of Computer Science and Communication Engineering, JiangSu University, JiangSu, 212013, China

    YuRen Wu, Xiang-Jun Shen, Stanley Ebhohimhen Abhadiomhen & Yang Yang

  2. Department of Computer Science University of Nigeria, Nsukka, Nigeria

    Stanley Ebhohimhen Abhadiomhen

  3. School of Mechanical Engineering, JiangSu University, JiangSu, 212013, China

    Ji-Nan Gu

Authors
  1. YuRen Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiang-Jun Shen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Stanley Ebhohimhen Abhadiomhen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yang Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ji-Nan Gu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiang-Jun Shen.

Ethics declarations

Conflict of Interests

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wu, Y., Shen, XJ., Abhadiomhen, S.E. et al. Kernel ensemble support vector machine with integrated loss in shared parameters space. Multimed Tools Appl 82, 18077–18096 (2023). https://doi.org/10.1007/s11042-022-14226-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-14226-8

Keywords

Search

Navigation