Advertisement

Neural Computing and Applications

, Volume 31, Issue 7, pp 2117–2130 | Cite as

Support vector-based fuzzy classifier with adaptive kernel

  • Hamed Ganji
  • Shahram KhadiviEmail author
  • Mohammad Mehdi Ebadzadeh
Original Article

Abstract

Fuzzy neural network (FNN) and support vector machine (SVM) are popular techniques for pattern classification. FNN has a good local representation power and human-like reasoning benefit. But the learning algorithms used in most FNN classifiers only focus on minimizing empirical risk. SVM has excellent generalization performance due to focusing on simultaneously minimizing both empirical and expected risks. But this performance is dependent on choosing an appropriate kernel function. To overcome these challenges some efforts have been made to combine SVM and FNN; however, the methods proposed so far are faced with some major problems such as significantly increasing number of fuzzy rules. In this paper a support vector-based fuzzy classifier with adaptive kernel (SVFC-AK) is proposed. SVFC-AK implements the mentioned combining idea and yet not have disadvantages of the previous methods. This is realized by leveraging an underlying relation between FNN and kernel SVM. More precisely, SVFC-AK is a TS-type FNN for which the parameters are tuned through a new adaptive fuzzy kernel SVM. Also, the proposed fuzzy kernel, which is calculated automatically, is inspired by the fuzzy rule-based representation of FNN. Moreover, the fuzzy rules of SVFC-AK are generated using a modified clustering method based on Gaussian mixture model, which determines the number of required rules automatically. The experimental results illustrate that the classification performance of SVFC-AK is competitive or even better than some related methods, while the number of its fuzzy rules or hyper-parameters is substantially smaller.

Keywords

Fuzzy neural network (FNN) Support vector machine (SVM) Adaptive fuzzy kernel Pattern classification Clustering 

Notes

Compliance with ethical standards

Conflicts of interest

The authors declare that they have no conflict of interest.

References

  1. 1.
    Aeberhard S, Coomans D, De Vel O (1992) The classification performance of rda. Department of Computer Science and Department of Mathematics and Statistics, James Cook University of North Queensland, Technical Report, pp 92–01Google Scholar
  2. 2.
    Aiolli F, Donini M (2015) Easymkl: a scalable multiple kernel learning algorithm. Neurocomputing 169:215–224CrossRefGoogle Scholar
  3. 3.
    Argyriou A, Micchelli CA, Pontil M (2005) Learning convex combinations of continuously parameterized basic kernels. In: Auer P, Meir R (eds) Learning theory. Springer, Berlin, pp 338–352zbMATHCrossRefGoogle Scholar
  4. 4.
    Batuwita R, Palade V (2010) Fsvm-cil: fuzzy support vector machines for class imbalance learning. IEEE Trans Fuzzy Syst 18(3):558–571CrossRefGoogle Scholar
  5. 5.
    Berthold MR, Diamond J (1998) Constructive training of probabilistic neural networks. Neurocomputing 19(1):167–183CrossRefGoogle Scholar
  6. 6.
    Blake CL, Merz CJ (1998) Uci repository of machine learning databases. http://www.ics.uci.edu/~mlearn/MLRepository.html
  7. 7.
    Chai Y, Jia L, Zhang Z (2009) Mamdani model based adaptive neural fuzzy inference system and its application. Int J Comput Intell 5(1):22–29Google Scholar
  8. 8.
    Chang CC, Lin CJ (2011) Libsvm: a library for support vector machines. ACM Trans Intell Syst Technol (TIST) 2(3):27. http://www.csie.nte.edu.tw/~cjlin/libsvm/
  9. 9.
    Chapelle O, Vapnik V, Bousquet O, Mukherjee S (2002) Choosing multiple parameters for support vector machines. Mach Learn 46(1–3):131–159zbMATHCrossRefGoogle Scholar
  10. 10.
    Chen Y, Wang JZ (2003) Support vector learning for fuzzy rule-based classification systems. IEEE Trans Fuzzy Syst 11(6):716–728CrossRefGoogle Scholar
  11. 11.
    Cheng W, Juang C (2013) A fuzzy model with online incremental svm and margin-selective gradient descent learning for classification problems. IEEE Trans Fuzzy Syst 21(99):324–337Google Scholar
  12. 12.
    Chiang JH, Hao PY (2004) Support vector learning mechanism for fuzzy rule-based modeling: a new approach. IEEE Trans Fuzzy Syst 12(1):1–12CrossRefGoogle Scholar
  13. 13.
    Cortes C, Mohri M, Rostamizadeh A (2010) Two-stage learning kernel algorithms. In: Proceedings of the 27th international conference on machine learning (ICML-10), pp 239–246Google Scholar
  14. 14.
    Cristianini N, Kandola J, Elisseeff A, Shawe-Taylor J (2006) On kernel target alignment. In: Holmes DE, Jain LC (eds) Innovations in machine learning. Springer, Berlin, pp 205–256CrossRefGoogle Scholar
  15. 15.
    Cristianini N, Shawe-Taylor J (2000) An Introduction to support vector machines and other kernel-based learning methods. Cambridge University Press, CambridgezbMATHCrossRefGoogle Scholar
  16. 16.
    Diaz JG (2016) Supervised machine learning with kernel embeddings of fuzzy sets and probability measures. Ph.D. thesis, IME USPGoogle Scholar
  17. 17.
    Ebadzadeh MM, Salimi-Badr A (2015) Cfnn: correlated fuzzy neural network. Neurocomputing 148:430–444CrossRefGoogle Scholar
  18. 18.
    Ester M, Kriegel HP, Sander J, Xu X et al. (1996) A density-based algorithm for discovering clusters in large spatial databases with noise. In: Kdd, vol. 96, pp 226–231Google Scholar
  19. 19.
    Fan Q, Wang Z, Li D, Gao D, Zha H (2017) Entropy-based fuzzy support vector machine for imbalanced datasets. Knowl Based Syst 115:87–99CrossRefGoogle Scholar
  20. 20.
    Forghani Y, Yazdi HS (2014) Robust support vector machine-trained fuzzy system. Neural Netw 50:154–165zbMATHCrossRefGoogle Scholar
  21. 21.
    Fraley C, Raftery AE (1998) How many clusters? Which clustering method? Answers via model-based cluster analysis. Comput J 41(8):578–588zbMATHCrossRefGoogle Scholar
  22. 22.
    Fraley C, Raftery AE (2002) Model-based clustering, discriminant analysis, and density estimation. J Am Stat Assoc 97(458):611–631MathSciNetzbMATHCrossRefGoogle Scholar
  23. 23.
    Gabrys B, Bargiela A (2000) General fuzzy min-max neural network for clustering and classification. IEEE Trans Neural Netw 11(3):769–783CrossRefGoogle Scholar
  24. 24.
    Guevara J, Hirata R, Canu S (2013) Kernel functions in takagi-sugeno-kang fuzzy system with nonsingleton fuzzy input. In: 2013 IEEE international conference on fuzzy systems (FUZZ). IEEE, pp 1–8Google Scholar
  25. 25.
    Guevara J, Hirata R, Canu S (2014) Positive definite kernel functions on fuzzy sets. In: 2014 IEEE international conference on fuzzy systems (FUZZ-IEEE). IEEE, pp 439–446Google Scholar
  26. 26.
    Guevara J, Hirata R, Canu S (2017) Cross-product kernels for fuzzy set similarity. Working paper or preprintGoogle Scholar
  27. 27.
    Heo G, Gader P (2011) Robust kernel discriminant analysis using fuzzy memberships. Pattern Recognit 44(3):716–723zbMATHCrossRefGoogle Scholar
  28. 28.
    Hsu CW, Lin CJ (2002) A comparison of methods for multiclass support vector machines. IEEE Trans Neural Netw 13(2):415–425CrossRefGoogle Scholar
  29. 29.
    Huang HC, Chuang YY, Chen CS (2012) Multiple kernel fuzzy clustering. IEEE Trans Fuzzy Syst 20(1):120–134CrossRefGoogle Scholar
  30. 30.
    Jawanpuria P, Nath JS, Ramakrishnan G (2015) Generalized hierarchical kernel learning. J Mach Learn Res 16:617–652MathSciNetzbMATHGoogle Scholar
  31. 31.
    Juang CF (2002) A tsk-type recurrent fuzzy network for dynamic systems processing by neural network and genetic algorithms. IEEE Trans Fuzzy Syst 10(2):155–170CrossRefGoogle Scholar
  32. 32.
    Juang CF, Chen GC (2012) A ts fuzzy system learned through a support vector machine in principal component space for real-time object detection. IEEE Trans Ind Electron 59(8):3309–3320CrossRefGoogle Scholar
  33. 33.
    Juang CF, Chiu SH, Chang SW (2007) A self-organizing ts-type fuzzy network with support vector learning and its application to classification problems. IEEE Trans Fuzzy Syst 15(5):998–1008CrossRefGoogle Scholar
  34. 34.
    Juang CF, Lin CT (1998) An online self-constructing neural fuzzy inference network and its applications. IEEE Trans Fuzzy Syst 6(1):12–32CrossRefGoogle Scholar
  35. 35.
    Kanzawa Y, Endo Y, Miyamoto S (2010) On kernel fuzzy c-means for data with tolerance using explicit mapping for kernel data analysis. In: 2010 IEEE international conference on fuzzy systems (FUZZ). IEEE, pp 1–6Google Scholar
  36. 36.
    Karypis G, Han EH, Kumar V (1999) Chameleon: hierarchical clustering using dynamic modeling. Computer 32(8):68–75CrossRefGoogle Scholar
  37. 37.
    Khan NM, Ksantini R, Ahmad IS, Guan L (2014) Sn-svm: a sparse nonparametric support vector machine classifier. Signal Image Video Process 8(8):1625–1637CrossRefGoogle Scholar
  38. 38.
    Kloft M, Brefeld U, Sonnenburg S, Zien A (2011) Lp-norm multiple kernel learning. J Mach Learn Res 12:953–997MathSciNetzbMATHGoogle Scholar
  39. 39.
    Lanckriet GR, Cristianini N, Bartlett P, Ghaoui LE, Jordan MI (2004) Learning the kernel matrix with semidefinite programming. J Mach Learn Res 5:27–72MathSciNetzbMATHGoogle Scholar
  40. 40.
    Lee HM, Chen CM, Chen JM, Jou YL (2001) An efficient fuzzy classifier with feature selection based on fuzzy entropy. IEEE Trans Syst Man Cybern Part B Cybern 31(3):426–432CrossRefGoogle Scholar
  41. 41.
    Lee MM, Keerthi SS, Ong CJ, DeCoste D (2004) An efficient method for computing leave-one-out error in support vector machines with gaussian kernels. IEEE Trans Neural Netw 15(3):750–757CrossRefGoogle Scholar
  42. 42.
    Lin CF, Wang SD (2002) Fuzzy support vector machines. IEEE Trans Neural Netw 13(2):464–471CrossRefGoogle Scholar
  43. 43.
    Lin CT, Lee CSG (1996) Neural fuzzy systems: a neural-fuzzy synergism to intelligent systems. Prentice-Hall, Upper Saddle RiverGoogle Scholar
  44. 44.
    Lin CT, Yeh CM, Hsu CF (2004) Fuzzy neural network classification using support vector machine. In: Proceedings of IEEE international symposium on circuits and systems, p 724727Google Scholar
  45. 45.
    Lin CT, Yeh CM, Liang SF, Chung JF, Kumar N (2006) Support-vector-based fuzzy neural network for pattern classification. IEEE TransFuzzy Syst 14(1):31–41Google Scholar
  46. 46.
    Mercer J (1909) Functions of positive and negative type, and their connection with the theory of integral equations. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character, pp 415–446Google Scholar
  47. 47.
    Michie D, Spiegelhalter DJ, Taylor CC (1994) Machine learning, neural and statistical classification. ftp://ftp.stams.strath.ac.uk/pub/
  48. 48.
    Ong CS, Williamson RC, Smola AJ (2005) Learning the kernel with hyperkernels. J Mach Learn Res 6:1043–1071MathSciNetzbMATHGoogle Scholar
  49. 49.
    Rakotomamonjy A, Bach FR, Canu S, Grandvalet Y (2008) Simplemkl. J Mach Learn Res 9(11):2491–2521MathSciNetzbMATHGoogle Scholar
  50. 50.
    Saitoh S, Saitoh S (1988) Theory of reproducing kernels and its applications, vol 189. Longman Scientific and Technical Harlow, HarlowzbMATHGoogle Scholar
  51. 51.
    Sollich P (2002) Bayesian methods for support vector machines: evidence and predictive class probabilities. Mach Learn 46(1–3):21–52zbMATHCrossRefGoogle Scholar
  52. 52.
    Tsujinishi D, Abe S (2003) Fuzzy least squares support vector machines for multiclass problems. Neural Netw 16(5):785–792CrossRefGoogle Scholar
  53. 53.
    Vapnik V (2000) The nature of statistical learning theory. Springer, BerlinzbMATHCrossRefGoogle Scholar
  54. 54.
    Vlassis N, Likas A (2002) A greedy em algorithm for gaussian mixture learning. Neural Process Lett 15(1):77–87zbMATHCrossRefGoogle Scholar
  55. 55.
    Wang J, Hua J, Guo J (2010) Fuzzy maximum scatter discriminant analysis with kernel methods. In: 2010 seventh international conference on fuzzy systems and knowledge discovery (FSKD), vol. 2. IEEE, pp 560–564Google Scholar
  56. 56.
    Wang Y, Wang S, Lai KK (2005) A new fuzzy support vector machine to evaluate credit risk. IEEE Trans Fuzzy Syst 13(6):820–831CrossRefGoogle Scholar
  57. 57.
    Xiong H, Swamy M, Ahmad MO (2005) Optimizing the kernel in the empirical feature space. IEEE Trans Neural Netw 16(2):460–474CrossRefGoogle Scholar
  58. 58.
    Zhong S, Chen D, Xu Q, Chen T (2013) Optimizing the gaussian kernel function with the formulated kernel target alignment criterion for two-class pattern classification. Pattern Recognit 46(7):2045–2054zbMATHCrossRefGoogle Scholar

Copyright information

© The Natural Computing Applications Forum 2017

Authors and Affiliations

  • Hamed Ganji
    • 1
  • Shahram Khadivi
    • 1
    Email author
  • Mohammad Mehdi Ebadzadeh
    • 1
  1. 1.Computer Engineering DepartmentAmirkabir University of TechnologyTehranIran

Personalised recommendations