Skip to main content
SpringerLink
Log in

An Iterative Method for Deciding SVM and Single Layer Neural Network Structures

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

We present two new classifiers for two-class classification problems using a new Beta-SVM kernel transformation and an iterative algorithm to concurrently select the support vectors for a support vector machine (SVM) and the hidden units for a single hidden layer neural network to achieve a better generalization performance. To construct the classifiers, the contributing data points are chosen on the basis of a thresholding scheme of the outputs of a single perceptron trained using all training data samples. The chosen support vectors are used to construct a new SVM classifier that we call Beta-SVN. The number of chosen support vectors is used to determine the structure of the hidden layer in a single hidden layer neural network that we call Beta-NN. The Beta-SVN and Beta-NN structures produced by our method outperformed other commonly used classifiers when tested on a 2-dimensional non-linearly separable data set.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Alimi AM (1997) Evolutionary neuro-fuzzy approach to recognize on-line Arabic handwriting. In: Proceedings of the international conference on document analysis and recognition, ICDAR, vol 1, pp 382–386

  2. Alimi AM (1997) Neuro-fuzzy approach to recognize Arabic handwritten characters. In: Proceedings of the IEEE international conference on neural networks, ICNN, vol 3, pp 1397–1400

  3. Alimi AM (2002) Evolutionary computation for the recognition of on-line cursive handwriting. IETE J Res 48(5): 385–396

    Google Scholar 

  4. Alimi AM (2003) Beta neuro-fuzzy systems. TASK Quart J 7(1): 23–41

    Google Scholar 

  5. Alimi AM, Derbel N (1995) New hierarchical control algorithm for large scale time-delay systems. Control comput 23(2): 48–52

    Google Scholar 

  6. Aouiti C, Alimi AM, Karray F, Maalej A (2002) A hierarchical genetic algorithm for the design of beta basis function neural network. In: Proceedings of the international joint conference on neural networks, vol 2, pp 1246–1251

  7. Aouiti C, Alimi AM, Maalej A (2002) A genetic-designed beta basis function neural network for multi-variable functions approximation. Syst Anal Model Simul 42(7): 975–1009

    MATH  Google Scholar 

  8. Aouiti C, Alimi AM, Karray F, Maalej A (2003) Evolutionary approach for the beta function based fuzzy systems. In: Proceedings of IEEE international conference on fuzzy systems, vol 1, pp 179–184

  9. Aouiti C, Alimi AM, Karray F, Maalej A (2005) The design of beta basis function neural network and beta fuzzy systems by a hierarchical genetic algorithm. Fuzzy Sets Syst 154(2): 251–274

    Article  MathSciNet  MATH  Google Scholar 

  10. Ayat NE, Cheriet M, Suen CY (2001) Kmod a new support vector machine kernel for pattern recognition: application to digit image recognition. In: Proceedings of international conference on document analysis and recognition ICDAR, Sep 2001, pp 1215–1219

  11. Bahlmann C, Haasdonk B, Burkhardt H (2002) On-line handwriting recognition with support vector machines-a kernel approach. In: Proceedings of the eighth international workshop on frontiers in handwriting recognition (IWFHR’02). IEEE Computer Society, Washington, pp 49–54

  12. Baudat G, Anouar F (2003) Feature vector selection and projection using kernels. Neurocomputing 55: 21–38

    Article  Google Scholar 

  13. Bayro-Corrochano E, Vallejo R, Arana-Daniel N (2005) Geometric preprocessing, geometric feedforward neural networks and Clifford support vector machines for visual learning. Neurocomputing 67:54–105, geometrical Methods in Neural Networks and Learning

    Google Scholar 

  14. Ben Jlaiel M, Kanoun S, Alimi AM, Mullot R (2007) Three decision levels strategy for Arabic and Latin texts differentiation in printed and handwritten natures. In: Proceedings of the international conference on document analysis and recognition, ICDAR, vol 2, No 4377086, pp 1103–1107

  15. Bezine H, Derbel N, Alimi AM (2002) Fuzzy control of robot manipulators: some issues on design and rule base size reduction. Eng Appl Artif Intell 15(5): 401–416

    Article  Google Scholar 

  16. Boser BE, Guyon IM, Vapnik VN (1992) A training algorithm for optimal margin classifiers. In: Proceedings of the fifth annual workshop on computational learning theory, COLT ‘92. ACM, New York, pp 144–152

  17. Bradley PS, Mangasarian OL (1998) Massive data discrimination via linear support vector machines. University of Wisconsin Madison, Mathematical Programming Technical Report 1998–2005

  18. Bravo C, Lobato JL, Weber R, L’Huillier G (2008) A hybrid system for probability estimation in multiclass problems combining SVMs and neural networks. In: Proceedings of the 2008 8th international conference on hybrid intelligent systems, HIS ‘08. IEEE Computer Society, Washington, pp 649–654

  19. Burges CJC (1998) A tutorial on support vector machines for pattern recognition. Data Mining and Knowledge Discovery 2: 121–167

    Article  Google Scholar 

  20. Burges CJC, Schölkopf B (1997) Improving the accuracy and speed of support vector machines. In: Advances in neural information processing systems 9. MIT Press, Cambridge, pp 375–381

  21. Campbell C (2000) Algorithmic approaches to training support vector machines: a survey. In: Proceedings of ESANN2000. D-Facto Publications, Belgium, pp 27–36

  22. Cauwenberghs G, Poggio T (2001) Incremental and decremental support vector machine learning. In: Advances in neural information processing systems, vol 13

  23. Chen S, Cowan CFN, Grant PM (1991) Orthogonal least squares learning algorithm for radial basis function networks. IEEE Trans Neural Netw 2(2): 302–309

    Article  Google Scholar 

  24. Chew HG, Bogner RE, Lim CC (2000) Target detection in radar imagery using support vector machines with training size biasing. In: Proceedings of the international conference on control, automation, robotics and vision (ICARCV), Singapore

  25. Collobert R, Bengio S, Williamson C (2001) SVMTorch: support vector machines for large-scale regression problems. J Mach Learn Res 1: 143–160

    Article  MathSciNet  Google Scholar 

  26. Cortes C (1995) Prediction of generalization ability in learning machines. PhD Dissertation, Computer Science Department, University of Rochester

  27. Cortes C, Vapnik V (1995) Support-vector networks. Mach Learn 20: 273–297

    MATH  Google Scholar 

  28. Cristianini N, Campbell C, Shawe-Taylor J (1999) A multiplicative updating algorithm for training support vector machine. In: Proceedings of european symposium on artificial neural networks ESANN, pp 189–194

  29. Deb AK, Jayadeva , Gopal M, Chandra S (2007) SVM-based tree-type neural networks as a critic in adaptive critic designs for control. IEEE Trans Neural Netw 18(4): 1016–1030

    Article  Google Scholar 

  30. Fukumizu K, Bach F, Jordan MI (2004) Dimensionality reduction for supervised learning with reproducing kernel Hilbert spaces. J Mach Learn Res 5: 73–99

    MathSciNet  Google Scholar 

  31. Gui R-Z (2006) Utilization of support vector machine based on neural network to suppress ocean clutter and zero frequency disturbances. In: Proceedings of IEEE International Conference on vehicular electronics and safety 2006. ICVES 2006, Dec, pp 496–501

  32. Hamdani TM, Alimi AM (2003) β-SVM a new support vector machine. In: David RFA, Pearson W, Steele Nigel C (eds) Artificial neural networks and genetic algorithms. Springer-Verlag, Berlin, pp 63–68

    Google Scholar 

  33. Hamdani TM, Alimi AM (2003) Support vector machines for digit recognition. In: Proceedings of second IEEE international conference on systems, signals and decision (SSD’03), Sousse, Tunisia, Mar 2003

  34. Hamdani TM, Alimi AM (2004) How are β-SVM good kernels. In: Proceedings of world computer congress (WCC): artificial intelligence applications and innovations (AIAI), Aug 2004, pp 383–392

  35. Hamdani TM, Alimi AM, Karray F (2006) Distributed genetic algorithm with bi-coded chromosomes and a new evaluation function for features selection. In: Proceedings of IEEE Congress on evolutionary computation 2006. CEC 2006, pp 581–588

  36. Hamdani TM, Won J-M, Alimi AM, Karray F et al (2007) Multi-objective Feature Selection with NSGA II. In: Beliczynski B (eds) Adaptive and natural computing algorithms. Lecture Notes in Computer Science, vol 4431. Springer, New York, pp 240–247

    Chapter  Google Scholar 

  37. Hamdani TM, Alimi AM, Karray F (2008) Enhancing the structure and parameters of the centers for BBF fuzzy neural network classifier construction based on data structure. In: Proceedings of IEEE international joint conference on neural networks 2008. IJCNN 2008. (IEEE World Congress on Computational Intelligence). Jun 2008, pp 3174–3180

  38. Hassine R, Karray F, Alimi AM, Selmi M (2003) Approximation properties of fuzzy systems for smooth functions and their first-order derivative. IEEE Trans Syst Man Cybern A 33(2): 160–168

    Article  Google Scholar 

  39. Hassine R, Karray F, Alimi AM, Selmi M (2006) Approximation properties of piece-wise parabolic functions fuzzy logic systems. Fuzzy Sets Syst 157(4): 501–515

    Article  MathSciNet  MATH  Google Scholar 

  40. Joachims T (1998) Text categorization with support vector machines: learning with many relevant features. In: Nédellec C, Ruveirol C (eds) Proceedings of ECML-98, 10th european conference on machine learning, Springer, Heidelberg, pp 137–142

  41. Joachims T (1999) Making large-scale support vector machine learning practical. In: Advances in kernel methods: support vector learning. MIT Press, Cambridge, pp 169–184

  42. Joachims T (2000) Estimating the generalization performance of an SVM efficiently. In: Proceedings of the seventeenth international conference on machine learning, ICML’00. Morgan Kaufmann Publishers Inc, San Francisco, pp 431–438

  43. Kallel IAAM (2002) A multi-agent approach for genetic algorithm implementation. In: Proceedings of the IEEE international conference on systems, man and cybernetics. vol 7, pp 358–363

  44. Kallel I, Baklouti N, Alimi AM (2006) Accuracy preserving interpretability with hybrid hierarchical genetic fuzzy modeling: case of motion planning robot controller. In: Proceedings of the 2006 international symposium on evolving fuzzy systems, EFS’06, No 4016715, pp 312–317

  45. Kherallah M, Haddad L, Alimi AM, Mitiche A (2008) On-line handwritten digit recognition based on trajectory and velocity modeling. Pattern Recognit Lett 29(5): 580–594

    Article  Google Scholar 

  46. Lin C-J (2002) A formal analysis of stopping criteria of decomposition methods for support vector machines. IEEE Trans Neural Netw 13(5): 1045–1052

    Article  Google Scholar 

  47. Lin C-T, Yeh C-M, Liang S-F, Chung J-F, Kumar N (2006) Support-vector-based fuzzy neural network for pattern classification. IEEE Trans Fuzzy Syst 14(1): 31–41

    Article  Google Scholar 

  48. Lu J, Plataniotis K, Venetsanopoulos A (2003) Face recognition using kernel direct discriminant analysis algorithms. IEEE Trans Neural Netw 14(1): 117–126

    Article  Google Scholar 

  49. Mangasarian OL, Musicant DR (2001) Lagrangian support vector machines. J Mach Learn Res 1: 161–177

    Article  MathSciNet  MATH  Google Scholar 

  50. Masmoudi M, Samet M, Alimi AM (2000) A bipolar implementation of the beta neuron. Int J Electron 87(6): 675–682

    Article  Google Scholar 

  51. Moalla I, LeBourgeois F, Emptoz H, Alimi AM (2006) Contribution to the discrimination of the medieval manuscript texts: application in the paleography. In: Lecture Notes in Computer Science, vol 3872, Springer-Verlag, New York, pp 25–37

  52. Mukherjee S, Osuna E, Girosi F (1997) Nonlinear prediction of chaotic time series using support vector machines. In: Proceedings of the 1997 IEEE workshop, neural networks for signal processing [1997] VII, sep 1997, pp 511–520

  53. Müller K-R, Smola AJ, Rätsch G, Schölkopf B, Kohlmorgen J, Vapnik V (1997) Predicting time series with support vector machines. In: Proceedings of the 7th international conference on artificial neural networks, ICANN ‘97. Springer-Verlag, London, pp 999–1004

  54. Navia-Vázquez A (2007) Letters: support vector perceptrons. Neurocomputing 70(4–6): 1089–1095

    Google Scholar 

  55. Osuna E, Freund R, Girosi F (1997) An improved training algorithm for support vector machines. In: Proceedings of the 1997 IEEE workshop, neural networks for signal processing [1997] VII, sep 1997, pp 276–285

  56. Osuna E, Freund R, Girosit F (1997) Training support vector machines: an application to face detection. In: Proceedings of IEEE computer society conference on computer vision and pattern recognition 1997, Jun 1997, pp 130–136

  57. Plamondon R, Alimi AM (1997) Speed/accuracy trade-offs in target-directed movements. Behav Brain Sci 20(2): 279–349

    Google Scholar 

  58. Plamondon R, Alimi AM, Yergeau P, Leclerc F (1993) Modelling velocity profiles of rapid movements: a comparative study. Biol Cybern 69(2): 119–128

    Article  Google Scholar 

  59. Platt J (1998) Sequential minimal optimization: a fast algorithm for training support vector machines. Microsoft Research, TechReport MSRTR-98-14, Apr 1998

  60. Platt JC (1999) Fast training of support vector machines using sequential minimal optimization. In: Advances in kernel methods: support vector learning. MIT Press, Cambridge, pp 185–208

  61. Raudys S (1998) Evolution and generalization of a single neuron: II. Complexity of statistical classifiers and sample size considerations. Neural Netw 11(2): 297–313

    Article  Google Scholar 

  62. Romero E, Márquez L, Carreras X (2004) Margin maximization with feed-forward neural networks: a comparative study with SVM and AdaBoost. Neurocomputing 57:313–344. New aspects in neurocomputing: 10th European symposium on artificial neural networks 2002

    Google Scholar 

  63. Suykens JAK, Vandewalle J (1999) Training multilayer perceptron classifiers based on a modified support vector method. IEEE Trans Neural Netw 10(4): 907–911

    Article  Google Scholar 

  64. Vapnik VN (1995) The nature of statistical learning theory. Springer-Verlag, New York

    MATH  Google Scholar 

  65. Xia Y, Wang J (2004) A one-layer recurrent neural network for support vector machine learning. IEEE Trans Syst Man Cybern B 34(2): 1261–1269

    Article  Google Scholar 

  66. Xu J, Zhang X, Li Y (2001) Kernel MSE algorithm: a unified framework for KFD, LS-SVM and KRR. In: Proceedings of international joint conference on neural networks, IJCNN ‘01, vol 2, pp 1486–1491

  67. Zouari H, Heulte L, Lecourtier Y, Alimi AM (2003) Simulating classifier outputs for evaluating parallel combination methods. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol 2709. Springer-Verlag, New York, pp 296–305

  68. Zouari H, Heutte L, Lecourtier Y, Alimi AM (2004) Building diverse classifier outputs to evaluate the behavior of combination methods: the case of two classifiers. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol 3077. Springer-Verlag, New York, pp 273–282

Download references

Author information

Authors and Affiliations

  1. Research Group on Intelligent Machines (REGIM), University of Sfax, National Engineering School of Sfax (ENIS), BP 1173, 3038, Sfax, Tunisia

    Tarek M. Hamdani & Adel M. Alimi

  2. Department of Electrical and Computer Engineering, University of West Florida, 11000 University Pkwy, Pensacola, FL, 32514, USA

    Mohamed A. Khabou

Authors
  1. Tarek M. Hamdani
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Adel M. Alimi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mohamed A. Khabou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tarek M. Hamdani.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Hamdani, T.M., Alimi, A.M. & Khabou, M.A. An Iterative Method for Deciding SVM and Single Layer Neural Network Structures. Neural Process Lett 33, 171–186 (2011). https://doi.org/10.1007/s11063-011-9171-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-011-9171-3

Keywords

Search

Navigation