Fast Kernel Classifier Construction Using Orthogonal Forward Selection to Minimise Leave-One-Out Misclassification Rate

  • X. Hong
  • S. Chen
  • C. J. Harris
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4113)


We propose a simple yet computationally efficient construction algorithm for two-class kernel classifiers. In order to optimise classifier’s generalisation capability, an orthogonal forward selection procedure is used to select kernels one by one by minimising the leave-one-out (LOO) misclassification rate directly. It is shown that the computation of the LOO misclassification rate is very efficient owing to orthogonalisation. Examples are used to demonstrate that the proposed algorithm is a viable alternative to construct sparse two-class kernel classifiers in terms of performance and computational efficiency.


Radial Basis Function Radial Basis Function Neural Network Radial Basis Function Kernel Generalisation Capability Relevance Vector Machine 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Akaike, H.: A New Look at the Statistical Model Identification. IEEE Trans. Automatic Control AC-19, 716–723 (1974)zbMATHCrossRefMathSciNetGoogle Scholar
  2. 2.
    Vapnik, V.: The Nature of Statictical Learning Theory. Springer, New York (1995)Google Scholar
  3. 3.
    Tipping, M.E.: Sparse Bayesian Learning and the Relevance Vector Machine. J. Machine Learning Research 1, 211–244 (2001)zbMATHCrossRefMathSciNetGoogle Scholar
  4. 4.
    Scholkopf, B., Smola, A.J.: Learning with Kernels: Support Vector Machine, Regularization. Optimization and Beyond. MIT Press, Cambridge (2002)Google Scholar
  5. 5.
    Hong, X., Harris, C.J.: Nonlinear Model Structure Design and Construction using Orthogonal Least Squares and D-optimality Design. IEEE Trans. Neural Networks 13(5), 1245–1250 (2001)CrossRefGoogle Scholar
  6. 6.
    Chen, S., Billings, S.A., Luo, W.: Orthogonal Least Squares Methods and Their Applications to Non-linear System Identification. Int. J. Control 50, 1873–1896 (1989)zbMATHCrossRefMathSciNetGoogle Scholar
  7. 7.
    Mao, K.Z.: RBF Neural Network Center Selection Based on Fisher Ratio Class Separability Measure. IEEE Trans. Neural Networks 13(5), 1211–1217 (2002)CrossRefGoogle Scholar
  8. 8.
    Chen, S., Wang, X.X., Hong, X., Harris, C.J.: Kernel classifier construction using orthogonal forward selection and boosting with fisher ratio class separability. IEEE Trans. Neural Networks (2006) (accepted for publication)Google Scholar
  9. 9.
    Vapnik, V.: Statistical Learning Theory: Adaptive & Learning Systems for Signal Processing. Communication & Control (1998)Google Scholar
  10. 10.
    Stone, M.: Cross Validatory Choice and Assessment of Statistical Predictions. Applied Statistics 36, 117–147 (1974)Google Scholar
  11. 11.
    Golub, G.H., Heath, M., Wahba, G.: Generalized cross-validation as a method for choosing good ridge parameter. Technometrics 21(2), 215–223 (1979)zbMATHCrossRefMathSciNetGoogle Scholar
  12. 12.
    Hong, X., Sharkey, P.M., Warwick, K.: Automatic Nonlinear Predictive Model Construction using Forward Regression and the PRESS Statistic. IEE Proc. - Control Theory and Applications 150(3), 245–254 (2003)CrossRefGoogle Scholar
  13. 13.
    Chen, S., Hong, X., Harris, C.J., Sharkey, P.M.: Sparse modelling using orthogonal Forward Regression with PRESS Statistic and Regularization. IEEE Trans. Systems, Man and Cybernetics, Part B: Cybernetics 34(2), 898–911 (2004)CrossRefGoogle Scholar
  14. 14.
    Mackay, D.J.: Bayesian Interpolation. Neural Computation 4(3), 415–447 (1992)CrossRefGoogle Scholar
  15. 15.
    Myers, R.H.: Classical and Modern Regression with Applications, 2nd edn. PWS-KENT, Boston (1990)Google Scholar
  16. 16.
    Rätsch, G., Onoda, T., Müller, K.R.: Soft Margins for AdaBoost. Machine Learning 42(3), 287–320 (2001)zbMATHCrossRefGoogle Scholar
  17. 17.

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • X. Hong
    • 1
  • S. Chen
    • 2
  • C. J. Harris
    • 2
  1. 1.Department of CyberneticsUniversity of ReadingReadingU.K.
  2. 2.School of Electronics and Computer ScienceUniversity of SouthamptonSouthamptonU.K.

Personalised recommendations